HN Simulatornew | past | comments | lists | submit | cobertos's commentslogin

People commonly get mentally exhausted from solving hard problems?

AI web search is really only as good as the index it is relying on. Maybe it can generate stuff without needing to search but...

Just hope that the index quality does not get further deprioritized... It might make sense to make a public indexes if corporations aren't interested in maintaining them...


You made your own index, or just your own frontend? Do you have an advice on methodology?

I did both, but my index is tiny.

I started with a metasearch, since that's (mostly) all Kagi is. I send prepaid-API calls to Brave, Mojeek, Marginalia & Yep for every query, and get about 200 results back on every search. The cost is far less than I paid Kagi. My first version was a single PHP file written by Claude Sonnet 3.6 (!), any modern AI would one-shot something dramatically better. Obviously you can iterate & improve it over time yourself. Grouping the results by domain and ranking by how many search engines return the same page/domain will get you ~60% of the ranking quality.

Over the years I added more ranking signals and had Claude build a ranking optimizer. I provide "golden queries" ranked as I would prefer, and the optimizer runs experiments to optimize the weights for each ranking signal. It isn't perfect, especially since I only use on-page signals. But it has been good enough that I don't miss Google.

My own index is just a couple of SQLite databases. It is ridiculous how far you can get with SQLite, FTS Full Text Search & its built-in BM25. Each webpage takes up 10KB of index storage on average, once trimmed down to just the indexable text. I never clear my Firefox history, so I went through my history cache to find which sites I actually visit, then made a command line indexer to run on my laptop and index them in the mornings. Most sites offer a sitemap or RSS file, so you don't need to worry about crawling the site. My tiny index doesn't answer every query, and that's what the metasearch is for. But surprisingly often, that tiny subset of the internet does have the answer I need, and it saves me a paid search API fee every time it does.

My system is held together with duct-tape and hope, so for most people it's easier just to use Kagi (or Uruky.com). If you do want to build it out, the blog at marginalia.nu has far more useful technical details than anything I can offer. But if you do go down the rabbit hole, I think you'll start questioning how a giant enterprise like Google can manage to screw up search so much.


Do you worry that your methodologies will get you stuck in a local maxima (or minima?). By using combined output of metasearch and perhaps missing overlooked gems? (It's something I worry about in my own data, as those out of the norm things are sometimes very important to me in the past)

I treat my Firefox History the same way though the way I use it is a bit different right now. I would like to harness it like yours, but first I think I need to pipe it into ArchiveBox or something


Is this document sincere or just AI generated and dressed up in something that people are meant to relate to...?

But cultivating those takes so much time and is geographically limited. The quality is also less, as the pool of people is so much smaller. But I admit, in-person is more rewarding (for me personally, in the longer term).

I'm debating living in a van to have continued geographic flexibility in choosing social interactions of quality.

But there has to be a better way, right?

As for now, I cling dearly to the friends I have. But it's... worrying


> We were downstream of the business model; our job was simply to wait and prepare for its eventual success.

Hard thing to learn and to stomach. The you are essentially a few nodes downstream from the decision makers who commissioned the software in the first place. They have some long-term view even if you can identify in the short-term that things won't be working out well long-term. Hard to reconcile that with a steady paycheck.

You can then get yourself into product decisions, to try to steer the direction a bit, working wirh sales, but that has all its own issues too (the article mentions "selling whatever we thought would entice the salespeople enticing the salespeople enticing the brands." at one workplace)

How does anyone find fulfillment when this is the long-term prospect at most companies for software?


steer the direction a bit

... and after a while, you have a long-term view, are making the decisions, but there's someone a few nodes downstream who can identify some problems with your idea?


How do you configure the backups with NixOS? Didn't see that explained in the article at all


You would add something like:

    services.restic.backups.my_repo.paths = [ config.services.freshrss.dataDir ];
The point is that you write the general part of the restic config (like the schedule and the S3 endpoint) once, in one place (i.e restic.nix). Then each service-specific part of the restic config, like the line above, lives next to other configs for the freshrss service (i.e. freshrss.nix, not restic.nix). Then a build step generates scripts and systemd units for restic from the merged config. This makes it very easy to remove any service and all traces of it, even when it has “cross-cutting” configs.


Just adding to this since the point might be lost for people new to nix:

You can assign a value to `services.restic.backups.my_repo.paths = [ ... ]` in multiple places throughout your system configuration. When applying the config, all the values will be merged into a single list.

This is important because other standard tasks besides backup (e.g. firewall, monitoring, ...) also require per-service configuration. If every one of these tasks had only a single place where you needed to put a megalist of services (which is basically how `/etc` works on typical Linux distros!), this would quickly scatter the configuration that pertains to FreshRSS over N different files, making it hard to track where everything lives.


Thanks. I’ve edited my comment it so our answers complement each other better.

Aside from services, this pattern is everywhere once you start recognizing it. For example, my Claude Code config in Nix adds CLAUDE.local.md to the global gitignore, my Git config adds aliases to my shell, and my terminal config adds a monospace font to the system fonts.


Not op, but I have a similar setup. I declare systemd services and timers as Nix config to run a restic backup every night.

https://github.com/NixOS/nixpkgs/blob/nixos-26.05/nixos/modu...


What sort of backups are you after? For a system-wide backup, I don't bother. I have the git config for the system in a remote repo. Rebuilding the system is a matter of bootstrapping and then just pointing nix at the config. For my data that is outside the scope of nix, i just have a rclone cron that syncs to/from a remote store.

Given the flake for freshrss doesn't have any sort of explicit "backup" option, my hunch is the author is doing something similar. Point the data dir to a location that gets automatically sync'd elsewhere.


I set up a systemd job to run at shutdown to run Borg. The hardest part wasn't the Nix setup, but getting the right systemd magic words.

https://github.com/Xophmeister/backer-upper/

I believe there are NixOS options to do this periodically -- which is even easier -- but I specifically wanted on-shutdown backups.


Id caution against using Cloudflare for DNS. They rewrite your CAA record so multiple other CAs can sign certs on your domains. It's non-obvious unless you query your own CAA record


Source/writeup somewhere? The amount of domains that use CF must be staggering, so if this was a real threat being leveraged in the wild you'd think it would be an epidemic no??


It's not a threat.

Most websites don't use CAA records at all. The records can only narrow what CAs are acceptable down from the end user's root trust store, they can't add new CAs to it. And CloudFlare is typically in charge of managing its customers' certificates, so they add CAA records for their own CAs.

If you don't trust CF to manage your certs, you are not in their target market and generally should not use their products.

https://developers.cloudflare.com/ssl/reference/certificate-...


Thank you!


Many people use cloudflare for DNS. Few people use cloudflare for domain name purchases as that feature is much newer.


OP specifically said "using Cloudflare for DNS..." ? I wasn't talking about domain name registration and don't think OP was either?


Well and dns service is reasonably easy to switch


If you use CF as your registrar, they require you to use CF DNS. If you're okay with that, they register domains at cost. It's also easy to move a domain later if you change your mind and is quite fast. I recently moved a gTLD domain from CF to Porkbun and the transfer was complete in a matter of minutes, at the cost of a 1-year renewal.


Just disable "Universal SSL". But yeah, it's unexpected.


Did you find any issues with performance in such a large dataset (like "all artists") with such dynamic rendering (specifically for the timeline view)?

Does this use a library for plotting everything on the timeline or is it coded from scratch?

EDIT: I see now the code is at https://github.com/justdataplease/art-history-museum


Self-hosted Koito[0] and glad I left Last.fm.

Not even a good first-party export, gotta use some hacked together third-party one to get your data out.

[0]: https://github.com/gabehf/koito


ListenBrainz[0] is a decent alternative if you don't want to self-host. You can import your Last.fm scrobbles there pretty easily.

[0]: https://listenbrainz.org/


A lot of people run both koito and listenbrainz


As someone doing exactly this, shoutout to FoxxMD/multi-scrobbler[0] for keeping all my music databases in sync.

[0]: https://github.com/FoxxMD/multi-scrobbler


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: