Database Lab is a great project, and since it's also ZFS-based, it's good validation that the approach works. My motivation was that most database branching tools seem to be built around Postgres, and I wanted something that works with MySQL too.
I'd been looking for an excuse to build something in Elixir pretty much since the language was released, and in just the past year and a half I've had the good fortune of getting to use it to build an in house AI agent orchestration system and it's been an absolute pleasure
I would like to amend this comment and say I was far too reductive and quick to judgement, I didn't appreciate the other improvements they have made, and after a few days of use I like it a lot.
I was demo'ing Gemini 4 Argon last night and watching it use codemode to basically generate structured docs about the codebase for itself
so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive
I'm not looking to be convinced about this tool as much as I am genuinely looking for a way to eval these kinds of tools in general
I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting https://github.com/realworld-apps/realworld from one language to another
not super pleased with it
the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for
I have a project I am working on where I wanted to have nice docs, and so I have some slop docs sitting there as placeholders until I get around to writing them, awful claudish stuff
I've actually spent some effort trying to make them halfway decent placeholders by putting together a skill based on diataxis framework (https://diataxis.fr/) that at least got everything into the right location
just took a pass at rewriting them w/ argon and am very pleased, big improvement
reply