We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible.
Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environments and write tests against that, so they are as E2E as can possibly be.
The problem then shifts from reviewing code to reviewing tests and that's why we built our own platform. We have a UI that can diff tests, so we know what changed, and a visual way to inspect what the tests actually did. A test could drive a browser like a user would, and in our UI we get a replay of that browser interaction to look at. The browser is talking to a real version of our backend, and the tests can perform assertions against the database and fakes and really anything in our system.
That's the point, we don't review the test code either. Our platform gives us a UI for inspecting not the test code but what actually happened during the test. Like a browser replay, the results of a database query, assertions against those, etc.
This is much more information dense than something like the tests and is a representation of what actually happened during the tests, rather than what the test itself did (which I agree sucks to review, especially AI-generated).
The framework is our own that bundles/adapts some familiar components: Jest-like asssertions, Playwright browser API, typed database client from Bun, Kubernetes client, etc. The tests are written in Typescript but the main code doesn't have to be (just runs containerized in the environment).
We've been building the platform and using it continuously since May but setting it up on a new project takes like 1-2 days of largely autonomous coding agent work. We are just two engineers on our team but have been onboarding other startups to the platform recently so there are a few different teams using it now for their own codebases. It's fully generic so works for any infra or stack.
We run the entire stack (browser, frontend, backend, database, etc) in a Linux VM, so latency between each of the pieces is as tiny as can be. This is quite different from "standard" E2E tests I've seen where the test browsers uses something like a persistent staging environment.
The real key is that we can fork that entire Linux VM to take different paths down our testing scenarios, and can run multiple of them in parallel. Tests may look something like:
new user signs up:
|- creates a todo
|- ...
|- ...
|- creates a list
The two nested tests then start from the exact same point, where the previous test left off, but can run in parallel. With enough hardware, the full suite will run as fast as the slowest branch of the test tree. When we switched away from our previous integration test suite to this (not E2E), our tests actually became faster because they share setup through the forking.
Sure. It's a bit hard to quantify because we need to run these on bare-metal machines and the unit cost is pretty high.
We run our test workload as well as a few other startups' that we have onboarded on one AWS ARM bare-metal machine at $1.7k a month. We don't saturate that machine fully either so I'm not really sure what the amortized cost would be. Certainly more expensive than Github Actions but not by a crazy amount, and the value we get out of it is way higher than GA.
Agreed, we had agents writing standard integration/unit/e2e tests but turned out to be a ton of work reviewing just that and hard to wrap one’s head around as a developer.
We ended up building our own test platform to reduce the human bottleneck in the verification loop.
Takes the form of a hermetic test environment that run all our code and infra in a way that matches production. Our tests interact with it from the ”outside”, e.g. by driving a browser, a terminal or making API calls.
We still have coding agents writing those tests but the big difference is that we record the browser and the API calls and can review and verify through a UI without having to do any manual testing. It also serves as a regression test suite with E2E coverage, which is kind of a must to avoid coding agents making assumptions and breaking things.
Not a huge fan of the term ”software factory” but working like this I actually think it’s achievable.
While doing local development where your environment is already setup and dependencies are installed doing something like this seems viable after the initial effort.
But how would this work in a software-factory setup, where coding agents run on fresh VMs or when using something like Claude Code on the web? How do you quickly recreate the required environment and dependencies there?
That’s essentially what we do. The agent writes the browser tests (and the other tests), we just review the replays of them afterwards from the test suite.
reply