Does it have to be one or the other for every PR? For me that is a _lot_ in the grey - i.e. LGTM and even if it was wrong, the blast radius is very small. Combined with auto testing and rollbacks.
I know it's a trite example, but one example: css on a superuser-only tool has lower blast radius. We structured the codebase/files/etc to make this type of analysis easy.
But ultimately we do rely a lot on AI to perform assessments - there are multiple lenses applied.
Good verification suite, code review, phased rollout as appropriate, monitoring, and quick rollback when needed. Most of these already existed pre-AI. With agents becoming part of the team, these processes need to be followed even more strictly. Good news is that it becomes easier with AI. Just my opinion.
> 3 -- How many times you have to run that critical path. Most orgs run it 2-3x
This is the one we went after. Our agents have to run tsc, lint and the unit tests before they push (it's in a hook), and CI was then doing all three again from scratch. Now the agent records what it ran as a git trailer on the commit and the PR build skips those steps [1]. Anything headed to main still runs everything, so the trailer only has to be trusted on the intermediate pushes.
Builds went from ~10 minutes to 1.5 over a few weeks, and a good third/half of that was this change. It tends to compound too - getting the build time down led to lower contention, warmer caches, faster throughput. All the usual queuing theory stuff.
I experimented with something similar, but running the actual gha jobs locally. Were you doing any verification to ensure the agent actually run all tests? do you sitll run remote CI? how do you deal with side effects of work that integrates with other PRs in some way? Were you sandboxing them?
I love this. I'm thinking we could make a git trailer of the evidence oid we create -- would make the UX much better.
I maintain witness.dev (CNCF) and cilock.dev (Apache 2.0 open core) that builds provenance around SDLC (or ADLC steps). We are experimenting using this to just let the agents land code after they prove their testing. Right now we are expecting a 2x increase in throughput at the same amount of compute
PRs are part of a software development process meant for collaborating humans. If no humans are reviewing anything, having PRs as part of the process is pretty pointless.
I mean, I agree conceptually. But the PR is just a useful unit of work.
My PRs get assessed for if a human is needed. I could probably do that at the branch level, but PRs just has all the metadata and CI/CD tooling etc etc.
Depending on the domain, human review can be mandatory. (i.e. if you're a public company and the thing you're touching can materially affect the company finances)
As many have said - my strong steer build something[^1].
There are _great_ tools for this, but the biggest part of this is managing your own attention, and mental load. You can craft something that really helps _you_ manage all the spinning plates.
In the case of my harness, I hoover up all the PR state into one place and then have a bunch of rules on how I interact with them (or not). Even down to a "swipe right to merge" card system. Works for me.
My recommendation: A cloudflare PWA fully vibe coded with zerotrust on the front. Don't get too attached, tools are changing fast. Make it work and be prepared to ditch it when something better comes along.
reply