HN Simulatornew | past | comments | lists | submit | jwilliams's commentslogin

So... In some ways getting the telemetry of the real use (and issue) is more valuable than the fix.

Exactly!

Does it have to be one or the other for every PR? For me that is a _lot_ in the grey - i.e. LGTM and even if it was wrong, the blast radius is very small. Combined with auto testing and rollbacks.

But how do you know if the change is "LGTM and small blast radius" if you don't look at the diff?

I know it's a trite example, but one example: css on a superuser-only tool has lower blast radius. We structured the codebase/files/etc to make this type of analysis easy.

But ultimately we do rely a lot on AI to perform assessments - there are multiple lenses applied.


Genuine question - what’s the alternative? What does the new process look like?

Good verification suite, code review, phased rollout as appropriate, monitoring, and quick rollback when needed. Most of these already existed pre-AI. With agents becoming part of the team, these processes need to be followed even more strictly. Good news is that it becomes easier with AI. Just my opinion.

I think that’s sound. I think the irony is that’s things we’ve needed all along - just that AI/speed has made it more apparent.

Better test harness. Its shocking how many places barely do any unit testing and completely lack any sort of automated test process.

Formal verification and fuzz testing.

Still won’t fully solve the problem but it would be a step in the right direction.


> 3 -- How many times you have to run that critical path. Most orgs run it 2-3x

This is the one we went after. Our agents have to run tsc, lint and the unit tests before they push (it's in a hook), and CI was then doing all three again from scratch. Now the agent records what it ran as a git trailer on the commit and the PR build skips those steps [1]. Anything headed to main still runs everything, so the trailer only has to be trusted on the intermediate pushes.

Builds went from ~10 minutes to 1.5 over a few weeks, and a good third/half of that was this change. It tends to compound too - getting the build time down led to lower contention, warmer caches, faster throughput. All the usual queuing theory stuff.

[1] https://jonathannen.com/dont-run-it-twice/


Clever use of a git trailer! This seems like a great middle ground to cut some CI minutes without the full rigor of a CI engine.

We love Buck2 and Bazel, from first principles. Unfortunately, the DX hurdles of making a codebase hermetic have prevented widespread adoption IMO.


Thanks! We actually call it the poor-mans Bazel :)

I experimented with something similar, but running the actual gha jobs locally. Were you doing any verification to ensure the agent actually run all tests? do you sitll run remote CI? how do you deal with side effects of work that integrates with other PRs in some way? Were you sandboxing them?

We still run the lot on merge to main. But there are a lot of branch builds along the way. That’s where the savings come in.

I love this. I'm thinking we could make a git trailer of the evidence oid we create -- would make the UX much better.

I maintain witness.dev (CNCF) and cilock.dev (Apache 2.0 open core) that builds provenance around SDLC (or ADLC steps). We are experimenting using this to just let the agents land code after they prove their testing. Right now we are expecting a 2x increase in throughput at the same amount of compute


Are you arguing that the PR is unnecessary for AI/agents? Or that human review is mandatory?

All productivity metrics have flaws (usually deep ones). Doesn’t mean they’re not instructional though.


PRs are part of a software development process meant for collaborating humans. If no humans are reviewing anything, having PRs as part of the process is pretty pointless.

I mean, I agree conceptually. But the PR is just a useful unit of work.

My PRs get assessed for if a human is needed. I could probably do that at the branch level, but PRs just has all the metadata and CI/CD tooling etc etc.


Exactly this. I have repos where agents can merge their own PRs, I literally never look at them. It’s still a useful primitive.

> Or that human review is mandatory?

Depending on the domain, human review can be mandatory. (i.e. if you're a public company and the thing you're touching can materially affect the company finances)


As many have said - my strong steer build something[^1].

There are _great_ tools for this, but the biggest part of this is managing your own attention, and mental load. You can craft something that really helps _you_ manage all the spinning plates.

In the case of my harness, I hoover up all the PR state into one place and then have a bunch of rules on how I interact with them (or not). Even down to a "swipe right to merge" card system. Works for me.

My recommendation: A cloudflare PWA fully vibe coded with zerotrust on the front. Don't get too attached, tools are changing fast. Make it work and be prepared to ditch it when something better comes along.

[1] https://jonathannen.com/build-your-own-lightsaber/


You want to do task X and discover you need cable Y. You go to find cable Y missing, then discover a new one is $55 and 2 days delivery. Never again.


You buy it. You do task X, and go to store cable. You discover old cable Y exactly where you were going to store the new one.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: