That is interesting, but... is it actually improving structure?
We're mid-way through a similar process at work. Rewriting a legacy app in a new language, with new architecture and new features.
And it's a mess.
We're at 10x loc (admittedly, the new programming language is more verbose than the old one), comments make no sense. Yes, we have ~100% coverage, but most of the tests are meaningless. The agent keeps removing our tests to replace them with tests that are easier to pass, breaking code invariants, removing all the engineered data structures and replacing them with stringly-typed code, etc.
And of course, given the number of LoC (and the fact that the agent rewrites so much code all the time), it's physically impossible that all of them were reviewed by a human being.
AI made it possible, insofar as upper management would never have greenlit the project without AI, but I can't escape the feeling that we're building on quicksands.
In my experience, when you attempt something like this it grows like a crystal. If you start badly, with poor tests and poor architecture, that is the code that has the attention of the model going forward. More than any rules you give it, what matters to the model is all the code that’s already there. I’ve found myself in such a situation from going to fast and expecting new hyped models to “just take care of it”. The solution was to stop and take stock. I ended up deleting most of the dead-end and using the models to refactor the rest. From that point on, new code from the models was at the standard I needed, following rules already expressed in the code base.
If your code base is script-kiddy garbage, your models are only going to keep writing more garbage. You should throw it all away. I expect that isn’t an option. Try instead instructing the model that the code you have is legacy garbage written in a hurry that needs siginificant refactoring, but that right now I need one new feature written to these new standards instead. Take it one small piece at a time. Don’t let context go beyond 100k. Good luck!
10x loc? wow. That makes me wonder how it will perform. And how you will review that, since if it is physically impossible for a human then who will take responsibility for what you release?
>> That is interesting, but... is it actually improving structure?
Can you be more specific? "Improve structure" can mean different things to different people.
We've ensured that agents strictly adhere to code architecture rules, using both agentic review rules and deterministic CI gates. Everything from file naming, location and namespacing to the "shape" of each file that shares the same role are consistent. We can trace functions of the same role across the same set of modules, with abstractions and code reuse where they make sense. Once someone learns the code in one part of the codebase, their understanding of that "spine" translates to other areas - the only differences relate to the specific business rules governing those areas.
>> And of course, given the number of LoC (and the fact that the agent rewrites so much code all the time), it's physically impossible that all of them were reviewed by a human being.
Code review was difficult at the start due to the volume of code. One insight we gained midway through though is that the value humans bring to code review is judgment and business context. So we created a code-atlas skill that creates an artifact for PR reviewers. That artifact highlights the most important parts of the code. For example, if a PR author has made the choice to use soft-deletes when the product uses hard-deletes for everything else, that is flagged by the code atlas, and a human reviewer can use that to ask questions about it. Meanwhile, things like boilerplate are de-emphasized; any problems associated with them are reliably found and flagged by AI reviewers anyway. The refreshing part about this is that this has almost completely eliminated bike-shedding: people no longer argue about module naming or whether a comment is worded properly.
> Code review was difficult at the start due to the volume of code. One insight we gained midway through though is that the value humans bring to code review is judgment and business context. So we created a code-atlas skill that creates an artifact for PR reviewers. That artifact highlights the most important parts of the code.
Yeah, we did that, too.
But every time I end up, for some reason, digging up deep in the code, I realize that it's not nearly sufficient in our case.
We're mid-way through a similar process at work. Rewriting a legacy app in a new language, with new architecture and new features.
And it's a mess.
We're at 10x loc (admittedly, the new programming language is more verbose than the old one), comments make no sense. Yes, we have ~100% coverage, but most of the tests are meaningless. The agent keeps removing our tests to replace them with tests that are easier to pass, breaking code invariants, removing all the engineered data structures and replacing them with stringly-typed code, etc.
And of course, given the number of LoC (and the fact that the agent rewrites so much code all the time), it's physically impossible that all of them were reviewed by a human being.
AI made it possible, insofar as upper management would never have greenlit the project without AI, but I can't escape the feeling that we're building on quicksands.