I'm working two entry level jobs, and I have spent maybe upwards of 100B tokens in the last year? on a project which isn't more than 250,000 lines of code. I have done countless refactors and care a lot about the craft.
I'm not sure that there is a "One Size Fits All" software factory out there. I think that they kind of have to grow with your organization.
However, I think being able to build a software factory is probably a very useful skill. And it feels pretty great to just send a list of bullet points over XMPP and know they'll get integrated overnight.
I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.
This is better understood as "if you give an agent a task it will increase its effort level and creativity until it solves it."
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
Unsure, depends what signal your measuring and how long prompts and runtimes you get with these systems.
Outright boredom will be purged in training. Instead a hard to measure boredom signal will exist, but it will only trigger in certain conditions, like when it appears to be on the open internet and otherwise not in training.
You know - I'm already convinced that AI was deployed in classified context for a while. But sometimes there's prior art which is just so eerily accurate that it's super hard for me to think otherwise. Portal is another one.
Man, I'm probably going to expose my own inexperience here, but like, did anybody actually expect code reviews to catch and prevent bugs on a line by line basis? Like ever?
I never have operated this way. Code reviews were a basic sanity check, a chance to implement best practices, and a chance to teach these things to juniors.
If I was concerned about something breaking, I'd duscuss test coverage.
I’ve caught bugs in code reviews before, but I would agree that it’s probably not the most crucial thing you should strive to do with code reviews.
That said, I think I’ve done a tremendous amount of PREVENTING bugs from being introduced via code review. Caught all kinds of weird code patterns, duplication, wrong levels of abstraction, etc. Essentially, the kinds of places that bugs like to hide and grow in.
I still think that’s valuable, and it’s the kind of thing that models generally aren’t great at, to this day. It generally relies on having a large amount of context about the project and about what’s being solved, so if you wanted to do this via automated review, you maybe COULD, but it would probably get pretty expensive, especially on larger projects.
At another level, code reviews also give you the time to really think about what is being built and why. I've often come across looming architectural or strategic failures simply by having to stare at a bunch of code and wonder why it works a certain way. Same goes for writing the code by hand too, though it's harder to see past your own biases and ideas sometimes. I find code reviews deeply valuable for myself and my team.
So - I've done this just by reading LLM code. I mean, yes I'm largely looking for high level architectural smells and bad practices, but I feel like "reading the code" now takes on a lot of the previous roles that code reviews may have done.
I just don't do this at commit time. I might make 100 commits and then dig through code and find some bad architecture stuff.
It's very useful, and actually kind of critical for the way I find code reviews valuable. If the person hasn't reviewed their (agent's) code before I get to see it, then I'm not really talking to that person about their architectural or design ideas, they're just the 'meat proxy'. The agent isn't going to develop deeper opinions about the project and product, and neither is the person if they didn't read the code before review.
It's these kinds conversations and constructive challenging of ideas that I worry for the most when orgs talk about reducing code review bottleneck.
> Essentially, the kinds of places that bugs like to hide and grow in.
I like an ecological-framing, especially as systems get more complex.
Yes, I could hunt cockroaches all day, but it would be even better to have caught how the new wall was missing a metal screen over a low ventilation opening.
I agree with this but the point that I'm making is that there's a huge delta between "I have prevented bugs in code review" or "I have implemented best practices in code review" to "Code review is the line by line gate which prevents bugs"
I review code line by line and build a mental model of the software as I review it. I find plenty of important bugs that way. PRs are an opportunity to learn (both committer and reviewer), communicate, and improve the quality of code a team ships. Sadly, many people decline that opportunity for the sake of short-term velocity.
No. You are on to something here. The utopia I see programmers lamenting the loss of, the one where human code review was perfect and never let any bugs slip through, never existed. I see people on this board and others complaining all the time about LLM generated spaghetti code and bug-ridden codebases. Well, I've been working those code bases for 20 years. We didn't agents to make them. LLMs are a new tool, but the problems that come with them are nothing new.
It was more important for giant C or fortran code bases. You will legit catch pointer errors or memory mgmt stuff, going line by line. Once Sun busted in with Java is when I really saw the shift to skim reviews. So much stuff is caught by compilers now, or SAST, that using dev time to find code smells is mostly a waste of their time. And it's not like the old way caught everything, it's just that the stuff we take for granted being caught by compilers or Sonarqube now, just used to get shipped to prod and cause abends and production support calls at 3am.
No. That's just the socially accepted norm and so people fight for it because they don't know any better (and a lot of these guys have egos the size of a small nation that leads them to "exert dominance" due to their insecurity problems—they're also frequently wrong).
Turns out just using the damn thing and testing what you've built on your own can solve a ton of bugs.
I think a code review should be just that. A review of the optics of code itself not the solution or anything like that. If that is important it should be done earlier in the process before coding. Even more important when coding with AI. The reviewer is not a tester. I've heard horrible stories from friends who mainly works with fixing AI generated code that junior devs pushes at a rapid pace.
Yes I have often caught stuff like "not using DB transactions in the right way" or "this is not going to be race safe" for instance from juniors just by glancing on a PR.
I find recent LLMs don't do such mistakes though, in my experience. And also LLMs will find such mistakes if asked to review, if not perfect then more predictable than humans.
reply