The LLMs have been trained on all kinds of code. They often choose a hacky solution because that's what they encountered in similar situations in their training. It generally only takes a few words to guide them to a better design than what they saw in the wild, but if you don't guide them, they follow the herd.
It generally only takes a few words to guide them to a better design than what they saw in the wild, but if you don't guide them, they follow the herd.
That certainly can be true but I would challenge "generally". If you're working on something novel then it's not exactly unusual for even SOTA models to produce a first attempt that makes some very bad assumptions but then get stuck when instructed to use correct axioms instead. This looks a lot like the suduko meme - without any real insight into the problem the models end up just iterating around in circles and never finding a solution.
You probably won't see this very often if you're working on simple CRUD web applications where there are millions if not billions of lines of code in the training data set. IME you will probably see it quite frequently if you're working on - for example - software to control some kind of equipment or analyse the results of some kind of experiment where the LLM might have little if any training data about the field you're operating in.
Any time the LLM can establish a tight feedback loop it can iterate effectively enough to make good progress. But if for one reason or another that feedback loop is long, the more guidance and direction they seem to need. If they cannot correct their course based on feedback from their own output they need an outside observer to push them towards different paths. CRUD stuff is the obvious easy answer. It represents some absurd amount of software development work and is drastically overrepresented in training data. For good and bad. Lately I've been working on a diffusion model for terrain generation. The goal is to be able to recreate realistic terrain based on water flow and erosion on top of plate tectonics for much longer geological time terrain adjustments. These are much longer feedback loops. Sometimes it'll spend 6+ hours training a diffusion model on my 5090 just for the quality to degrade compared to the last experiment. So letting the model go wild trying different approaches and evaluating the results doesn't work nearly as well as when it can iterate 1000 times in those 6 hours while checking its own answers.
Any time the LLM can establish a tight feedback loop it can iterate effectively enough to make good progress.
Not any time. Your feedback loop must provide some direction so the iteration is at least heading towards an improvement. For novel applications you might not have a convenient way to do that yet.
Even with that provision there's still a possibility of incrementally updating output converging on a local maximum. That local maximum might not be a solution at all. If it is a solution then the model might still miss much better solutions that a real developer with more insight might find.
As you said yourself the models need external feedback if they don't have sufficient information to make adjustments by themselves. In some cases that feedback might be "Start again because you're hopelessly off track". Sometimes even when they start again they just get hopelessly off-track again. That can happen even if they are explicitly prompted to do something else. So even closely supervising the LLM and not using self-iterating agents isn't guaranteed to get good results for novel applications.
Maybe I'm overly pedantic, but in my mind a feedback loop involves both feedback and a loop. So by my reckoning the "feedback loop" must, by definition, guide the direction of the next step. I hold this true of feedback loops with zero LLM tooling involved. It's the same thing. The "entity" determining whether to continue the loop or not must have a good enough definition of success or improvement. Of course there is a lot of variability in how effectively the feedback is delivered. That's one of the variables the human setting up the loop is responsible for establishing in our brave new world.
You're right though that in new territory you may often have domain experts who still cannot adequately define success in an automated fashion because this is literally something that hasn't been created before. So they must be a part of the loop. They cannot necessarily put into code and descriptions what "better" or "right" is for an LLM to follow. But they can recognize it when they see it. In this case, it's still best to create the tightest feedback loop you can. If the LLM is waiting 2 days for your feedback, the loop is 2 days long.
I would say that a system with some sort of binary evaluation that indicates whether it succeeded or not is receiving feedback. It could also be some kind of sliding scale - how close did it get?
Either of those might be useful for determining a good future direction to iterate in but that would surely depend on the nature of the problem and how it's possible to search for solutions. I'd still call it a feedback loop even if the evaluation wasn't enough to point in a specific direction for a further iteration though. YMMV.
That's a great perspective. Come to think of it, I wonder about the quality of the build123d code it generated when I vibe-coded a prototype complex assembly. Now I'm curious.
