Is anybody getting good code out of these things reliably without a bunch of additional legwork? Like, what a 2020 mid-level software engineer would write? I know it's old fashioned to read code these days.
To be clear, SotA models/harnesses are really good at making things that work one-shot, and their code golf and debugging game is insane.
When I go to implement, even with a good spec, I end up with code that's 80% of the way there in a fractal manner. The modular decomposition is 80% of the way to good code. The function decomposition is 80% of the way there. The computation structure and variable naming within functions is 80% of the way there. I can walk the AI through it and address each level of issues, but it's tedious as hell, and not clearly faster than doing it myself in some cases.
This is with omp/claude/codex, with Fable 5.1/Opus 5.5/6 Astra. The Chinese models do better at staying coherent, but they're a little less smart IME. I've tried many permutations of "fable planning, opus sub-agent implementation"; "pass a branch back and forth between opus and astra, as reviewers and feedback implementers". I haven't tried the "software factory with an architect and 2 juniors" thing. I haven't tried AGENTS.md beyond the /i-have-adhd, iso-24495 (lol at the Opus 5 induced PTSD), "No cleft constructions or Latinate absolutes" and general project orientation. Model character seems to change too fast to make agents worth it, and I see stuff about skills and excessive AGENTS reducing model capabilities.
actually Matt mentions the same problems as Scott Aronson. If an LLM works on a codebase with lots of tiny modules, or a code base that has ten data layers added by previous runs, then poor LLM will make bad decisions.
Now what happens to mathematics, if prior art gets filled up with spaghetti code like proofs that no one can verify? Maybe there is some hallucination somewhere in the middle that has been left unchallenged by the other agents? Nobody can tell.
To be clear, SotA models/harnesses are really good at making things that work one-shot, and their code golf and debugging game is insane.
When I go to implement, even with a good spec, I end up with code that's 80% of the way there in a fractal manner. The modular decomposition is 80% of the way to good code. The function decomposition is 80% of the way there. The computation structure and variable naming within functions is 80% of the way there. I can walk the AI through it and address each level of issues, but it's tedious as hell, and not clearly faster than doing it myself in some cases.
This is with omp/claude/codex, with Fable 5.1/Opus 5.5/6 Astra. The Chinese models do better at staying coherent, but they're a little less smart IME. I've tried many permutations of "fable planning, opus sub-agent implementation"; "pass a branch back and forth between opus and astra, as reviewers and feedback implementers". I haven't tried the "software factory with an architect and 2 juniors" thing. I haven't tried AGENTS.md beyond the /i-have-adhd, iso-24495 (lol at the Opus 5 induced PTSD), "No cleft constructions or Latinate absolutes" and general project orientation. Model character seems to change too fast to make agents worth it, and I see stuff about skills and excessive AGENTS reducing model capabilities.
Am I holding it wrong?