That is...not at all my experience of Opus 5.5, which has absolutely both spotted weird workarounds, then spotted the underlying bugs which were leading to the workaround, then spotted the race conditions which were masked by the workarounds.
It's hard to say because no one actually describes what and how they are doing to get AI to successfully refactor or operate in large code bases.
My experience is, slop on slop. Frontier models still can't handle even modest sized refactors without explicit hand holding. I still don't trust models with production scale refactors.
Apparently, everyone has the magic sauce and I'm blind. Or our definition of quality and production grade is very different.
Some misc advice. Your first task with ai is to use it to build constraints and guard rails for the ai.
That probably means a test suite that is at least twice the size of your code base. If you started there, great, but increase it to integration and benchies if you’re mostly holding units. It actually helps a bit if you can use a different language for this too but it depends on project, ymmv.
After that you have to either let go of certain style/idiom things or enforce ruthlessly with deterministic hooks everywhere.. no middle ground there and prompts will never work.
You also need to deliberately channel chain of thought with workflows rather than hoping for the best.. most of the generated product here shouldn’t be committed or preserved at all.. non-public doc flows and ticketing systems help.
Like people, an ai can easily create code that it cant actually maintain. It’s a matter of discipline now, same as before. Always using the smartest model is wrong, but double wrong on “wide” changes like a big refactor.. it leaves you no good options if you get into trouble.
> That probably means a test suite that is at least twice the size of your code base.
If you've got a test suite that's twice the size of the code base then those tests are almost certainly tied to implementation details and will make the code a lot less like clay.
Is this always true? I'm honestly not sure. For unit tests I completely agree with you, for integration tests I'm not so sure—validating permutations does not necessarily couple you to code, and may in fact be good discipline for user expectations.
AI absolutely solves that problem. Just about every prompt "fixes" the dried out pieces of the code. And it's only getting better at that every month.
Sometimes I feel like people are experiencing AI so differently that it makes talking about it nearly impossible.