So nobody's bothered by the fact that five out of nine models went nuke in 100% of cases regardless of the language?..

Yeah that’s why they delegated code gen to deterministic tooling and saved the model for input fuzzing - let it fight the data instead of legacy syntax

For the article they just mocked it out for unit tests. But in reality you really can't - that’s a massive pain in the ass. Rewriting pure math is easy, but mocking CICS transaction isolation in java means spinning up these monstrous adapter frameworks that just tank performance

Obviously without a proper RAG pipeline or web search it is just going to hallucinate with maximum confidence. It would be more interesting to see the results with top tier models that have proper alignment to refuse to answer when token probability is low. As it is they just proved that people tend to trust well written text in a chat ui

It would be interesting to measure not just the accuracy, but how many people actually decided to double check the answer