Frontier models have seen more Mathematica and Powershell code than I ever have in their training, yet they really struggle to produce working output. They seem heavily tuned to Linux too. Despite adding skills to rectify this, they still fail to realize they're running on Windows and waste tokens. A human with this much training wouldn't have this problem. There are evidently still some pretty big holes still.
This is probably a harness problem rather than a model problem. GitHub Copilot will happily and effectively use Powershell while Claude Code struggles in my experience.
I have spent a small fortune in tokens trying to get Anthropic models to attempt proofs of things like the irrationality of the Euler-Mascheroni constant, and to make lattice reduction / SVP algorithms that rival SOTA methods like the G6K sieve, and all sorts of difficult frontier problems. It says 'honest' as a euphemism for giving up, because it knows I'm being a crank, and because the training data contains a lot of pessimism.
But I can get around this by first having it write a prompt for itself discussing the problem followed by a rousing speech about never giving in and admitting defeat, but pressing on towards results regardless of how hard matching SOTA results is.
