This is a case where we basically did test it on everything! Because the emulation community maintains conformance tests with over 3k of golden instructions. (And this is why Opus was able to so effectively build it; it found the tests)
Yeah, but that still doesn’t mean there aren’t theoretical bugs. Testing every input would effectively mean testing every stimulus, which is combinatorially impossible. You might still have subtle timing bugs, bugs with very uncommon instruction sequences etc.
But my point was exactly that the (extremely vastly) smaller subset of tests that you performed is more than enough to proclaim a CPU to be “working”.
First off, Opus 5.5 built working weights and code with only two prompts:
1. Read this article and any relevant source code: https://divagr.com/blog/qwenputer. Then explore if you could emulate the chip and ram in a Nintendo Entertainment System (as people do today with FPGAs in MISTRs) using this technique. If so, what size model would you have to build and if you could use smaller faster ones, could you get it to be at the standard clockspeed.
It returned charts well illustrating any model would be _slow_. But it's still fun, so the second prompt was:
2. I think we should build it! Just for fun. It's okay if it's not fast. We want to be able to have the weights, and a cli to run them that we can point at a ROM.
Now while this was an _epic_ run by Opus (5.5 is a crazy capable model), this is a surprisingly well-structured challenge.
There is a long-lived and incredible community of classic gaming emulation work online for it to reference. And that community has created…conformance tests! Opus found them and built against one, that has >3k golden instructions: https://github.com/christopherpow/nes-test-roms/blob/master/...
Also, there is the original Qwenputer repo that Opus used to drive the weight writing. That was essential.
But this is a great example of what these models can do with proper reference materials, great specs, conformance tests, and clear direction.
> There is a long-lived and incredible community of classic gaming emulation work online for it to reference. And that community has created…conformance tests! Opus found them and built against one, that has >3k golden instructions
I'm doing something similar but not for an emulator, using tests other projects in a similar space use to validate things.
I also know a friend that feeds Claude Gherkin formatted test criteria, and apparently that yields insanely good Claude output.
And then once you get over the knee-jerk reaction the twitter community been teaching you, you might look at the actual syntax and the way the language conceptualizes stuff, and might actually learn something new and useful :) Or, at least have something constructive to say, even if it's negative or whatever, instead of just wasting space.
You're making an assumption here. I don't read Twitter, I have no idea what reaction that community has and I don't care.
My insight on Cucumber comes entirely from years of experience, through several organizations using or considering using it. Several times I was successful in stopping its planned use.
I thought the negative comments would come through the sarcasm, but point taken that it's not constructive.
I think Cucumber (Gherkin) and any natural-language attempts are rooted in a core delusion that product managers or non-technical people will write these test cases. I've never seen product managers writing or even reading these cases. They are not written at the functional specification or an SRD level, because that level is too high and too imprecise to capture the realities of testing behavior.
The natural language of Gherkin is parsed by a set of rigid regular expressions, which do not support the full richness of the English language, and subtle changes like adding or dropping an article or lightly altering the word order will render those steps unparsable. So it's not natural language, it's a programming language with all the syntactical strictness of one, but highly sugared and natural-looking.
You can write behavioral tests at the same exact level as Gherkin in any reasonably-expressive language, likely whatever language your code is written in. You can choose to have fixtures, and high-level structure and you can banish low-level detail from your test cases. Code can do all that, and MAINTIAN EASY refactorability. Code has natural ways in which it can create new layers of abstraction over repetitive ideas, at whatever level is convenient.
But with Gherkin, you're faced with "now I have to write new steps that call the implementing code of the old steps or figure out if I can call the old steps directly and have them still work in a sequence". It is no longer a matter of using your "Extract Method" or copy-pasting some lines into a method with parameters.
Because Gherkin introduces so much more friction/resistance to refactoring, those feature files are much less likely to be refactored. Thus, the copy/pasting of blocks of steps will continue and the tests will start losing that higher level expressiveness that was the justification for using Cucumber.
Testing behavior is a good idea. Writing your tests at a very high level, where you can understand the whole scenario at a glance, and give it data tables to consume, and follow that Given/Then structure, is all possible with code.
GLM 5.2 was the first open model that I was happy to use as my main. It's a fantastic model. 5.3 cost a lot more per task, was slower and didn't get that much better results for me, so I tried 5.3 Flash for a bit, it was ok, but didn't feel as magical as 5.2 and I have now landed on DeepSeek 4.1 Flash as my main.
It's not just you-- the claudisms in 5.3 are pathetic and drive me crazy, I was very sad to see it go in that direction and consider it a big step down in that respect.
BUT OTOH the performance of 5.3-flash are impossible to argue with-- it has the same pathetic claude behavior full of "Honest" and "lever" and "load bearing" tripe... but it's scortchingly fast on hardware that is pretty slow running 5.2/5.3.
As a result I use 5.3-flash a lot, in spite of feeling kind of gross about its language. I don't enjoy using it the way I enjoyed GLM-5.2, but its pretty effective.
Wrote about this awhile ago, and it hasn’t changed:
> Time and time again, when talking to people who rely on ChatGPT, Claude, Perplexity, and other general AI tools, I hear them say, “AI is incredible. It handles nearly everything I throw at them.”
> “What does it fumble with?” I’ll ask.
> “Well, it still gets things wrong when it comes to my line of work.”