The Qwen3.8-Flash-Next model is ok for reverse engineering. It can definitely solve simple tasks, but it gets easily confused, and burns tokens like crazy.
(As an aside: I think representing tools for AI as virtual SQL tables is much better than traditional MCP servers, and that this approach deserves a lot more attention. Props to Elias Bachaalany for thinking out of the box and putting it together).
The results of the benchmark are below. It's sorted by cost, but I added notes about each AI model for myself. The notes try to talk about intelligence, speed, and cost. Intelligence is a subjective measure which I assign to each model by reading its thinking tokens and watching its tool calls. You can definitely tell which models are more intelligent, even on this small benchmarks, by their tool use, false starts, and how easily they fall for the simple misdirection in the crackme.
+----------------------------------+--------------------------------------------------------------------------------+-----------------+-----------------+-----------------+
| Model / route | Experience / outcome | Total tokens | Output tokens | Cost |
+----------------------------------+--------------------------------------------------------------------------------+-----------------+-----------------+-----------------+
| Xiaomi MiMo v2.6 Flash | Incredibly fast and intelligent; solved. | 225,291 | 3,281 | $0.005 (0.5c) |
| GLM 5.3 Flash / OpenRouter | Least intelligent group but cheapest; solved. | 340,006 | 11,205 | $0.01 |
| ChatGPT Luna Latest | Very fast, concise, token-efficient, intelligent, and cheap; solved. | 401,837 | 5,119 | $0.01 |
| Qwen 3.8 Flash | Initially confused and failed, then eventually solved. | ~418,000 | 16,676 | $0.02 |
| Qwen 3.8 27B | Fast; one wrong answer, then quickly self-corrected. | ~774,784 | 12,658 | $0.02 |
| DeepSeek Flash / direct API | Fast, cheap, and accurate; solved. | 369,334 | 9,089 | $0.03 |
| DeepSeek Flash / OpenRouter | Fast and intelligent; solved in 76 seconds. | 371,689 | 13,888 | $0.03 |
| Claude Haiku 5.5 | Fast and surprisingly intelligent, but overly thorough and token-heavy. | 662,346 | 26,244 | $0.03 |
| Thinking Machines Inkling | Low intelligence; good speed and relatively token-efficient; may loop. | N/A | N/A | $0.03 |
| Xiaomi MiMo v2.6 Pro | Overthought the task and wandered into tangents. | ~420,644 | 13,638 | $0.05 |
| GLM 5.3 / OpenRouter | Cheap and intelligent but behind DeepSeek; solved in about 2m 20s. | ~702,473* | 11,879 | $0.08 |
| GPT Sol Latest (Sol 6.1) | Slow, but highly intelligent and extremely token-efficient; solved in ~89s. | 294,957 | 1,406 | $0.15 |
| MiniMax M3 | Least intelligent comparison; many wrong turns and repeated small mistakes. | 4,546,552* | 116,804 | $0.20 |
| GPT Terra Latest | Very fast, very smart, and token-efficient; solved. | 229,888 | 2,342 | $0.20 |
| Mistral Large 4.0 | Medium speed; small mistakes but reached the solution. | 303,687 | 4,041 | $0.21 |
| Kimi K2.7 Code | Less intelligent, slow, expensive; not worth using. | 100,740 | 8,938 | $0.22 |
| Claude Opus 5.5 | Highest intelligence and token efficiency; pretty fast; cheaper than Sonnet. | 312,837 | 1,862 | $0.24 |
| Qwen 3.8 Max 0902 | Fast and intelligent, better than Kimi K3, but token-hungry. | 1,305,241 | 26,390 | $0.25 |
| Kimi K3 / OpenRouter | Faster and cheaper than direct K3, but still very expensive. | 750,064 | 5,922 | ~$0.32 |
| Meta Muse Spark Open | Intelligent and medium speed, but among the more expensive runs. | 872,850 | 10,284 | $0.35 |
| Gemini 4 Argon | Extremely fast and intelligent; strong value for capability. | 384,046 | 2,870 | $0.35 |
| Grok 4.7 | Very fast and intelligent, but expensive. | 655,648 | 4,217 | $0.46 |
| Meta Muse Glimmer 30B | Fast but low intelligence and token-inefficient; cancelled unsolved. | N/A | N/A | ~$0.50 wasted |
| Claude Sonnet 5.5 | Smart and fast but very token-inefficient and expensive. | 1,243,285* | 20,381 | $0.57 |
| Kimi K3 / direct API | Intelligent but not notable; extremely slow and far too expensive. | 205,710 | 3,215 | $0.72 |
| ChatGPT Astra Latest | Medium speed, token-efficient, highly intelligent; cost similar to Fable. | 278,603 | 1,536 | $0.83 |
| Claude Fable 5.1 | OK speed, very expensive, medium token efficiency; solved in ~53s. | 397,806* | 2,857 | $0.85 |
| Gemini Flash | Refused the reverse-engineering task. | N/A | N/A | Not reported |
| Thinking Machines Inkling Small | Got stuck immediately and was terminated. | N/A | N/A | Not reported |
+----------------------------------+--------------------------------------------------------------------------------+-----------------+-----------------+-----------------+
```
The latest Deepseek-Flash model was the most impressive of the open models. Intelligent, fast, cheap. What more do you want? I think the Xiaomi MiMo - Flash model did well, but it might be a fluke, given how poorly the Xiaomi MiMo Pro model performed (it wandered off on a million side quests).
What's also interesting is how competitive Luna is to the lower end Chinese models. It's quite a bit more intelligent, burns a lot less tokens, is plenty fast and very cheap. After doing this experiment, I don't think it's a foregone conclusion that the American AI labs will be having the bottom of the market eaten away by the Chinese models. It seems they (at least OpenAI) can compete on the bottom end as well.
Finally, the benchmark's results reinforce a few things people have been saying about Sonnet and Terra, namely, they're not competitive for a range of tasks. For easier tasks, Haiku/Luna are plenty capable. For harder tasks, Opus and Sol are so much more intelligent, that they can get them done quicker and cheaper. Maybe there is a place for them in the "Opus designs and Sonnet implements" workflows, but in that case, why not "Opus designs and Deepseek implements?"
PS Code harness used was Claude Code. $20 in openrouter credits was used to access the models. Kimi K3 and Deepseek were also accessed directly via the API and then compared to Openrouter. Kimi K3 specifically was much faster on Openrouter, and the difference in output tokens to get the same results for Deepseek/Kimi makes me think the Openrouter providers are hosting significantly quantized versions of the model. Goes to show that Moonshot.ai's acceptance testing of providers is a useful thing, otherwise your model might look artificially bad when it's hosted as heavily quantized by providers who dont care about quality, just top-line cost and speed.
When running benchmarks mind how some providers lobotomise some models. For example DeepSeek Flash is fine when run via DeepSeek. The same model via Deepinfra and open router (despite setting routing to "exact") was far worse.
Regarding Qwen3.8-Flash-Next only Alibaba matches the quality I get when run local inference.
Mind I run mxfp4 and int8 kv (before I run unsloth's q4_xl). And it is much more capable than providers that claim full precision. I have no idea what they do to mess it up, maybe they quantize the kv?
Thanks for the data-centric perspective. It reinforces my vague impression that DS 4.1 Flash eclipses almost everything out there, or at least ties it. We're really lucky to have that one.
I feel like I’m getting good value out of my subscription and they have the cheapest GLM-5.3 API rates I could find in Europe. I moved to them recently from Mistral for the better prices.
They do have a slightly different daily inference capacity allowance, but it’s actually not too bad once used to it. I was forced to up my game and figure out what GLM-5.3 Flash and Deepseek-4.1 Flash could actually do for me, rather than just throwing everything at GLM-5.3. Quite happily chugging along with a nice balance across them now.
I am hoping Melious make the new DS 4.1 Flash 1017 available soonest.
I think you're on to something. It's really concerning that the people with the money to run these ad campaigns are so removed from the bulk of the populace that they don't even bother targeting them with all the billboards anymore, just each other.
Otoh maybe I'm looking at it wrong, and maybe billboard advertising has gotten so ineffective that it's cheapened to the point that you can tuck it into your campaign as an afterthought. If you get one solid enterprise lead from it, it pays for the whole year. And besides it has branding value.
Either way, definitely feels a bit dystopian to me. How alienating must it feel for the service worker commuting to his dead end job where even the billboards don't make sense anymore?
This tracks, in my experience the 27B is better at coding and instruction following. I'm shocked at how much of a difference the dense models vs MoE makes.
But it's a moot point, because for local inference on consumer hardware, the MoE is so much faster.
My mistake, I just assumed their Byfron anti-cheat is kernel level because that's the most common reason why anticheats take issue with Wine. If their anti-cheat operates entirely in usermode, then their decision to block Wine is even more baffling to me.
It is not kernel level, they use a number of pretty fancy user mode tricks to prevent people messing with the game.
They encrypt every page and register an exception handler that decrypts the page when a fault is reached to keep memory secure. They do cache timing checks to see if a page was recently in the working set outside of the expected control flow and will flag you if you fail the check too many times because it indicates tampering.
While the page decryption trick should work in Wine just fine (ZeroIT Lab Theia does that too, right?), I don’t remember if Wine provides workingset info like Windows.
Yes, Wine was (EDIT: partially) compatible with the Roblox anti-cheat, and there was a period in time where Roblox with its anti-cheat ran on Wine. Eventually, however, a decision was made to block Wine.
EDIT It looks like the Roblox anti-cheat devs themselves explained why Wine was blocked:
Yes, it does. Wine lets you register a VEH. They do other stuff to prevent Wine and virtual machines in general that I am not fully aware of.
Even then it's just a fancy party trick, you can suspend the process call the decryption function enumerating all the pages and take a full dump to get the binary
FYI; Theia is a very bad product. It absolutely tanks binary performance and does little to prevent a moderately competent engineer from cheating in your game.
Are the other Zero IT Lab products like their code virtualizer of similar quality to Theia? My only exposure has been that one RE//verse presentation and shitposts on UC.
I can't speak for all the products, but from what I have heard it's pretty bad. Pay close attention to the shitposts on UC, they regularly get taken down but have some interesting information in them.
I've been sick with stomach flu the past few days, so I put together a small benchmark for all the top AI models where I ask them to reverse engineer a very simple small crackme (https://www.crackmes.one/crackme/6ab4ce5195b976f8f1300b54) using Ghidrasql (https://github.com/0xeb/ghidrasql)
(As an aside: I think representing tools for AI as virtual SQL tables is much better than traditional MCP servers, and that this approach deserves a lot more attention. Props to Elias Bachaalany for thinking out of the box and putting it together).
The results of the benchmark are below. It's sorted by cost, but I added notes about each AI model for myself. The notes try to talk about intelligence, speed, and cost. Intelligence is a subjective measure which I assign to each model by reading its thinking tokens and watching its tool calls. You can definitely tell which models are more intelligent, even on this small benchmarks, by their tool use, false starts, and how easily they fall for the simple misdirection in the crackme.
(gist is more readable on mobile) https://gist.github.com/markoglasgow/8812c19b6fbffed30b2661b...
```
```The latest Deepseek-Flash model was the most impressive of the open models. Intelligent, fast, cheap. What more do you want? I think the Xiaomi MiMo - Flash model did well, but it might be a fluke, given how poorly the Xiaomi MiMo Pro model performed (it wandered off on a million side quests).
What's also interesting is how competitive Luna is to the lower end Chinese models. It's quite a bit more intelligent, burns a lot less tokens, is plenty fast and very cheap. After doing this experiment, I don't think it's a foregone conclusion that the American AI labs will be having the bottom of the market eaten away by the Chinese models. It seems they (at least OpenAI) can compete on the bottom end as well.
Finally, the benchmark's results reinforce a few things people have been saying about Sonnet and Terra, namely, they're not competitive for a range of tasks. For easier tasks, Haiku/Luna are plenty capable. For harder tasks, Opus and Sol are so much more intelligent, that they can get them done quicker and cheaper. Maybe there is a place for them in the "Opus designs and Sonnet implements" workflows, but in that case, why not "Opus designs and Deepseek implements?"
PS Code harness used was Claude Code. $20 in openrouter credits was used to access the models. Kimi K3 and Deepseek were also accessed directly via the API and then compared to Openrouter. Kimi K3 specifically was much faster on Openrouter, and the difference in output tokens to get the same results for Deepseek/Kimi makes me think the Openrouter providers are hosting significantly quantized versions of the model. Goes to show that Moonshot.ai's acceptance testing of providers is a useful thing, otherwise your model might look artificially bad when it's hosted as heavily quantized by providers who dont care about quality, just top-line cost and speed.
reply