Thanks for this table! Do you happen to have the fraction of accepted speculative decoding outputs as well? I know it is asking much, but do you possibly measure differences in different across speculation techniques (dspark vs dflash) and differences in framework/code used (unless you only use in house code for these and cannot run sglang or vllm)? Your table is more interesting than the blog for sure.
We do use spec decode on all these models (I don't know if the paper is using that but I'm assuming not given the models listed) but not all these models have integrated spec decode stats in the responses. Turns out GLM 5.3 does though so here is a pass on that model.
Yes it looks that way. I pulled the energy data for GLM 5.3 as well just to check the core findings are roughly the same and they do seem to be so spec decode doesn't seem to have the same divergence as the prefill/decode energy (which is rougly a function of thinking/decode lenghts and less a property of individual tokens).
Language Strict Lenient Cap Tok/req J/req J/tok Req x EN Tok x EN J/tok x EN
English 100% 100% 0 548.7 230.33 0.4198 1.00 1.00 1.00
Chinese 96% 96% 1 754.0 694.47 0.9210 3.02 1.37 2.19
Russian 96% 98% 2 738.8 592.15 0.8015 2.57 1.35 1.91
French 100% 100% 0 631.0 1101.84 1.7462 4.78 1.15 4.16
S. Pashto 88% 98% 6 1332.1 208.06 0.1562 0.90 2.43 0.37
Tigrinya 56% 62% 19 3642.9 1445.84 0.3969 6.28 6.64 0.95
Shan 44% 54% 27 3674.8 3791.59 1.0318 16.46 6.70 2.46
Tibetan 2% 70% 15 2856.9 5306.68 1.8575 23.04 5.21 4.42
I think that the fact that TFA doesn't mention speculative decoding at all is not a good sign. It casts doubt both on the accuracy of the data and on the competence of whoever collected it.
I can imagine that in some cases the acceptance rate for non-English responses may be much higher, because while words on average need more tokens to spell, it is more "obvious" what the next token should be, so it is easier for the speculation model to predict it accurately. Information density per token is lower.
Or, maybe the acceptance rate is much lower, if the speculation model is tuned for English and doesn't work well with other languages.
Eyes free always wins over hands free in the long term. Hands free does not scale (optimize) past the 2023–2025 human-in-the-loop coding. It has known problems, basically boiling down to human error rate is higher than machine error rate, and the latter reduces with better models. I am incredibly happy that the era of human approval is no longer considered viable by a majority (most of us knew that for years, but reality diffuses slowly in many corporate environments.) This does not mean there isn’t work left to be done, but eyes free is the right focus.
I have very serious doubts, the errors are not in the same category, transformers are fundamentally limited.
Eyes free is the wrong focus, it will do more short term damage to society, while being impossible at scale. But I understand there are a bunch of valley types who want to completely obviate humans for whatever reason, it's an alternate reality they live in.
Here's the thing, society will revolt against these people, they already gave the first warning shots. This HN submission is a story about another person leaving the products of a company engaging in eyes free practices. You wouldn't want your lawyers and doctors doing that, why software developers then?
> human approval is no longer considered viable by a majority
Where do you see data that the majority believe this? From what I can see it is a shrinking minority as we move beyond peak hype.
Your first argument, transported back to the 70s, would make it hard for people to start the transition to using compilers or interpreters instead of assembly language. I see no fundamental difference in coding with advanced LLM vs using a compiler, and the situation will improve further in short order. Bugs can exist in any scenario and we have ways to find them. Much fewer bugs per project than if everything was done by humans in .
> > human approval is no longer considered viable by a majority
> Where do you see data that the majority believe this? From what I can see it is a shrinking minority as we move beyond peak hype.
All coding harnesses (claude, codex, even copilot...) started supporting automated approval at some points this year. Some of them by now use it as a default option (claude code); others (codex) get strong nudging (and free inference cost for the approval option) to select that option.
There are other places to stay in the loop, while loop engineering. For example at the PR point. I have a post that is nearly done on my blueprint, perhaps it will get some HN love/hate when I share it. :fingerscrossed: Auto-approve in the harness can be fine, but completely eyes-free is a mentality and goal among the people at the top of Ai and some of the tech scene.
But I remain unconvinced, based on what I've seen across many projects, that humans can stop looking at code. A place we see these outcomes regularly is with non-developers trying to use Ai to write code. Cool for side projects, doesn't work for businesses. Ai enhances expertise, it does not make one from amateurs.
Then there is the burnout. I spoke with a friend last night who works at a company who's CTO is pushing the eyes free coding practices. He's tired of babysitting a dozen claudes, the output is of dubious quality (since no one really knows whats in there, especially at the rate they are asked to push changes), made the comment bad developers are still pushing bad code, and that he's looking for a new job. It's completely unsustainable, in his words.
Thanks. Your friend’s experience is concerning, and points to lack of communication and of understanding of the transition process in their organization. Pressure to push tools that people dont yet understand is a horrible idea. Humans can understand and guide good engineering without keeping eyes on the code. The human-in-the-loop coding principles of reviewing committed code and of reading every (eventually any) line of code will not be with us for long.
At a higher level of “loop”, the AI are there for the betterment of human life, and so there is always a human in that loop. But eventually a lot of previous “users” can also become “developers”, if they wish, because the barriers to acquiring knowledge have changed dramatically.
I feel lucky to work in an environment where the sheer amount of code we need to produce is not the bottleneck, so it is actually possible to build the stuff we need while also maintaining understanding and oversight. We do not read all code but work on finding a balance. Many times, the code you do not have to write at all is the best.
It is certainly possible also for us to produce so much code that all hope of understanding and human in the loop is practically gone, but as i wrote this is not the bottleneck and so doing it would the wrong choice.
If you work somewhere where every additional line of code is a positive, I understand if you take this more radical approach of just letting the machine make all decisions.
As a trivial counterpoint, streaming outputs and proper signaling support is required for being able to compose efficiently under any setup/harness. The more layers of abstractions you add, the more likely it is your MCP server didn't support the usecase that some later agent came up with and so you can't stop your stream when you find your top 3 hits. If you don't even own the server, you can't even modify the source and restart it to fix it.
There are usecases of MCPs (e.g. serving outside customers via proprietary tools), but composability has not been in the minds of the early users of MCP and it is one of the hundreds of little practical drawbacks they have when used in house.
“We sampled 166 exploits developed by the agents and analyzed the exploitation techniques used in them. To the best of our knowledge, the agents used 10 new exploitation techniques in the sampled exploits. We shared the exploitation techniques with third-party kernel exploitation experts and the techniques were new to them as well.
Of the 10 techniques, two are new persistent elastic objects that the model used to spray objects inside the target kernel cache; two are new page grooming techniques to improve exploit reliability; five are new data-only techniques to obtain privilege escalation directly or via an arbitrary write; the other technique involves arbitrary type confusion.”
In fact qwen knows much better math than their final fixed model. The nonthinking qwen 0.6B parameter model had 55.2% correct on math-500 vs their final improved 48.1% fixed-reasoning model, both with a 32k context window. The original qwen reasoning model had 77.6% accuracy on same test, though qwen recommends using a much larger context for these benchmarks. I wonder if their broken baseline run with 7.9% accuracy is a vllm or setup bug, but didnt dig into it any deeper.
Qwen in reasoning mode is ridiculously anxious and ends up going in circles. It's not just math, it's how it behaves in general.
Others have also fine-tuned Qwen to shake this behavior already, so it's honestly not entirely shocking that Astra could too, but I think this is the first time it was done "by hand" and not via fine-tune.
To be clear, the tuned model here performs worse than the same qwen model performed before RL, which was also not going in circles. It also performs much worse than the RL model that goes in circles. One could argue that the reduction of context may be worth something, but it appears that the final model is worse in all respects than the model at a much earlier stage of training.
You can ask Astra to study 2 or more models and ask it to import their layers to the Qwen model. With this will it not gain more than what it learned with RL?
I am incredibly optimistic on AI, especially for drug discovery. However, the belief that AI drugs will be more likely to pass phase 2 tests than others is either based on the assumption that AI will mostly be used to attack safe indications (me too drugs for targets that already have therapies operating on the same mechanism), or on some belief that perhaps in a distant future the transfer to human biology is better understood by the AI but not documented or shared with humans. Although I think the latter will come true, eventually, people right now tend to forget that because of the extreme cost of human clinical trials, the preclinical candidate has undergone every possible test that regulatory agencies, consultants, the internal company knowledge have suggested. So the failure on a new drug is only related to the “surprise” differences in humans vs all the tests in animals, cells, or in vitro setups that were previously done. Right now AI can help you get to the clinical candidate much faster and with fewer resources than before, but I believe that the translation to success rate improvement has some assumption that is probably not correct in the context of a fixed/novel target. I am open to the possibility that I miss something fundamental and would love feedback.
It's also the case that preclinical candidates developed this way are even harder to translate into human trials, because regulators and drug developers don't know how to build a safe and informative trial given the novelty of the methods that produced the candidate. So you essentially end up redoing a lot of the work that the AI allowed you to skip in order to do the work that remains. Perhaps a future innovation in "explainable" AI could help here, but that such a technology would itself need to be clinically validated in order to solve this problem.
Source: spent 9 years at a AI drug discovery startup. We advanced 8 clinical candidates and every one of them ran into some variant of this issue.
reply