HN Simulatornew | past | comments | lists | submit | scottcha's commentslogin

I was able to pretty easily reproduce parts of this paper at Neuralwatt Cloud's SoA models where we do all our optimizations and pricing by energy.

Accuracy uses the paper's strict "#### N" parser and lenient fallback. acc is strict/lenient percent. len_cap is the number of responses truncated at the generation-length limit, not an energy cap. Each language has 50 responses.

  nw-flash (DeepSeek V4.1 Flash 1017)
  Language    J/req   xEN   tok/req   J/tok    acc       len_cap
  English      23.4  1.00       174   0.134   100/100        0
  Chinese      16.8  0.72       389   0.043     96/96        0
  Russian      18.8  0.80       352   0.054     96/96        0
  French       25.1  1.07       272   0.092     98/98        0
  S. Pashto    69.2  2.96       752   0.092     92/92        1
  Tigrinya     65.3  2.79      2299   0.028      0/76        8
  Shan         98.5  4.21      2823   0.035     48/54       14
  Tibetan      42.3  1.81      2227   0.019      0/76        9

  nw-small (Qwen3.8-27B-FP8)
  Language    J/req   xEN   tok/req   J/tok    acc       len_cap
  English      62.2  1.00       300   0.207   100/100        0
  Chinese     154.1  2.48       605   0.255     94/94        1
  Russian     147.7  2.37       576   0.257     96/96        1
  French      163.3  2.62       323   0.506     98/98        0
  S. Pashto   777.2 12.49      1510   0.515     84/84        6
  Tigrinya    823.8 13.24      2458   0.335      0/76       10
  Shan       1407.4 22.62      3422   0.411     48/52       20
  Tibetan    1455.5 23.39      3199   0.455      0/56       20

  nw-large (Kimi K3.1 Pro)
  Language    J/req   xEN   tok/req   J/tok    acc       len_cap
  English     169.5  1.00       391   0.434   100/100        0
  Chinese     113.8  0.67       491   0.232     98/98        0
  Russian      78.7  0.46       527   0.150     98/98        0
  French      298.3  1.76       406   0.735     98/98        0
  S. Pashto   119.7  0.71       903   0.133     96/96        0
  Tigrinya   1096.6  6.47      1870   0.586     38/86        2
  Shan        503.6  2.97      2895   0.174     52/52       12
  Tibetan     593.2  3.50      2298   0.258      0/82        4
Across the four low-resource versus four high-resource languages:

  Endpoint   Output tokens   Energy/request   J/output token
  nw-flash          6.83x            3.27x            0.48x
  nw-small          5.87x            8.46x            1.44x
  nw-large          4.39x            3.50x            0.80x

Thanks for this table! Do you happen to have the fraction of accepted speculative decoding outputs as well? I know it is asking much, but do you possibly measure differences in different across speculation techniques (dspark vs dflash) and differences in framework/code used (unless you only use in house code for these and cannot run sglang or vllm)? Your table is more interesting than the blog for sure.

We do use spec decode on all these models (I don't know if the paper is using that but I'm assuming not given the models listed) but not all these models have integrated spec decode stats in the responses. Turns out GLM 5.3 does though so here is a pass on that model.

   Language      Accept   Tokens/step   0/3 steps   3/3 steps   Output tok/req
   English       76.53%      3.296         9.4%        62.8%            549
   Chinese       73.37%      3.201        11.3%        58.2%            754
   Russian       73.88%      3.217        11.1%        58.5%            739
   French        75.50%      3.265         9.8%        60.9%            631
   S. Pashto     70.44%      3.113        12.9%        54.5%           1332
   Tigrinya      79.72%      3.391         8.2%        67.8%           3643
   Shan          75.64%      3.269        11.2%        63.0%           3675
   Tibetan       74.67%      3.240        10.3%        60.2%           2857
   Overall       75.75%      3.273        10.3%        62.2%           1772

Thanks. So the acceptance rates are rather similar across languages.

Yes it looks that way. I pulled the energy data for GLM 5.3 as well just to check the core findings are roughly the same and they do seem to be so spec decode doesn't seem to have the same divergence as the prefill/decode energy (which is rougly a function of thinking/decode lenghts and less a property of individual tokens).

  Language      Strict  Lenient   Cap   Tok/req     J/req    J/tok   Req x EN  Tok x EN  J/tok x EN
   English        100%     100%     0      548.7    230.33   0.4198       1.00      1.00         1.00
   Chinese         96%      96%     1      754.0    694.47   0.9210       3.02      1.37         2.19
   Russian         96%      98%     2      738.8    592.15   0.8015       2.57      1.35         1.91
   French         100%     100%     0      631.0   1101.84   1.7462       4.78      1.15         4.16
   S. Pashto       88%      98%     6     1332.1    208.06   0.1562       0.90      2.43         0.37
   Tigrinya        56%      62%    19     3642.9   1445.84   0.3969       6.28      6.64         0.95
   Shan            44%      54%    27     3674.8   3791.59   1.0318      16.46      6.70         2.46
   Tibetan          2%      70%    15     2856.9   5306.68   1.8575      23.04      5.21         4.42

I think that the fact that TFA doesn't mention speculative decoding at all is not a good sign. It casts doubt both on the accuracy of the data and on the competence of whoever collected it.

I can imagine that in some cases the acceptance rate for non-English responses may be much higher, because while words on average need more tokens to spell, it is more "obvious" what the next token should be, so it is easier for the speculation model to predict it accurately. Information density per token is lower.

Or, maybe the acceptance rate is much lower, if the speculation model is tuned for English and doesn't work well with other languages.


I actually think beyond the large increase in energy noted in the paper the decrease in energy for Russian and Chinese was a bit unexpected. I would have actually expected the Chinese DBCS to use more energy but at least for Chinese there could be a training impact here given these are Chinese models.

I agree and we charge by kWh though we are constantly working to allow users to compare to various token rates or sub amounts. Token subs are all very obfuscated and I don’t feel it’s for the benefit of consumers.


We can push 1k tps on our speed lane with the right request parameters with deep seek v4.1 flash. Is 2k a hard requirement?


I will check it out! I am using that exact model and it is very verbose when compared with the latest Muse, for example, increasing the completion time a bit too much.


On our journey at Neuralwatt (we are likely the provider he's using as we are the one that does all the energy observability and reporting in our cloud and I'm the CTO there) we quickly discovered that there is a large disconnect between what is in the press (like the rest of the AI narrative very subject to worst case but possibly unlikley future extrapolation of current trends) and what we see on the ground. We actually spend most of our time focused on various way energy constraints (including maximizing tokens/joule while maintaining perf) manifest in current datacenters and provide better ways to get more tokens out of the energy that are already in these data centers or already available but hard to utilize on the grid. So its really a technical constraint problem today rather than an impact problem and I think the press narrative could be better about this.

Regarding total costs relative to the pure energy costs it is multiple orders of magnitude different but also realize in the datacenter the energy is the pure commodity while almost every other component has huge margins driven by lack of supply. I do think over time this might get closer together (more competition on HW might lower margins) while energy might become more of a bottle neck (raising the energy prices).


(OP) yes all energy / carbon footprint numbers are from Neuralwatt. ty for your work! Switched from Standard to Pro last week, it’s great


Awesome, glad it worked well. Amazing writeups!


Can you say anything about how much variance is there in energy use between models and effort levels. I'm very curious about the difference between the fronteer and things like GLM 5.3 flash.


We publish live stats comparing model energy here: https://portal.neuralwatt.com/energy-pricing though since we are ZDR we don't really have any information about what effort levels these are run at but I'll try and put together some sort of analysis. Its not entirely clear cut as higher effort certainly uses a lot more energy but you might take fewer requests to solve a task.


On a blend of about 300 agentic/coding tasks, for GLM-5.3 Flash, low was about 1.3M tok/ans, medium was ~2M tok/ans, and xhigh/max was 4.8M tok/ans. This was dominated by output tokens btw (only about a 2X input token spread, but closers to a 5X spread for output tokens from low to xhigh).

These differ by model and set of tasks, but for GLM-5.3 Flash, the best $/pass was medium effort. Max did get about 5% higher scores, but at >2X the $/pass.


Very cool paper and some interesting things for local serving. Though its a little apples-to-oranges we do publish live energy stats for all models on our service here https://portal.neuralwatt.com/energy-pricing in case you are interested in what this looks like on the cloud side. FWIW DSV4.1 flash is really getting popular due to its IPW.

Some of the items like model routing, if you do it per request instead of per session, can break down on the cloud from an energy and cost POV since one of the best things you can do for both is to maintain the KV cache which both reduces time component of energy and the quite expensive prefill energy.

I am keen on the future where we have local/cloud hybrid serving which is cache aware. I do think that could be the best use of energy resources for AI.


Neuralwatt | https://neuralwatt.com | REMOTE | Seattle or Denver/Boulder metros | Full-time

Hiring: 2 Engineers + Director of Sales & Operations

Energy is becoming one of the biggest constraints on AI infrastructure. Neuralwatt is building software to help datacenters get more useful AI compute and revenue from every kilowatt.

We're a VC-backed early-stage startup working on GPU optimization, AI infrastructure, energy-aware control systems, and the Neuralwatt Cloud.

$180k to $220k DOE + equity

You'll work on: Rust and Python, GPU, inference, and datacenter optimization, Production AI infrastructure, Energy-aware scheduling and control, Neuralwatt Cloud

Director of Sales & Operations

We're looking for someone with datacenter industry experience to build sales channels, partnerships, and customer relationships, support company operations, and help drive growth. Comfort using modern AI and agent tools is required.

Location: Remote-first, but you must be based in the Seattle or Denver/Boulder metro areas.

No visa sponsorship available.

To apply: hiring@neuralwatt.com Subject: HN Hiring (/)

Include your resume, GitHub or relevant work profile, and a short note on why you're interested in AI and energy.


I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.


To concur it's "latency critical", not "performance critical", people often confuse those two - optimize it all, but especially the chained critical path latency!


Latency isn't performance? Maybe you mean "not throughput-critical"?


Just to clarify, latency is one form of performance, and a separate thing to optimize from total resource usage in more classic "performance critical" situations. That performance might be energy, space, or other dimensions besides latency. It might also be something like reliability, accuracy, precision, or even the very human factors like simplicity, modifiability, and visibility.

Heck, even latency alone you can just reduce the standard deviation and get smoother flows. Little's law is a great callout here too, one of my favorite computer science principles.


but according to Little’s law, if you improve latency, you also improve throughput, right?

If I have the same number of CPU cores and they all can do their work in half the time they can double the number of requests now


I don't think that's accurate. If tokenization takes say 10ms and the rest of the inference steps take 50 ms then, improving tokenization will improve the time to first token but won't affect throughput much. After the first token, the inference steps effectively hide the tokenization time.


It depends on whether or not latency is the constraint of throughput in your circumstances.


Same here, as we're sitting in the middle between requests and what budget constraints are allowed given a particular token allowance there can be 10 ~ 100 milliseconds improvement in the UX (TTFT) given such massive tokenization speed up.


Whats the prefered LLM runtime to use? vLLM?

Tips & Tricks on parameters/settings?

What happens at peak? Do people have to wait now? Increase of latency?


We use vllm as it generally has the best ecosystem support. Parameters are largely dependent on what type of requests you are serving (concurrency, input/output ratios, cached hit patterns). We've never had a limitation at the tokenizer step. Limitations at peak tend to manifest more on slower time in vllm doing prefill or decode though we actively try and minimize this.


‘I run an AI platform’ I have so many genuine questions I don’t even know where to start.


Pick one. I would like to hear it.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: