Many benchmarks now measure the total cost or energy usage per completed task.

You don't have to send every single request twice, just the ones that are haven't returned in time. Wait until some threshold, such as your p95 latency, and send your backup request after that. Return whichever request comes back first, and it should cut your tail latency without doubling your cost, since it only duplicates the small % of requests at the tail.

Google calls this a 'hedged request': https://cacm.acm.org/research/the-tail-at-scale/

I think it is more likely that a smaller model (<400B) with similar intelligence gets developed long before the hardware to serve a 2.4T model gets cheaper than 10k.

Ah, I hadn't thought of that. To what degree have we seen this already? What would you consider as the biggest jump in intelligence per weight?

It's been pretty consistent, the smaller models (hundreds of billions of params) usually catch up in 6 months or so to their frontier counterparts, at least on benchmarks.

Note this is the 07/31 release of DSv4 flash and not the "preview" that they put out a couple months or so ago.

I've been running this model locally for a week, and the preview version before that. This updated one feels like a whole tier up. It's very capable for debugging and analyzing documents/data I upload.

The killer feature, IMO, is the speed. On 2x RTX Pro 6000 Blackwell, its ~8k tok/s prefill and ~250 tok/s on a single stream. I saw 1000 tok/s with ~64 concurrent streams on vLLM.

That's fast enough that you can interactively chat with it without switching tabs while you wait, and its a ~300B (13B active, hence the speed) model so the responses are also very good. It's actually more convenient now for me to direct 95%+ of my day to day usage to my local model, and only use Claude Fable for really big coding tasks.

Until this model was released, I was contemplating spending even more money on hardware to run GLM5.2 (~750B) at reasonable speeds, but I no longer feel that need. This is smart enough, and I think it only gets much better for local models from here.

What quantization level is that? Because official endpoints are slow.

It doesn't need extra quantization. The official weights are natively mixed precision FP4/FP8, so it fits in ~160GB. The API slowness is probably from being batched with other concurrent user requests. The provider's aggregate throughput gets higher but per-stream speed slows down.

V4 Flash fits entirely in two RTX Pro 6000s without any quantization at all.

From opencode go $10/mo plan I get between 60 t/s and 100 token/s even with large contexts of 150k+ tokens.

I wouldn't call 80 t/s slow.

You are right, relatively to other llm providers this is not slow. But if you think what is possible when you have 1000t/s a sec you might find it slow.

That's across 64 concurrent streams; you could make more concurrent requests to DeepSeek API no?

I get around 60-110 TPS on tensorx.ai hosted models.

What runtime are you using with the 2x RTX Pro 6000 Blackwell machine? I have the same setup and tried DSv4 Flash on vLLM and ran into a ton of kernel bugs that don't seem to have been fixed yet.

so..you guys have $20k+ rigs?

This one works great. https://github.com/ormandj/sglang-deepseek-v4-flash-sm120

This also works great: https://github.com/antirez/ds4

> On 2x RTX Pro 6000 Blackwell, its ~8k tok/s prefill and ~250 tok/s on a single stream.

For reference, on a 1x B300 it's over 400 tok/s decode on a single stream.

I wish I could say that it performs reasonably on my hardware. 8x Radeon AI Pro 9700XTs, and I can't get it to hit double-digit tokens per second. Neither vllm nor llama.cpp, with various combinations of quants, draft models, and parallelism methods can get it to run at a tolerable speed. I'll be sticking with StepFun 3.7-Flash for the foreseeable future :(

I'm getting like 25 tok/s on 2x RTX Pro 6000. This is with llama.cpp, but I had GPT tune it for me. I was under the impression vLLM was at most ~2x faster, and usually for highly parallel loads. Any tips on where I should look first for an obvious blunder?

I'm guessing tensor parallelism or similar?

You're gonna want a custom vLLM build.

Here's a runbook: https://github.com/local-inference-lab/rtx6kpro/blob/master/...

If the newer builds aren't working, you might try running the old v6 build (based on the eldritch-enlightenment image). gilded-gnosis gave me some problems that I haven't bothered to track down, the old builds are still gonna blow away llama-server performance. And that's before you get hooked on vLLM's PagedAttention and can run multiple sequences without a ton of extra overhead.

I haven't live there since 2020, but before that it was all various SaaS and data pipeline related billboards.

It doesn't sound like it used to be any better. Maybe we don't need to complain about AI destroying things that already sucked, like billboards or Linkedin.

And at the turn of the century it was all about dot coms.

It's expensive to run it locally at full quality, but at least on my setup, its about 5 times faster than any API, and is completely private.