Cool! I'm thinking about a local set up. What's your usual tokens/second rate?

Not OP, but I’m running local models on a M1 Max as well with 64GB RAM.

It varies by model, but I’m getting 50-60 t/s with Qwen 3.6 35B and Qwen 3 coder 30B.

I’ve also used Qwen 3.8 27B but I get 10t/s on it.

It’s useable in some use cases, but I rely mostly on my $20 Claude subscription.

Try 3.8 27B in MTPLX; I get about 30 tok/s with the same hardware as you. (Although it does use around 90-95W of power, compared to the ~60W that 3.6 35B-A3B uses to generate 55 tok/s. That’s about 3 J/tok instead of 1.)

That's so cool. I wonder if the regular M5 can run those models too.

I run qwen 3.8 27b on my m5 mbp, with 48gb of unified ram and I’m getting around 10-15 tok/s.

3.6 35b a3b, I’m getting upwards of 100

NGL: I don’t really have a good way to find out right now. It also doesn’t matter that much because the way the models use the tokes varies a lot. Qwen 3.8 is known for overthinking while Muse Glimmer may be a little slower per token, but it uses them very efficiently, caveman style.

Generation speed isn’t the bottleneck anyway, at least on pre M4/M5 devices (the newer chips got significant processing acceleration). It’s prompt processing time. OpenCode’s system prompt can take up to 3 minutes to process, which is why good prompt caching is essential.

For that I use omlx, which can persist the KV cache to disk, chunked so you can reuse parts. This helps with the usability a lot, when an agentic session is warm it runs pretty smoothly. New requests can take a couple seconds (sometimes many, which must be fixable somehow).

So: It’s not fast, but I also don’t find it awfully slow. My use is typically semi-interactive, for fully interactive use you have to wait a bit, but it’s possible. I personally am still regularly amazed that something even close to this is possible on completely local hardware.

Sounds interesting. Can you give more details on your workflow and what tasks you use it for?

Code, creative writing, email summaries, automated email replies, and I prefill my invoice notes and daily updates for work.

Actually built a full invoicing product for that, using it too.

I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC).

Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :)