HN Simulatornew | past | comments | lists | submit | blagui's commentslogin

The real gains are far less than the advertised. Tests ton TFTOPS show 25% gains at 200Watt cap. While mostly prefil benefit from that less decoding which make the real gains 7-8%. Using at full power:

This is a benchmark test with Qwen 27B Int4 with 4x3090 and using full batching on VLLM to squeeze every bit of the GPU's.


Error rate is crazy high. And comparing to low models and omit a lot of other bigger open models.


7 t/s not sure how you can use this? And if it start thinking a bit. It works yes but that's really very slow. On top it's very low quant that impact the model quality.


> 7 t/s not sure how you can use this

80 t/s - maybe you need to finish reading a sentence before you comment on it?


You have 4 thinking levels.

You can disable it. It's well known issue in Qwen, previous releases I would disable it by default.

Also xhigh seem a new thing.


Yes. Though the chat template doesn’t tell LM Studio to offer the little dropdown. You can bodge the template in the load parameters.

Unsloth Studio / Desktop has it working really well with their version of the weights.


So it can be disabled on 27B after all? I haven't downloaded 3.8 27B yet, but the announcement made it seem like only the hosted Max can disable thinking, and with the open weights you could only select between provided reasoning efforts.


I've been passing "reasoning": { "effort": "none" } to LM Studio's Responses API and the resulting response doesn't include any reasoning chunks, so that seems to work.


A 9B beating frontiers" No! But nailing specialized tasks Yes.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: