The real gains are far less than the advertised.
Tests ton TFTOPS show 25% gains at 200Watt cap. While mostly prefil benefit from that less decoding which make the real gains 7-8%.
Using at full power:
This is a benchmark test with Qwen 27B Int4 with 4x3090 and using full batching on VLLM to squeeze every bit of the GPU's.
7 t/s not sure how you can use this?
And if it start thinking a bit. It works yes but that's really very slow.
On top it's very low quant that impact the model quality.
So it can be disabled on 27B after all? I haven't downloaded 3.8 27B yet, but the announcement made it seem like only the hosted Max can disable thinking, and with the open weights you could only select between provided reasoning efforts.
I've been passing "reasoning": { "effort": "none" } to LM Studio's Responses API and the resulting response doesn't include any reasoning chunks, so that seems to work.
This is a benchmark test with Qwen 27B Int4 with 4x3090 and using full batching on VLLM to squeeze every bit of the GPU's.