It’s not reasoning, the hardware demo uses a 3.-something generation Llama 8B.
But it’s proven they can automate this (they didn’t etch eight billion weights by hand after all, obviously), so now the interesting question is whether they can scale it to more recent aka bigger models.
After all, there’s already very useful models even for productivity at 27 or 35B.
It's not really "instant", i.e. the text is still generated token-by-token, it's just super fast. Reasoning would work with this model without any changes to the chip but it's disabled for speed.
> Not sure if modern models "think" only by outputting blocks
That's pretty much it - a small refinement to "Chain of Thought" prompting, where you tell the model explicitly in the prompt to "Think step by step" or similar, so it writes out more steps before giving a final answer, potentially catching some errors. The "thinking" models are tuned to do that without being prompted to, and to output the "thinking" markers around it, so they can be hidden from the user.
Well at ≈15k tps, I think the bottleneck will not be the model (reasoning or tool calling), and attention will be shifted to the harness’ engineering again.
Probably, the usual initial suspects for “what makes computation slow” will become a focus point that needs to be optimized again: file access, network, etc.
But it’s proven they can automate this (they didn’t etch eight billion weights by hand after all, obviously), so now the interesting question is whether they can scale it to more recent aka bigger models.
After all, there’s already very useful models even for productivity at 27 or 35B.