I think the real value here is not as a customer-facing agent/chatbot but for for automated processes. Think of all the companies out there that have LLMs doing simple tasks like categorizing customer feedback emails. For such tasks, you don't gain much from better models, so if you could run it 10x cheaper on a slightly older model, it would absolutely be worth it. Pretty much any place people are currently running a flash model could benefit from this since they're already deciding that speed+price is worth using a less capable model.
I find speed alone would be a game changer for current models. I hardly find any task anymore that the current frontier models can't do with max reasoning after several rounds of feedback (provided sufficient instruction and the right harness). But waiting an hour or more for reasoning to finish is getting really cumbersome. If they could do the same in seconds (and for cheap of course), I'm pretty sure we'd pretty soon see major software companies pop up that are run by a single human.
I've let it research and implement pretty complex stuff for my 3d engine. Like a realtime volumetric cloud rendering solution. The kind of stuff you can do yourself, but it would probably take weeks, if not months. I spent quite a bit discussing the design with it, knowing some details of how it should work from game dev conference talks. Once we agreed on a design and implementation plan with key steps and intermediaries, i just said "now implement it" and it happily designed algorithms, wrote and ran tests, generated and looked at screenshots and basically did everything you would expect from a really senior engine programmer for more than an hour of wall time. The end result was something that worked amazingly well after testing it myself and suggesting minor corrections that were my fault for not specifying them in the first place.
There’s some kind of tradeoff between speed, cost, and quality for every application. I would be perfectly happy with a model 6 months old that was 50x faster for many uses. Right now I use either Opus (for smart stuff) or Flash without thinking (for fast stuff). I would take an even dumber model for more speed (lower latency in particular).
I think this would make sense for consumer hardware, not for AI companies.
AI companies constantly update/change stuff, new models come out, new requirements, etc.
But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.
Makes sense. Actually to expand, I believe this can make a lot of sense for industrial robots and such which have a more or less fixed job and latency matters more, so a well tested model may be more valuable than need to keep updating them
You don’t need this chip to do that. Computer vision has used machine learning for decades. The task you’re describing is pretty rudimentary and an off the shelf model with a control system would do it way cheaper.
It was just a random example, you could think of it as being a lot more complex (detect which type of food it is, what detergent to use, how much water, remember patterns, learn over time, adapt, etc.)
It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size scales directly with model size. The reason they have llama3.1-8B as their model in https://chatjimmy.ai/ (their demo) is because it's very hard to do anything bigger than that, you literally start running out of physical space on the chip, and bigger chips cost money. You can't have a dishwasher using a chip the size of a B200 die to run a model you could run on a Raspberry Pi.
What's worse is that this is when it's already quantized to ~3 bits per parameter (which is fairly lobotomized). Yes, the chip will run it 1000X faster than the Raspberry Pi, but it will only be stupid faster.
Their press release explicitly lists that their HC1 puts the entire Llama 3.1 8B model on one 815 mm² TSMC N6 die, with about 53 billion transistors.
815 mm² is fucking huge. An RTX 5090 is 750 mm². A mid-to-high end consumer CPU die is something like 300 mm², with a lot of budget parts being significantly smaller than that, down to ~70 mm². Every square mm costs money.
If they upgraded from N6 to N3P they MIGHT be able to get as much as a ~35B class model on a a chip which fits in the reticle limit, maybe, probably not, because model weights aren't the only thing that needs to fit on the chip.
There are very serious issues with agentic performance in this setup, which is exactly where you would want something really fast. Their Llama 3.1 demo lists a context of 6,144, which is dramatically lower than the 131,072 Llama-3.1 supports.
Reasoning models are barely usable with contexts that short.
The reason for this is that to actually get those speeds, the KV cache needs to live in SRAM. You can't bake the KV cache into the circuitry since it... changes. They clearly don't have enough SRAM, and the problem gets worse the bigger you make the model since KV cache grows (sort of) with model dim. The longer you want to make your context, the more of your chip needs to be SRAM.
Frankly, I don't see the use-case for this tech. It's too expensive and too inflexible. Just doing what Cerebras did and making a wafer-sized chip which is mostly SRAM is a much better solution to serving LLMs at extreme speeds and you don't need to make a new chip every time a new model comes out.
Yeah, makes sense, if it's good for very small models only, then there's no point, as those van already run on cheap consumer hardware.
Yet, maybe it can work well enough, so that as a manufacturer, you don't pay $50 for a PI, but only $0.50 for a tiny "hard-coded" chip.
The advantage can be that, as a LLM, as opposed to other types of chips, the use-cases could be more varied, so same chip could be use in different devices (robo vacuums, security cameras, ball-shooting training robots, etc.)
Lots of chiplet interconnects hurt your max performance pretty severely and drives up your complexity a lot. There's a reason they want to keep it monolithic.
I already harped on Cerebras, but their approach of just copy/pasting a whole bunch of identical functional blocks, over-provisioning the chip by ~8%, and then just fusing off blocks with defects allows them to effectively have 100% yield on a wafer-sized monolithic chip. This is very desirable, and just another reason I like their approach better.
Perfect for consumers. You buy it and then you need to buy a new one in a couple of years. If they can make them affordable they'll sell like hotcakes.
And the second hand market. I'd love to see this integrated into motherboards like RAM. Someone could have a motherboard with 4 sticks of different AI with various models. Swap, change and trade.
The thing is, right now it is exploding because we are at the beginning of it. At some point, it will plateau at a specific level, and not that much quality will be gained. There is however leaps to make for efficiency.
The same can be said about the CISC computer: yes, new processors introduce new instructions that do something slightly faster, you could still crunch that with an older processor. The real benefit comes in clock cycles (that's why Arm with a reduced set can compete with x86).
Also: there are myriads of models, for myriads of tasks. Not all have the same development gains as we see for general purpose AI. If you etch those, you reduce your bill by factors down.
It also democratises models: Instead of running them on a cloud server by some company, you can run them at home, for coding tasks, without the need of internet connection, etc.
Already models have gotten really good at a lot of things.
A lot of people would probably be happy to stick with the same model for a year or two if it’s 10x faster and cheaper.
And perhaps older models can become cheaper over time as newer models come out on new silicon for a higher price. That incentivizes people to stick with older models.
They expect a sort of breakpoint at which each subsequent model version will only be marginally better than the previous ones, thus allowing them to retain their value for some time. Their business doesn’t work if each year the new model demolishes the previous one in terms of performance.
That’s not going to be true forever. As models mature, we will hit diminishing returns. Major improvements will come annually rather monthly - matching the roughly annual release of new processors. Model ROM’s will likely get integrated into die packages just like DRAM now.
There isn't a fast churn in the underlying pretrained model, nor RL. It's mostly orchestration around the model. Said another way you could just pretrain and RL for longer.
Also I believe there is both a market for extremely fast local inference with current model performance and that such fast inference would unlock unforeseen usecases. Especially as TPS approaches early computer clock cycles and data rates.
also, as this scales, what would this mean for closed-weight hosted models? i imagine it's possible (but difficult) to re-derive model weights by de-lidding and inspecting the die... so will this only ever be used for open-weights models?
pretty much everything is “1 or more versions behind” by the time it comes out. the question is whether or not it’s still useful? at some point, presumably not every application will need the latest cutting edge huge model.
Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out.
Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.