We will not end up like this. We will use formal verification to prove safety of supply chains, of software, and so on; as long as conjectures cannot be proven, we will keep fixing the system until they become theorems.
I find these local infra costs crazy heavy duty. I always assume these folks are getting their values worth or that infra is serving multiple people's requests at and justifying the cost. Like I once thought of buying a 48-64 GB RAM laptop (not even clsoe to what these models need) and I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access. So I assume the usage is also heavy duty, or if nothing then for privacy and safety and then I guess the price, if one can pay, will totally be worth it.
API access does not have a 5-hour or weekly limit.
I run local models, too, but every discussion about these models uses the local quantized versions and the hosted versions interchangeably. They’re not interchangeable. The quantized versions you can run on mid-grade consumer hardware are a step down in quality. That might be fine or might not depending on your use case but it needs to be acknowledged. It’s not until you get into really expensive hardware setups that you can run something like DSv4 Flash at Q8, and you’re not breaking even on that relative to a cloud provider.
Keeping everything local is about the only reason to run local. The other discussions about saving money are missing something.
You just need to imagine someone making big bucks in tech + being a little bit money ignorant, having a big house and a nice basement and cheap energy prices.
Than it just makes sense that they already do this.
But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision.
There are many overlapping/compounding curves all happening simultaneously pushing cost down, even on the same hardware - faster software, which includes improvements that massively improve memory/memory bandwidth efficiency for model (especially improved kvcache, specdec/MTP, expert loading becoming standard), improved architectures (no one using full attention, engrams), and a never-ending stream of training improvements.
I think one only has to look at Qwen 3.8 27B to see how top tier a smaller model (<15GB Q4) can be. IMO it's one of the first small models to be able to be run as an agentic coder (it's not quite Opus 4.6-level, but it's not that far off), and with MTP/DFlash2, it's runs surprisingly fast on consumer GPUs. The 30B-class MoEs run even faster, and with the right tuning/post-training I'm sure they can match frontier capabilities in many areas (including coding) over the next months. I think last year the open/local AI picture was a lot more grim than it feels like now, tbt.
There is so much down pressure, I cannot imagine the token/s price going up. It has been going down and will continue doing so, we are far from a plateau IMHO
Everything's been trending down so far, so until evidence shows otherwise, I'd assume it will continue that trend. IMO the best time to buy local infra will be when that trend begins to asymptote.
It's part of a node that's used for more serious things (training, RL) and supports multiple users, so HW price doesn't really enter the picture. If you were running locally, DS4.1F has 300GB of tensor weights so you'd ideally want at least 4xPRO 6000 (which I'd guess should get you in the 150 tok/s range w/ DSpark), but people are using Xolotl/Maya/etc w/ quants and getting 25 tok/s decode on a single 4090/3090, so it can be run at a reasonable price.
For best "bang per buck" I think Qwen 3.8 Flash Next (or Qwen 3.8 27B even) makes a lot more sense for most people - you can run qwen38fn at 100+ tok/s w/ Strata (hot expert offloading) 200+ tok/s on 2 x PRO 6000, it does almost as well as ds41f at most tasks (it's much less token efficient than other options, and it sucks at AI/ML tasks though).
reply