These are going to be so awesome to buy in 5-8 years!
Right now I’m running 2x 3090 and 128 gb ram on 3.8 flash-next180b (the quant is 97gb) and it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit. For agentic work it does slow down quite a bit when the context fills up… but what do you want for running a 180b model on consumer hardware?
Right now I have it set up as an administrator for our asterisk PBX and radius server, and it does higher level judgment on our cctv footage for notifications to security. It’s been really good so far. Agents are the new UI, everything just works over SSH now.
Lol. I’m totally with you on agentic UX development. It’s frequently terrible, and you can tell which companies are telling everyone to 10x with AI. But the problem is testing, not the coding. My first stab at a UX render usually sucks too, it’s the iteration that brings perfection. It’s the same for agentic development.
What I meant though wasn’t agents building UX. I meant that in our case at least, the agent is the UX. “put a new extension 117 for Sam at the port next to Jane’s, and have it ring through to his cell when he’s not in the office. Leave the old 117 as is but only ring where Sam is, if that is known” basically, agents just do everything with SSH, and you can talk to them instead of poking around with a mouse. We don’t build ux for internal systems anymore. Local inference is strong enough to be reliable now with the correct tools and policies, and for 10-20k in hardware you are self contained. (Obviously $more if you are going to have a lot of people using it at the same time)
The “one wierd trick” is to build testing jugs for everything. Before a configuration is deployed, it’s tested against a toy instance, all existing functions are tested in all common cases, new functions verified, inputs fuzzed, adversarily abused, and stress tested against dummy loads. If its code, it gets examined against development policy, self containment, explicit interface walls, race conditions, code standards, complexity, factorability, and often A/B testing of different architectural choices. Except cryptography and a couple of other things, we generally prohibit the use of any libraries, it’s just not needed and the code never passes our tests anyway.
People are just YOLOing it and assuming that if it basically functions and looks right then it must be perfect. You’d have the same luck if you told your UX guy “test only the obvious, and if it looks about right at first glance, merge it. Don’t do extensive testing.”
The real thing with AI is that now you can afford to do a ridiculous amount of automated testing, setting up test fixtures, hardware simulators, mannequins of all the external interfaces, fuzzing, etc. Our testing code is 2-3x our product code, and we are shipping 10% of the bugs we shipped with humans, and it’s still 1/4 the cost and 4-10x the speed.
People shipping all the crap that is everywhere now have bad process engineering. Plain and simple. If you try to build software with AI like you used to with people, you’re going to have a bad time. The pipeline is very different.
> My first stab at a UX render usually sucks too, it’s the iteration that brings perfection
With AI, your iteration cycle gets 10x/100x quicker. For the same developer effort, you can effectively try out 10s of ideas the same period of time.
Iteration also has a human component. When a person works hard and writes the code, they are personally invested in its success. So iteration, or, chucking out someone’s hard work is psychologically damaging and causes human conflicts in teams. If AI wrote it, and AI deleted it, no-one gets upset. All that matters are good ideas. Reducing the human cost of iteration is just as important as the time saving.
And, now you have freed up implementation time, you now have way more time for testing, that thing that everyone wished they had more time to do, with a super intelligent 100x developer (ai) as your coding partner, and a 100x lever for automating regression testing. Bad UX has zero excuses.
>With AI, your iteration cycle gets 10x/100x quicker. For the same developer effort, you can effectively try out 10s of ideas the same period of time.
This, except not (just) for UX, but for everything else too. Tonnes of ideas and side project people had on hold for lack of time and/or knowledge, have now been built and trialed thanks to AI.
You see this in the recent boom of reverse engineered multi platform releases of vintage video games that were once tied to a specific console.
Or people reverse engineering their crappy IoT home smart devices to build better multi platform UIs controllers for them.
Which is why I'm really struggling to understand how tech people can still now say with a straight face "AI isn't useful for me". Maybe they just lack imagination?
kudos. Precisely this. People are doing agentic design backwards very often: YOLO then cry. But, if first you write skills, specs, set up orchestration, architecture and flows, including review layer, discrimination layer and trajectory management layer, then you task the agents with creation of verifiable testing and static analysis rig, based on the golden samples of high-quality manually engineered code, and only then you tell then to write conformant code.
Agents (and LLMs) are inherently non-deterministic. In coding practice, this translates to garbage code in the long run. The solution is simple - make the process as deterministic, verifiable and rigid as possible. Suddenly, you get all the bonuses of agentic flows (insane initial speed, trivial cost, express iterations etc.) with none of the usual problems... and suddenly, humans have hard time matching the quality of the final artifact with hand-written code, because we suck at testing, fuzzing and debugging - because we're basically single-threaded and overfocused. Agents, OTOH, excel at it, and context management is also trivial compared to humans.
It works just fine. I've been using it for over three months myself once I tuned my MCP server. I run a 25-30 agent fleet with zero issues, I control it with Codex or Claude Code.
This isn't something you'd be able to manage with a normal UI since every agent has its own at of settings. My agents come in sets, I usually change a set or multiple sets, 10-15 settings per agent. And my harness hasn't made a single mistake once I worked out all the kinks from MCP.
The closest I can think of is AMDs MI350P it has 144gbs of hbm3.
The press says amds price is 30-40 thousand dollars. On eBay there is exactly 1 going for close to 90 which is crazy because for that amount you could buy a MI350X and all the related hardware to actually use it and it has double the vram.
So I would peg it at 40-80 thousand dollars given it's Intel, and it's lpddr5 not hbm and not gddr7 so performance is likely to be lackluster.
I use Claude Opus 5.5 and Codex Sol 6.1 in my job and I love it. I wish be able to run local models which have power at least current Opus 5.5 on medium hardware which does not cost millions $. This is my dream, running LLMs and AI agents to just create things for myself. On my really old hardware I can only run simple and small 1.5B model (Qwen2.5-Coder-1.5B) and well... Ihmo it works fine with offloading to GPU and using llama.cpp but it does a lot of mistakes so I deciced to stop using it.
At work we've got a 96BG Blackwell card serving Qwen3.8:27b at well over 100tok/s. Most of the time it's way faster than codex. Just this morning I set qwen and codex on the same task and Qwen came back flagging problems Sol 6.1 missed.
Local models are going to absolutely eat OAI's lunch before too long. The only moat left is hardware cost, and they can't keep it artificially high for too much longer.
> (...) it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit.
This is exactly why I interpret the aggressive price cutting by OpenAI on their Luna model line as a clear sign they are scared shitless of local models, and how they can trigger a "the king has no clothes" moment for AI companies like OpenAI.
> as a clear sign they are scared shitless of local models,
The number of people with hardware to run a model like that is very small.
It's going to be many years before hardware to run medium size local models is widely available. Running mid-size local models doesn't even register as a factor in pricing decisions right now.
It's such a small part of the market and will be for many years, until RAM and other hardware prices come down.
> The number of people with hardware to run a model like that is very small.
True, but the number of people who pay the non-subsidized API price is also very small, and it's all in big companies who would have no issue buying or renting hardware for that purpose.
Ram prices suggest there is going to be a mighty glut of the stuff in a couple of years. We may easily see 32->64->128 GB system RAM off the back of that and CPUs seem to be moving to unifying VRAM and main memory.
I don't think running a ~200B parameter model is that out of reach, it might be usual by the mid 2030s for people in the tech world. Factor in the crazy improvements in quality given a number of parameters too.
but the number of small to medium sized companies which really want to run (some of) the AI usage under their control and are willing to drop some 100ks on it aren't that few (especial for non US companies given the state the US is in)
> but the number of small to medium sized companies which really want to run (some of) the AI usage under their control and are willing to drop some 100ks on it aren't that few
To this I'll add companies who don't mind paying for a non-US service provider to deliver the compute required to run local models.
Just like some users are happy to run GPT6 Luna, some users are happy to run gemma4 et al.
Idk, I’m running 180b on $2500 of hardware at 120 T/S (single user of course) but I bought it before the clown show. Now it’s probably more like $10k lol
I expect I’ll be able to run 1/2 terabyte models in 5 years for a couple $K in hardware. Inference is going to be a lot like Bitcoin mining. Only the peak of the power/performance curve is going to be profitable. Which means a glut of surplus hardware that can’t make money but makes tons of tokens.
Many years is only long in relation to the AI hype cycle. In terms of a human lifespan it is not that much, maybe 10 years at most and that's pessimistic. We technically live in that future already it is just very unevenly distributed.
> The number of people with hardware to run a model like that is very small.
That's not the problem anyone is facing, and analysis anyone is doing. At all.
The problem is the huge AI-related budget that companies who dove head first onto AI are facing.
Therefore the problem everyone is faced with boils down to whether they keep paying the likes of OpenAI a de-facto business tax, or they replace that tax with a one-off CapEx invoice followed by a utilities bill.
And you don't need a frontier model to do most things. In fact, the biggest challenge in using local models in agentic coding is the models supporting these agent-based tools.
> It's going to be many years before hardware to run medium size local models is widely available.
Nonsense. Speaking for myself, my employer already introduced a hard cap on AI credits while rolling out hardware support for local models, and even went to the extents of rolling out workstations with 64GB of RAM as a baseline. Consequently, we are using the likes of gemma4 for a fair share of everyday tasks. This has a transformative effect on your workflows because suddenly you don't ration tokens, and agent orchestration doesn't become a cash guzzler. So far the only constraints are tool support and token throughput, but that's not a concern.
I wonder how this compares to a hard(-ish) cap on Sonnet-and-better models, and unlimited access to Haiku. That’s one of the policies we’re considering.
The average person may not even own a personal computer, has a company-owned notebook with barely 16 GB of RAM and maybe a TPU, and here you are saying that owning a prosumer GPU and more RAM than what most people will buy across their entire lifetimes is common.
I'd have to spend what, 5-10k today to run flash models? It can't be justified as long as I have Luna on 20$ sub. I'd have to be doing so much heavy unsupervised work for the economy to make sense.
The number of people who have 192GB RAM at their disposal is minuscule because there was no reason to have that much just recently unless:
1. You're a professional in some compute heavy field AND
2. You don't use a cloud or in-house enterprise infrastructure
A dual 3090 system shouldn't go above 600W wall power while running inference, even less with some weights in RAM as you need to do with "only" 48GB VRAM and this model. Unless you are running water cooling, the cards are likely sitting at 200-250W power limit each anyway so they don't run too hot. This is considering a consumer tier CPU and mainboard with about 90W TDP, not a dual CPU or something like a threadripper that can easily push the power draw to 1kw if it runs the calculations on the RAM weights. My dual 3090 setup usually drew about 200W-300W average with LLM workloads according to my HA historical data.
I'm working off a 128gb Strix system on mostly hydro and it's worth it for me. Qwen 3.8 27B and Flash Next, Gemma 4 and MOE models. It Keeps my thinking and methods local while moving at a speed I can actually read and think about.
It can be annoying when you're running something long and realize there is an issue. But hey, I'm not wasting paid tokens. Reset and try again. I've probably thought of a better way by then too.
> as a clear sign they are scared shitless of local models
Partially agree and partially disagree. Qwen 3.8 next (120b + 50b ngrams) is at the upper end of a local model, and serving it at good speeds (esp for pp) is not cheap. So no, oAI are not particularly scared that everyone will suddenly start using local models of this size / quality.
I do agree that having these models in the open is great pressure on the price point of "small" API models (like luna, haiku, etc). And that's good news for us consumers and for the field in general - more competition is always better.
And on the cost estimate, the true power of local models is the guarantee of running them on prem w/o any data going out. Price wise it is still cheaper to go API, for small teams.
The "upper end" of local models as of today would be something like MiMo 2.6 Pro or Kimi K3, either of which is significantly smarter than Qwen Flash-Next. Roughly comparable in smarts to the old GPT Terra series, not quite as capable as Sonnet. But yes, running these models locally at realtime-usable speeds and without heavy quantization would be even harder. They can be quite tolerable for plain Q&A chat and great for running multiple unattended workloads on a 24/7 basis, but that may be a bit of a niche.
Yeah, I should have worded that better. I'd say it's at the upper level of what can be ran on plain consumer hw (like a laptop, mac, etc). For ~k3 you'd realistically need dedicated hw anyways. Or a cluster of consumer hw, and that comes with its own problems and drawbacks (esp in pp which matters a lot for agentic stuff).
Kimi K3 has been demonstrated on a top-of-the-line laptop Mac running at several seconds per token (it has to fetch essentially all experts from SSD, since IIRC the dense part of the model takes around ~105GB on its own!). That's impressive enough if you only care about pure Q&A/chat or pure unattended tasks, but it's not what one would expect for agentic work. You'd need quite a bit of additional RAM to have good caching for the most common experts, and even then that would only help so much. MiMo Pro would be a roughly similar story.
> Or a cluster of consumer hw, and that comes with its own problems and drawbacks (esp in pp which matters a lot for agentic stuff).
Prefill actually ought to work well on a cluster AIUI, since it can leverage both pipeline parallelism and batching quite effectively.
I set different kinds of structures for different projects,
Complete with setting and ambiance. It’s like agents work better if they are role-playing. It’s extremely disorienting and people with marginal mental stability are going to really have a bad time. What have we wrought?
I’m running little offices of 5-10 persistent agents with defined roles and it’s crazy productive and code quality has never been better. I’ve been working on management of physical systems as well, and so far it’s been pretty solid.
Do you basically know that quality of code hasn't declined based on the number of incidents that are occurring? How do you even know what is happening?
Every bit of code gets 10-50 review passes against extensive smells, rules, best practices, etc. Naive reviews and documentation driven development is your friend. If your code isn’t 100percent testable (even code previously considered untestable, because now you can create simulators and jigs) and you aren’t spending 5x+ tokens on review over coding, you’re going to have a bad time.
It’s still 5-10x less expensive than people, and you end up with higher quality code in the end.
I'd instead say that for both it's really at 100%, and not any number in between.
Problem is that we can't define precisely what is good enough or when software is "finished": I mean, we are trying to do that with human languages, so it should not be a surprise.
Yes, LLMs are now similarly aware of the average context a human would be aware of, but for anything specific to the situation a human has better chances of resolving the ambiguity.
Is there any great documentation you have followed to build your workflow or has it been experimentation? I've been experimenting a lot and gotten some cool things done but every time I read up on what everyone else is doing I realize I don't have much imagination and so I miss out.
I’m afraid I just grew my process organically, starting with cutting and pasting code into chat lol. But now it basically runs on its own, and I check my issue board every few hours for judgement calls on ambiguous decisions or redirecting in the rare cases that even bringing in independent reviews from other models can’t solve a problem. You have to have mechanisms to stop thrashing, or you’ll end up throwing away a lot of tokens. I use the second attempt rule. If something can’t be solved in the second attempt (which included at least one naive review) then they call in the gatekeeper agents that gate merges and commits based on code quality and standards compliance for a pow-wow. If that doesn’t result in a solid resolution or there is a structural problem that cannot be resolved without major rewriting of other code, then it gets put on my board.
It’s basically an office of persistent agents with different defined roles and goals. The gatekeepers whole job is to refuse code. The engineers whole job is to write code that will pass. Each significant component gets its own engineer agent. There is an office manager that coordinates the agents, maintains the project documentation, keeps up the job boards, maintains scripts to keep agents pointed at the relevant onboarding documents As they come back from compaction, and manages office automation in general. Ther is an agent in charge of making sure the test coverage is as close to fully tested as possible, including refusing code that is not properly testable.
If it’s a big project, there may be an information management agent that manages the technical side of the documentation and maintains a searchable library, like an oracle. I may put in a personal assistant to help me to prioritize my involvement. Many projects also include an agent that creates mockups, jigs, and simulators so that tests can be run on code that needs a context.
I maintain another office just for research, because research can seriously sidetrack engineering. It has all of the standard staffing, but its whole job is to do research and write reports for other teams.
Agent persistence is maintained by having agents write down memories learnings and musings(not work related) and curiosities for the next instance to read in the onboarding for their seat.
Ethos, standards, practices, and policies are also in the onboarding process.
Well, yeah sorta since the only reason appliances connect to the internet is to steal data. I mean, if you buy a connected x that normally would not be connected, it’s 99 percent there to do nefarious stuff for its real owners. It’s like having a pet lion. Sure, it’s horribly irresponsible that someone sold you a pet lion, but. Uuuh you bought at pet lion. What did you think it was going to do?
How does this analogy go when people buy a house kitten and it turns out they have been sold a lion cub? Most people simply do not have the tech literacy to understand that what they a are buying is actually a lion, they thought they were buying a coffeemaker with some cool features. It's difficult to blame the victim when they would need to spend hours trying to understand why the thing mapping out their local network is something that they should even care about
In that context the person is a fool who shouldn’t be in charge of another life, because they’re incapable of basic prudence.
I don’t actually think that applies to coffee makers spying on people though. People shouldn’t be expected to understand how computer networks or ad tech spying works in the same way that literally any child or idiot should know the difference between a lion cub and a house cat.
In the analogy, there is no such thing as a house kitten. They are all cute and cuddly lion cubs. Society needs to develop a deep awareness of this, in spite of the ocean of fraudulent advertising to the contrary. (individual-liberty-protecting regulation like the GDPR would be nice too, alas)
Since most appliances now contain a general-purpose computer, it would be unfair to say that a device is incapable of hacking or hosting malware, because any device with the given sensors and radios and capabilities can be essentially reprogrammed at any time.
So, if we're looking at smart TVs with cameras and microphones and Wi-Fi and Bluetooth and all the connectors, or if we're simply looking at a an ordinary network device, they all fall under the umbrella of general purpose computer, and there is no way to trust their maker, or some equally capable programmer, not to turn them malevolent in some future update.
I don't view this as an issue of terms of service or of software or of your manufacturer. I view this as an existential and fundamental problem with dropping general purpose computers into your home and behind your DMZ.
Consumer operating systems like Windows and Apple have all kinds of countermeasures against this malicious use. But without the proper introspection and without the proper safeguards, a device that looks special purpose but is in fact general purpose is far more dangerous.
Getting technical - the way I see it, the problem arises from a combination of three things - sensors/access, Internet access, and source of software/authority.
Sensors/access is unavoidable, otherwise the device doesn't actually do anything useful. The point is it sets the scope for what the device is able to affect. When people say "set up a separate IoT VLAN" (that still has Internet access) this is basically what they're addressing - how a device can access other devices they may care about more.
Internet access is the catalyst that's created this whole dumpster fire - I don't care about the proprietary software on my keyboard/mouse/UPS/monitor/GPU/etc to nearly the same extent. I've got some TP-Link plugs that I control local network only. They don't get Internet access, so no updates, telemetry backhaul, etc.
The authority to update/configure/change that software is the crux. With proprietary software, there are no cuddly kittens period. Here we've got a case of a "legitimate" company choosing to be a bona fide attacker to increase their bottom line! The harm was exacerbated by a bug causing it to run amok, but even without the bug they are deliberately violating trust.
But even libre software can fall to security holes as well. Meaning you want to centralize the attack surface as much as possible, for administration's sake of keeping updated. "Internet" of things is basically the direct opposite of this - postulating many illegible fine-grained links between devices on different networks. Whereas really need more like the Home Assistant model, where peripheral devices may communicate over the network, but it's only ever over the local network. Think how ethernet is set up when used in industrial control networks (or at least how it should be set up, hehe).
How do you feel about thermostats? Are some things worth it? I've had a "smart" one for the past five years, part of a new furnace install, that I've stubbornly refused to connect to my wifi. Of course this means if we forget to turn the heat down while no one's home, there's nothing to be done about it.
It’s also not a coherent view to think that the welfare of the planet is more important than one single human life and then to go on living in the modern way. We live in a thick soup of duplicitous thinking. Let he who is without sin cast the first stone, and all that.
In the USA, they don’t disappear, they commit suicide in their car typically. In Russia, they fall out of windows and off of balconies. I don’t think it’s accurate to call the circumstances equivalent, or the regimes equivalent. But there are some remarkable similarities, and their goals are similarly structured - to keep those in power powerful and to enrich the wealthy.
I spin up “offices” for different projects, using a documentation heavy approach with procedures, policies, standards, and processes. Agent onboarding and orientation, etc. I usually have an engineer for each separate part, (one for a simulator to simulate the hardware, one for the user application, one for the data analysis and evaluation tool, one for the firmware on each type of device, one for schematic and board reviews, etc. ) then I’ll have an office manager in charge of policy and issue boards, agent rosters, etc, and a engineering governance agent that makes sure code is compliant and documentation / code is coherent before any merges. 6-15 agents in each office depending on the complexity of the task.
It sounds like open code could be pretty handy but it would nerf my Claude subscription (api!=subscription rates). Understanding my workflow, what models do you think might be suitable for those tasks outside of OAI and Anthropic?
Right now I’m running 2x 3090 and 128 gb ram on 3.8 flash-next180b (the quant is 97gb) and it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit. For agentic work it does slow down quite a bit when the context fills up… but what do you want for running a 180b model on consumer hardware?
Right now I have it set up as an administrator for our asterisk PBX and radius server, and it does higher level judgment on our cctv footage for notifications to security. It’s been really good so far. Agents are the new UI, everything just works over SSH now.
reply