After hearing some good things from my tokenmaxxing buddies (or whatever you want to call them, AI enthusiasts), I got a $20/mo sub to try out Neon and it's just breezed through some hardware graphics bring-up problems I've been struggling with on Sol/Astra and Fable. I'm gonna drop my $200 OpenAI sub this coming month in favor of Google's $200 plan just for more Neon tokens.
Interesting. I've been considering switching my OpenAI sub for Google also. I think a lot of the variance in results has to do with what programming languages you're using and what type of projects you're doing, but for me, on Swift/SwiftUI work, I've done exact head-to-head comparisons of OpenAI models and Opus 5.5, and Opus is so much better, so consistently, that I'm really wondering why I bother paying OpenAI $100/month. I use ChatGPT for some ancillary projects and personal stuff, but I don't even come close to using 100% of my plan. I might give Google a try.
> I've done exact head-to-head comparisons of OpenAI models and Opus 5.5, and Opus is so much better
I am just curious, have you always noticed this trend or is it somewhat more recent-ish? I swear, a year ago or so, I was having the complete opposite experience, but something changed in the 3-6 months. I do know nor understand Swift UI, so I have no opinions on the matter, but my Swift experiences have been the nearly identical. I felt like GPT used to be so much better with Swift (and C#) than Opus.
I can't speak to the historical trend because I've only been doing substantial SwiftUI work with these models for about three months. But in my head-to-head comparisons, it's not strictly code where Opus has been better. Codex has made some very questionable choices in app architecture, etc., which I don't see from Opus.
I'm not sure if I have a rotted agent memory/context or what, but Neon unblocked me in 20 min on hardware GPU bring-up and kernel graphics pipeline work that Sol 6.1 (and Fable 5.2 on another computer) had been struggling with for two weeks. I went from 2fps to currently at 285fps in about 8 hours in the same session.
Opus 4.x really struggled with secrets rotation in sops, even in 5.5 it's constantly emitting secrets to the shell history. Thankfully everything gets tested in dev, but Claude models are the reason why we have such good secret rotation now.
We've moved to e.g. deepseek flash 4.1. I can keep up with that, and if/when Flash 4.2 comes out, great. I can grok that. Google's naming convention just shows how out of touch management is with the market.
Android's silly food naming convention works when you do one release every ~18 months and you are the market leader. It makes zero sense when you're releasing 8 models a year
We've had the Tomahawk since 1983, the Minuteman since 1962, and at a much smaller scale, the Javelin since 1996. If you want to go back further, the Germans had the V2 as early as 1944. The cat has been out of the bag almost as long as we've had the gift of flight, and predating the modern computer.
I think people also just don't know the broader capabilities of those systems beyond point-and-shoot. There are a number of systems that have had the ability to loiter and look for targets before engaging. The recent advancement has been the ability to do that on higher resolution FPA imagers but it doesn't actually change much fundamentally.
What people should actually be worried about is an AI chain of command. The weapons are very secondary to the kill chain that initiates them.
> Landmines are a rudimentary form of killer robot
I don't think I've ever heard of a landmine that autonomously hunts down it's target. AFAIK you gotta go to where it's been buried and step on it. A killer robot will come to you with or without your consent or any action or lack thereof on your part.
If the price of the RTX 6000 (96gb blackwell gpu) drops to $2000 (original MSRP $8000, currently $16,000 usd due to "scarcity") then, all of a sudden, "we have ai at home" becomes real and why would you ever pay for a subscription for your company/family when you can run Qwen Flash Next (which runs great on consumer GPU via software like Strata) forever for free?
Engineering types and power users need not apply,but For Everyone Else, the "normies", Qwen Flash Next is probably "640k is enough for anyone" territory.
IPO, valuation etc depend heavily on frontier, or "good enough" AI to need to run on commercial grade GPUs. If Qwen Flash Next is good enough and runs on a $2000 GPU, why would you pay $20/30/100/200 per month in perpetuity?
I wouldn't pick Qwen Flash Next to add kernel namespacing and cgroups v2 to my custom OS kernel, but it will happily implement the Plan.md that Fable 5.2 wrote for a dollar.
> Engineering types and power users need not apply,but For Everyone Else, the "normies", Qwen Flash Next is probably "640k is enough for anyone" territory.
Why would any of those non-power-users buy a $2000 GPU that doubles as a space heater (seriously, try sitting next to one during long runs in the summer) and thousands more for a computer to host it, plus all of the trouble of setting it up, instead of a $20/month subscription?
I do have the hardware to run Qwen3.8-Flash-Next at home, and I still heavily use my subscriptions even for simple tasks. The experience of using the apps on my phone on the go is great and I always have the option to scale the model up to the task.
The people who are going to pay thousands of dollars up front to avoid $20 are power users and people who have different priorities and motivations, like independence or ultimate privacy. GPU prices dropping won’t change that much.
Also, you don’t need an RTX6000 for Q3.8FN. There are much cheaper options right now at lower speeds.
Even if you own a RTX6000 anyway it's not obvious that it's cheaper to run Q3.8FN on it than buying tokens. Google tells me that it can produce ~1 million tokens per hour and uses 450W of power, which for me comes out to 18¢ per million tokens. That's very similar to the price on OpenRouter. Maybe during the winter when all the waste heat cuts your heating bill a bit?
I did similar math for my card (9070 XT) and it is close to break-even on electricity cost alone nevermind the amortized cost and requirement to have a desktop for it to go into. The parallelization and faster speed of the cloud makes it feel more expensive but it's just doing the same work more quickly.
The strongest case for local AI is for anything requiring privacy/discretion but I don't think individuals will collectively prioritize this. Convenience is king for the majority followed closely by cost. Businesses who care about protecting their competitive edge seem like the more likely candidates for local AI but only if ZDR providers aren't good enough.
Local AI will probably end up where it is now: in the hands of hobbyists and/or privacy-minded people who already own a GPU for gaming.
This assumes status quo. The privacy part will become more important as subscriptions become more invasive, ad-ridden, and hostile, as it always seems to go.
Local will catch on as hardware availability improves and the community converges on common solutions.
Local models can still surprise us, a small 27B could be much stronger in 2-3 years than today, while capability to run it would exist on more devices.
The workstation edition defaults to 600 W, Max-Q to 300 W. 1M tokens/hr is 277 per second, which is roughly right for a single request at a time for a lot of models, not sure about Q3.8FN specifically, but these things scale really well to 16+ concurrents, I typically get 1-2k tps on Deepseek Flash when the cards are heavily loaded. MTP (a small network that predicts a few tokens which get checked in parallel against the full net, kind of like CPU branch prediction) makes a huge difference in single stream throughput.
And that $20 subscription isn't a good comparison. That gets you Astra or Opus. Better comparison is OpenAI's $8/month plan, which gets you Luna (which scores the same as Qwen on benchmarks). At that price, most of the cost isn't even the model; it's just hosting the infrastructure.
We're talking about RTX Pro 6000s, local models on those are not slower than cloud models. With MTP, you're generally at 200-300 tps on one of the big MoE models in single stream in code generation (the small MTP models tend to be better at code token prediction than prose). The big cloud models are typically <100 tps.
> Why would any of those non-power-users buy a $2000 GPU that doubles as a space heater
Because, unlike the $20 Claude subscription, your own hardware will not respond "sorry, but I cannot do that" when you try to hack a device or game that you own.
That used to be true. But now the Chinese model companies are just not training their open weight sets on cybersecurity tasks: there’s no censorship to remove, it just doesn’t have that specific ability.
Moonshot reportedly removed some training data, which dinged its cybersecurity ability somewhat, but Kimi still has pretty strong ability - they still reported >80% on their xdev-bench-hard, down from 90% before the data removal. DS 4.1 is supposedly quite good for it (85%), and Z.ai has been talking about how they're trying to improve GLM's cybersecurity abilities, so it doesn't seem like a broad trend.
> The experience of using the apps on my phone on the go is great and I always have the option to scale the model up to the task.
Just sidenote, this experience can be replicated with software like Paseo or some other options.
I usually have a frontier model orchestrating between dumber frontier, Chinese, and local models, using Paseo subagents, from my phone or whatever hardware I'm near.
Though I don't lean too much on the local models, I find them to simply be too stupid. They're fun for experiments though like quizzing them against Wikipedia or making them debate philosophy in loops.
If you run it at home, it won't be realistically doing it 24/7. 8 hours per day is already 33% utilization. If the model is actually running half of that time, it's already 15%.
So, there will be always business for someone to host one in a data center, and sell the tokens at a markup, that will still be cheaper than buying something you will only use 15% of the time.
I don't think it's currently possible to calculate a reasonable valuation for an AI company using conventional techniques. It is too new, it is still rapidly evolving, nobody knows how it will reshape the industry. It's more of a poker game where the underwriters are betting on how much the investors are willing to throw into a perceived once-in-20-years oportunity.
There's a lot of use cases where you may not be personally interacting constantly but the LLM is running a lot of the time.
A consumer-hostable model like qwen3.8 flash next take longer than opus or better models to complete software builds, and I set mine off before bed, before work, and over lunch to take the next stage of some feature. It is running pretty consistently, just over more time than if it ran on the more powerful datacenter hardware that we use at work.
Let's say, you get 80 tokens/sec, so 80 * 3600 * 24 = ~7M tokens or $3.3/day at the current OpenRouter token pricing. That's ~2 years to break even at $2000.
That's also about 7kWh/day assuming 300W, so net $2/day or ~3 years.
Given how fast the hardware changes, it makes no sense at all.
Also, in my experience, Qwen 3.8 heavily relies on reasoning to produce meaningful results. So where Gemini-Flash would just give you a usable result in like 5 seconds, Qwen would stumble on a trivial technicality (like completely forget '+' and '-' in a patch), then reason for a minute or two on "hey, but what about that requirement, maybe it is related"? and eventually come to the right answer. But the speed and the token use is abysmal.
I mean, it's a cool model and it's a big step, and it's crucial to have open-weight AI, but currently it's not practical beyond proof-of-concept.
BTW, the current subscription situation is silly. The vendor harnesses force you into excessive token use, and then the subscription gives you a massive discount compared to OpenRouter pricing. So it looks like a good deal. But if you get more structured with what you want the AI in each specific task, $100 on OpenRouter can get you surprisingly high mileage.
This is a crazy take. No one pays $200/m and uses a single agent at a time, and at $20/m it would take 100 months to pay back, and we don’t know whether frontier then will be bigger, similar, or smaller models yet.
The subscriptions (or rather, the model training) are subsidized at the moment, so all of this could change in a heartbeat. But e.g. Anthropic subscriptions seem to be terrific value AFAICT.
And that’s at $2000, which is never going to happen until these cards are several generations old. That’s probably close to or below cost.
I’m all for local AI, but comparing it to the subscriptions is not really a like-for-like comparison.
$200 / month is getting you access to frontier models that can't run on a $2000 GPU and let's you run lots of things in parallel. I'll very often run swarms of 10-20 agents that I couldn't possibly run locally without likely $100k-$1m of local compute?
Essentially is the same as "why use the Cloud" argument.
I could 100% see myself wanting local inference for some local use-cases: continuously monitoring my network for intrusions, checking my security cameras every few seconds, running long running video generation models, etc.
> Essentially is the same as "why use the Cloud" argument.
The case is much stronger. You can run pretty much every "cloud service" out there at home on hardware that is practically e-junk. It really does not take much to serve webpages, host email, or a fileserver. In all cases you get something better than a commercial solution.
And if you use your AI to set that up - because people complain the level of friction to keep such solutions working, then you can make your AI costs back in cloud costs.
I get quite all my work done with Claude Pro, which is around $20 a month, even using it at work. I could subscribe to this for 10 years until breaking even with your fictitious $2000 RTX 6000. Economically this just doesn't make sense.
If it drops to 2000, then it will take 100 months of $20/mo AI subscription to amortize out, not including the time value of money or the cost of running the machine. I don't think this is a good trade for most people.
But this doesn’t include token usage? For $20/m there are serious caps on usage limits. I mean I can cap out the usage limit in about an hour and have to wait 4 hours to prompt again then there is the weekly limit. So with local usage your unbound on these time constraints but like others have said there is a power cost.
$20 a month subscription won't get you very far with current models, but yeah buying hardware doesn't make sense financially unless you've got it utilized 24/7.
It's worth noting that media piracy is illegal and therefor complicated in most places, requires a lot of manual setup, and people remember seeing fines of tens or hundreds of thousands of dollars.
Installing ollama or lmstudio on a computer with a GPU is about as complicated as installing Netflix. Computers come from the store with GPUs installed and an OS.
Putting 2k in a 3% savings account already pays for three $20 months. Include the rest of the hardware around the gpu and electricity costs and you never save money by running your own hardware. Which makes perfect sense, since cloud providers have cheaper electricity and can use hardware more efficiently by running it at higher load. The reasons to run inference locally will never be economic. Perhaps we will reach the point where it becomes cheap enough not to matter so that nonmonetary considerations win.
>Qwen Flash Next is probably "640k is enough for anyone" territory.
This is a weird comparison to make, because 640k is known mostly because of how wrong this prediction was. It turned out that people wanted more, and the more RAM was available the more RAM hungry applications were created. So probably the opposite of what you meant (that Qwen will be enough for most).
> to add kernel namespacing and cgroups v2 to my custom OS kernel
To be fair, you wouldn't use an LLM for this.
I use LLMs for code review a lot, didn't try Qwen Flast Next but the bigger models do give better reviews so just having this case probably won't apply to me.
Sol 6.1 did it last night in two sessions, one each for namespaces and cgroups, I didn't even use max. Most of my benchmarks suffered, but not tremendously. I'm about 40% of the way through writing my own CRI to attach k3s to right now.
Curious why you want to use an LLM for such a task? I’ve built a hobby kernel before because it’s challenging, telling an LLM to implement a feature would get boring fast. Using LLMs at work is different, that’s often not challenging or fun… it’s work, but for hobby coding I just don’t get it.
I realized the other day that for my day job I only need about 10 apps; macos exists as a sheet of glass to run them on: a good terminal, vs code, chrome, codex/claude sdk, terraform, helm, kubectl, cpython, sops, nvim, email, slack, a generic windowing environment and a handful of other things (linters, templating utilities). Besides having a local container environment, as an SRE I am hoping to have this OS ready to daily drive as my primary work OS. I'm about six weeks in on one of these $200/mo plans and container support is the last feature I need to add before I start doing a performance tuning campaign of the process scheduler and memory handling. Adding these features by hand is interesting, but running my own completely custom OS on bare metal for work is way more satisfying. So far bare metal support is just raspberry pi 3/4/5 but I have risc-v working in an emulator.
at the very least where i come from, engineering types are those who most value privacy, and carry the greatest disdain for direct-to-customer saas in their own personal lives. diy ethic and that. QFN is as dumb as local is ever going to get, and you can run it at acceptable speed on a machine that gets good frames in a ue5 game on high and was built with enough ram for a virtual machine or two.
> add kernel namespacing and cgroups v2 to my custom OS kernel
deepseek can run locally on my strix halo and handles this kind of thing just fine. it really is good enough already for hands off programming, and hands on looping for slightly more complex stuff.
No one other than engineering types and power users want to run their own server.
Everyone else just wants their phones and laptops to connect to the cloud and do the magic for them.
Well the $200 plan will cost $20 instead because they have the same input costs, by your theory, or they don't, because energy isn't getting cheaper. Running your cluster at home in California can easily cost $900/mo
MoE next token prediction is higher than 80%; you can store the other experts in RAM and the engrams can be streamed off very mediocre nvme. It's been this way since September. I'm running it at home (although heavily quantized, there are limits) on 48gb ram and 12gb gpu. We run it at work on the 96GB RTX at a much better quant.
The technology is progressing so fast if there's not a breakthrough for three weeks, it's considered a red flag that the technology is stalling out. I'm not sure how fast academic papers typically take to run through the whole cycle, but it's not THAT fast. By the end of next week there will be six different agents and twenty startups working on this problem, and in two months Anthropic and OpenAI will have rolled it into their core product offering after aqui-hiring four of them. Two blog posts are equivalent to a paper for the next five years in this space, unfortunately.
I bought a TRMNL (and the upgraded perpetual dev license) about 14 months ago, it's been great, lives next to our coffee machine, I set it up back then and it's been totally maintenance free, besides having to charge it about every 2 months. They're assembled in the guy's dad's barn in the US and extremely developer/tinkerer/diy friendly. Highly recommend. "It Just Works".
The feature those provide isn't immediately obvious in a static screenshot, the tabs are "magnetic" and you can take your "word" application and "stick" it's tab next to your email tab, and now you have a "tabbed application" with the email app tab on the left, and the word app tab on the right. You might have one "window" that is your front end dev workflow (firefox, vscode, 2-3 term windows) with 5 tabs, and then a second window that is your backend workflow of vscode + 4 terminal tabs, and a database workflow that is pgadmin + 2 terminal tabs. For example. I haven't seen anything like it, and I'm not sure how it handles windows with a smaller fixed window size, like winamp.
Haiku made them magnetic. In classic BeOS you had to manually position the windows on top of each other, then hold shift and manually drag the tab over from the left. And they wouldn't be tethered together: moving one window wouldn't move the other. It was annoying enough that I never bothered. By the end of BeOS's life I was mostly using the Mac OS 8 titlebar style (accessed by holding down all the modifier keys and pressing the Be menu).
Yep, reeeally nice feature, used it on Linux. After that I wondered why apps bothered to implement their own tabbing when you could just stack Windows as tabs (assuming more WMs would adopt that feature).
It would just stack the windows of different sizes, so if you hit the Winamp tab the smaller Winamp tab would be at the top of the stack. Pretty simple and intuitive.
Damn, I spent some time getting Opus 5 to do this for Mac, being proud of discovering a new UI metaphor - bringing a set of workflow related windows into one “tabbed meta-window”.
reply