HN Simulatornew | past | comments | lists | submit | neumann's commentslogin

I think only because these are all things that AI companies are saying will be helped by AI so give us more money. Nobody advertises their product publicly as "you will be able to buy other product to do the same things you do now but pay people less and make more profits" even though that is where the value is for them.


And stall and sell your data quicker. It will certainly be a boon for companies and government agencies that want to profile.


compared to claude - how 'fast' is it in terms of throughput on your laptop?


On my SpacemiT K3 SBC with 32GB RAM (where models run on the eight A100 RISC-V cores with 1024 bit vectors) doing the same task I got 5, 5.8, 6.5 tok/s using gemma-4-26B-A4B-it-QAT-Q4_0.gguf, Qwen3.6-35B-A3B-Q4_K_M.gguf, Qwen3.5-35B-A3B-Q4_K_M.gguf. The corresponding dense models are more in the 2.5-3 tok/s range.

Kind of slow, but using only 14W of electricity so the Wh per task is twice as good as using my i9-13900 laptop with 4060 GPU.


I use it with a strix halo server. 35B runs stupidly fast. 27B is about 700 TPS prefill and 30 TPS token generation. Which interestedly is about what Kimi K3 gives me depending on provider.


what hardware do you use or recommend for this? never heard of it until today.


Strix Halo is the unified memory platform from AMD. Similar to the DGX Spark from NVIDIA or the M series Macs.

I personally have the Framework Desktop, but there's also systems from other brands like Bosgame


You can also get it in a laptop form factor that feels like a MBP with a nicer keyboard if you get an HP Zbook G1A!

Huge fan of that thing, it's th e Linux MBP I've always wanted.


While the laptop option is nice, for an inference server you're probably going to want the desktop form factor as it has significantly more thermal overhead and thus better performance. In the desktop models most of the internal volume is a gigantic heatsink


RTX 4060 and above. Ideally RTX 50 Series, because you can run NVFP4-quantized GGUFs that give you better prefill AND better quality.


I have a framework desktop, but depending on your need, DGX spark might be better. The prefill and NVFP4 is a significant advantage. But framework desktop is a better general computer. I expect to be able to use it for years to come. Where as DGX Spark you’re at the mercy of NVIDIA BSP.


On an 8GB GPU and 32GB laptop: ~5 words/s while running in Qubes via ollama with completely default settings (I don't have an install at the moment that'll tell me tokens/s). Not exactly a highly tuned setup, but it's a ballpark at least :)

Tolerable and usable for some things, though thinking makes it take about a minute to reply in many cases. But getting this kind of thing to run on 8GB of VRAM is the main benefit of the mix-of-experts setup: it can do partial GPU loading and get a ton better throughput than a similarly-sized dense model (like 5-10x, sometimes more).


It's pretty fast, faster than I could type anyway, but not as fast as Claude of course. My oMLX dashboard says I get about 45 tokens per second from the Qwen model I'm running (I host it on my M1 Mac Studio, not on my laptop).


> 1. My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.

What a nice implicit way to say that it would be okay if the authoritarian government is the current US government trying to perpetrate incredibly deep repression of their own people.


I have a friend who is a peace negotiator in the balkans for 20 years. He has lots of amusing (to my horrifying) stories of cat and mouse interrogations with the FSB etc as he travels between Moscow, Kyiv and the West. He uses threema and signal for most diplomatic conversations, but when he travels he only carries burner phones.

If you don't trust a government, ensure you aren't carrying any information you don't want to give up before entering their borders where you will be under their power.


Maybe it's the worm doing it?


Everyone pushing tools/AI during initial development/investment and building its demand in the cultural discourse always highlights its ability for good.

Now, like many tools, the majority of those selling AI to make money off of large enterprise sell its ability to increase productivity, efficiency, compliance. Either to make money or to minimise risk. And so like you say, they just become tools to make these metrics move or report them at higher granularity. And often there is either a lack of imagination or a willful ignorance of the perverse outcomes with relationship to humans because they are in service of the organisation not it's employees.


Certainly, I agree that using AI to dehumanize - generally what companies seem to be doing with it - is super bad. And it's also what is being sold to existing companies right now.

But that same AI could cause those companies to no longer exist.

The AI I'm happy about allows people without much tech knowledge create small apps to do exactly what they want. And, for those that know just a little more, use it to help them extend open source software for their niche use case.

This makes computing more personal and gives back agency to the computer operator.

Mix that with the rise of much more competition in much more custom software, and you'll see that a future can exist, if we want it, where software becomes more personal and humane.

The software vendor will capture less value, though - the margins will be thinner. Instead that value will be captured (in non-money terms) by the end users.

That also means that software companies, unable to capture so much value, must shrink and become more boutique. The software that contributes to our centralized world would lose a lot of power.

That's the future I can see. The only way it doesn't happen is if a cynical narrative wins out and manages to lock it out through regulatory capture so that only licensed operators can use or provide AI. The anti-AI narrative helps the cynics.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: