Local models requires investment of about $2500 into hardware, so that you can run models at decent tok/sec. You then need to basically write custom harnesses. All of my stuff runs on a llama.cpp backend on a dedicated inference machine, and a client runs custom code built with OpenAPI library to make inference calls to accomplish things.
The latent space of models, which basically steers the token generation into a particular direction is much coarser with the local models compared to big models. Small models known fundamentals well, but do not have the breadth to realize the particularities of combination of fundamental task given certain criteria.
As such, if you want to replicate big model behaviour, you need to basically do context augumentation. How you do that is up to you for your own needs.
But my question remains: isn't something like pi.dev good enough for this? Or are you using custom harness to describe the scripts/skills/etc that you create yourself?
Yes, this is a straightforward extension for Pi to have it continually steer the model at every turn. You could write your own with Pi itself or use an existing extension.
No. All agents out there basically operate on skills, mempry, and tools. There are almost none that do context injection with every prompt. You basically need an external system that modifies every submitted prompt with additional context
I am cheerfully using Opencode and pi.dev on a local model without thinking about any of that stuff, just running it straight out of the box and it's working great. I have access to an RTX6000 on a remote machine so using it for 'free' and it's replaced Claude for my personal projects. It gets stuck occasionally and gets things wrong from time to time, but so does everything.
I find myself having to write a lot more skills for Qwen3.8-27b to effectively use tools it hasn’t seen before, versus virtually no skill-writing needed for Claude anymore. It seems to understand novel tools and when to use them pretty intuitively.
For basic tasks, yea. For more advanced tasks, where the agents running on big llms will do more actions autonomously, the smaller models will sometimes stop execution or go off in wrong directions.
Gemma4:31b is better. Qwen is really good at speed (especially with megakernels), but if its going in a wrong direction, it will go in the wrong direction hard. Gemma can self correct with its thinking.
The only reason Qwen is more popular is because it can run decently fast on MLX, while Gemma needs a discrete GPU with 24gb of ram to run at usable speeds.
Also, theoretically speaking, any small model can be as good as a big model if all they have to do is summarize text. Back before agents were a thing, we had prototype coders running of local inference models from Mistral for ai dev. The workflow was to generate llm summaries of the codebase, and a summary of changes to be made. Then we would grab search results from the web, feed it to an llm to summarize, and then finally combine all of that context into a final inference run which would generate code, then the codebase would be run automatically, any errors would be llm analyzed, and an llm would generate the next change to be made in the above cycle.
Nowdays, we can do that with pi coding agent and Gemma4:31b. But we still have plenty of custom harnesses for things.
The context window left on my 24GB card with Gemma is to low to do any major task though. Maybe small diffs or a single direct task. But Qwen has enough context for agentic tasks.
32k context is quite limited in agentic context.
Qwen 3.8 27b is much better in agentic and coding workflows.
Gemma4 31b is much better in general assistant and language focused workflows.
I don't understand why megakernels haven't become the default choice. The speed advantage is crazy. I made a few changes to the openjet megakernel to run in a dual gpu (older gen, not blackwell) box and doubled my token generation speed on qwen3.8-27B (4-bit though).
I keep debating adding a third matching gpu to see if I can push the speed a bit more and move back to the 8-bit quant, but I'm stuck on the idea of just building a Blackwell gpu system despite the cost factor. Plus a third gpu is going to going to be weird for the tensor split and P2P data syncs and my workstation is old with 3rd gen PCIe etc etc.
Architectures (like Gemma4) have things like sliding window attention that make it pretty impractical to do on a GPU, you would have to do additional things to make it work, and the talent to do that, even with LLM assistance is very small.
Ah, that makes sense. I haven't bothered to look at current low-level NN architecture details. I'm a little surprised that there isn't a bumper crop of relatively well-informed talent to do gpgpu work. Although, when if I think about it, it has always been a little bit of a niche area.
>As a result, the model can describe algorithms, reason about them, or orchestrate tools that run them, but it cannot execute the steps itself. A system that cannot compute cannot truly internalize what computation is.
I mean, humans can't compute either, so not sure what they mean by this.
Unless you have a general interest in self sufficiency, spending time practicing this is going to be a waste of time given its risk of actually happening. Thats where the llm come in. The most likely scenario isn't some crazy invasion like in CoD games, but more likely temporary dissarray, and even that is fairly unlikely considering how many things are automated now.
Yeah that's where it comes down to what kind of disaster scenario you're actually thinking about preparing for. Though there's no amount of post disaster perusing of those Army manuals summarized by an LLM that's going to help you actually implement anything in them. The skill to use the info in those books without being a liability to yourself and others requires actual practice.
Also the way groups actually survive scenarios is rarely around actual fighting and is vastly more dependent on actual survival skills, cooperation, and organizing. The lone (or small group) survivalist fantasy is mostly an American larp.
Mini PC with Occulink, EGPU dock, 750w power supply and a refurbished 3090 is 2500. It can run Gemma4:31b at 1000+ tok/sec prefill, ~40 tok/sec generation (which is slow, but a fully connected model is way better for all tasks)
if you want more specialized partial activation models like Qwen, you can get 100+ tok/sec generation.
This beats anything that Apple has by a long shot. You don't need to run large models for usefulness. Large models can get things write the first time, but thats their only advantage. Running small models that are good enough like gemma4:31b fast allows you to do faster iterations.
And if you really want, you can also do multi GPU setups if you want slightly slower model loading with usb4/thunderbolt ports.
TLDR; local ai has already been useful for quite some time if you arent brainwashed into believing that Macs are the best for running local inference.
> "Man you Apple Fans really live under a rock... ...refurbished 3090 is 2500"
Before you make assumptions, there are valid reasons for the choice I made:
I paid $1k for a refurbished M1 Max 64GB macbook that can run Qwen 3.8 27B at 24tok/sec (looks like they are about $1.5k on ebay). In my case, Apple was a lot cheaper (used) if you're looking at price-per-token. Yes, it's almost twice as slow as a 3090, but it's also half the price /performance ratio.
Qwen 3.8 27B Q4 with 64k:
3090: ~40tok/sec
M1 Max : ~24tok/sec (MLX)
In fact, the Max can even do 265k context which isn't possible with a single 3090 last I checked (without dropping to Q3-Q2)
Go ahead and run Gemma4 on your M1, and you will find that its unusable at 12 tok/sec.
And this is also not touching prefill, which is the biggest thing that you need fast, and Apple sucks in this regard compared to nvidia cards. With gfx cards, you can throw big codebases at the local models and it will happily crunch through them in very little time.
Sure, spending 1.5k on a Macbook enables you to run a local model, and if you really have a good reason for running local (like running heretic/abliterated models) for tasks you don't mind waiting for, it works. The difference is, spend 1k more and let you never have to pay for tokens, and you can run inference all day long without your computer thermal throttling, like your Macbook.
And I don't need to make assumptions when you say idiotic stuff like "We're getting close to local AI being really useful", like you have any idea what you are talking about. On par with being an Apple fan though.
A $2500 Mac Studio with M5 Max running Qwen3.8-27b or Gemma4:31b in oMLX gets about the same speeds you quoted... and it comes with a warranty and has enough memory not to choke at longer context lengths.
Gemma4:31b, which is the better model with more complex architecture (and thus can't be compiled into a megakernel), is gonna be way slower in prefill, and ~50% slower than a 3090 even with oMLX. You can't get away from much slower shared ram vs vram, and less parallelism compared to 3090.
The other advantage of an eGPU setup, you can simply add/swap performance. Dual 4070s give you 24b of ram and you can still get about 80% of the performance of a single 3090, and actually you get faster prefill due to more parallelism.
So in terms of money/usefulness, Mac always loses.
I mean, I have had great success with prototyping stuff in python, then letting an llm translate it to C. My 3090 with Gemma4:31b does this very well with a few guidelines.
Dunno how much investment is needed into language optimization anymore.
That works well for greenfield code, but if working on compiler optimisations has taught me anything, it's that there is a lot of legacy code out there in the world that would benefit from re-compilation with optimisations - but would be very difficult to re-write for speed (regardless of whether the work is done by a human, AI, or other machine tool). JIT compilation fits very nicely in this niche, as it means that existing Python projects can get (theoretically) better performance without re-writing.
You can use static type analysers and runtime type profilers to add type annotations to a legacy codebase, but the interesting bit would be if you're built on top of a large untyped codebase you don't control like, say, Django, how far it can go.
In my experiments with this, maybe, sorta, it'll do some AST stuff then it'll go "dang this shit is dynamic AF" then suggest a static type inference tool, then a runtime type profiler.
People have been calling Python a "prototyping" language for decades. I'd be interested to see how many Python projects ever actually get rewritten in a more performant language. My guess is, up until the last five years that was almost never done. Even with LLMs to help, is that the best use of your tokens and humans' time? It doesn't add a feature. It could conceivably cut costs after enough time, but only if done correctly.
> My guess is, up until the last five years that was almost never done.
Because having just the hot loop in the performant language is often deemed good enough. And there's tons of Python code out there where the "hot loop" is already automatically in C (the various operations implemented in C by the runtime) or where you can get close enough e.g. with Numpy. Consider that things like Pygame can exist; for a lot of the games people want to make, the only thing that really needs to happen anywhere near "bare-metal" speed is blitting.
Doing it currently, Python to Go. Early indications are a 50-98% increase in speed (e.g. one testcase went from 26hrs to 30mins) for network/CPU heavy workload. Conversion via Qwem, lots of testing needed but compute savings will be worth it.
Did you mean “50-98% decrease in running time”? Just calculating from your 26hrs to 30 mins seems to imply so. In terms of speed that translates to 100% to 5100% increase in speed.
Where you using standard optimized libraries (e.g., NumPy for numerical computation) or did you do everything in Python? This kind of increase of speed seems to indicate extremely sloppy Python programming.
It's the same level of testing for us for either Python2->3 or Go (it's a legacy code base, low automated coverage).
Conversion is now (somewhat) easier, the testing will be the slow part. For the speed improvements, which we also need for a migration from on-prem to cloud, it's a no brainer vs the other options at our disposal (like GraalPy).
Has nothing to do with oligarchy, has more to do with that SpaceX is providing valuable services, the disruption of which can be worse than punishing those in charge.
reply