> And one of the hardest optimization problems of being human is deciding who to love.
I truly agree. But then what do you look for in a relationship ? Cause to any optimization problems there are constraints and an objective function to push the metaphor ?
Been laughing for the past 10 minutes, thanks for this one.
(I tried the death of my cat by car in front of my house in a Go Girl manner, absolute cinema)
Ladies, I need to tell you about the day Mr. Whiskers died in front of my house and accidentally launched my most powerful era.
It was a Tuesday. A sedan. A tabby who had genuinely never once respected a boundary in his fourteen years on this earth.
And in that moment, watching him go, something shifted in me.
Because here is what nobody tells you about grief: it will make you walk into the street in your bathrobe, screaming at a departing Camry, completely unhinged, in front of your entire neighborhood.
And THAT is when the sisterhood showed up.
Three women from my street — women I had waved at for six years without learning their names — came out of their houses. Not to comfort me. To scream at the Camry WITH me.
We did not know each other. We did not need to.
We were simply women, in various states of morning dishevelment, collectively losing it on a residential street at 8am, taking up ALL the space, unapologetically.
Mr. Whiskers would have hated every second of it. He was famously not a team player.
But we stood there anyway, four women building a table in a cul-de-sac, united by one flat cat and one disappearing Toyota, and I thought:
THIS is what they mean by collective power.
Rest easy, Mr. Whiskers. You changed the game for women everywhere.
You didn't mean to. But you never meant to do anything.
Here was mine, I thought it was very good. My prompt was: reinvigorating the desire for work after a sabbatical.
AI below ----
Six weeks into my sabbatical I was genuinely thriving.
Week seven I started narrating my own grocery runs in the style of David Attenborough.
By week nine I had developed a parasocial relationship with the pigeons outside my window. I named them. I had favorites. Gerald was getting too big for his boots and I resented him for it.
My therapist suggested I "find structure." My mother suggested I "get a grip." My accountant sent a single email that just said the word "savings" with no other context.
The thing that actually pulled me back wasn't a TED talk or a journaling prompt.
It was a documentary about competitive Scrabble players — specifically the 1999 North American Scrabble Championship — where a retired postal worker named Marlon Hill drove fourteen hours to compete, placed forty-third, and described it afterward as the best day of his adult life.
Forty-third.
He was beaming.
Something about that broke the spell. The sabbatical had convinced me that returning to work required a perfect reason. Marlon just needed a reason.
Gerald the pigeon watched me update my LinkedIn profile from the windowsill.
Optimizing speed is really the way to go.
Yet 24GB is not what everyone can afford.
Maybe we could take some of those 56tk/s and transfer into some free RAM space using MoE loading ? I'd be glad with a less than 10GB and more than 6tk/s model.
Unfortunately this is just the entry price for LLMs. With the exception of the Qwen 27B models, I personally haven’t found a ton of use cases for models less than 200B. With the right setup, fine tuning, etc, you can make small models do cool things, but hard to please everyone given the insane hardware costs at the moment and the comparably cheap API costs.
Small models are still great for lots of “simple intelligence” use cases, like annotating or summarising files and media; or even just basic chat when given web search tools.
My local NAS is private and I’m not going to send it off to APIs for captioning or metadata; but even Qwen3VL 8B does an excellent job at this, despite being quite old.
They are also really excellent for fine tuning. Unsloth and Tinker (from Mira’s TML) are great places to start.
If your use case is narrower than “coding agent for everything”, you can probably match frontier performances on that narrow domain with ~30b and exceed it with ~100b+.
I'm really excited about what's been happening couple last weeks for local inference. I feel like it all started after colibri [1] was released.
Great work !
Anyone got recommendation about what local model to use for what purpose ?
I feel like (as they were saying in moonshot blog post [2]) each llm can be an expert in its own categories and with several small local we might get good coverage for decent usage, granted each one is specialized enough.
https://github.com/antirez/ds4 coming out at the same time I started a new job and they gave me an m5 max a few months ago was the lightbulb moment for me.
GGUF is at least better than bnb. From what I know, bnb does not yet find a way to quantize MoE with enough accuracy, and maintain the dequant-MoE kernels. In the age of Qwen 3.0, people tried to make some bnb '4-bit' quants of MoE models, but actually the MoE part is not quantized. It's a pity that even Unsloth gave up low-VRAM finetuning with MoE (although they're making their GGUFs for inference), and the world of local training looks stagnated for months.
GGUF is maintained by all the llama.cpp developers. There are many quantization formats and algorithms under this container format, some are optimized for MoE (such as APEX quant), some for CPU and some for GPU, some work surprisingly well below 4-bit (and even near 1-bit). It also supports recent architectures like linear attentions and mHC.
