Interference optimization is a solved problem with no moat. Anybody can achieve this result, optimize for a particular hardware, optimize for a particular problem. In general, software is no longer a moat.
The moat is custom inference hardware, lower power, higher throughput, etc.
Perplexity looks bad and unsloth just did some good quants but they don’t perform as good on my asymmetric old dual 3090 box.
However, actual quality evals show literally zero degradation in “hard math” questions against full hosted model at a cost of 1.5x - 2x CoT tokens. I expect other quants to suffer from this too. This behavior seems to decrease as precision increases. I also suspect it’s the specific tuning a what experts I’m picking.
Nothing is stopping it. This model is woefully bad at accurate creative red teaming however. GLM finetunes on the other hand are pretty good. And I'd bet they are already deployed and doing all sorts of deeds.
One reason is that the llama.cpp team (GGML) has strict requirements that a human must understand the code they are contributing. If a project is fully vibe coded they can’t contribute. So a lot of projects where an AI went and coded a bunch of custom kernels to increase speed are left to their own devices.
I think this is a fine behavior. We can have upstream purists that are strict gatekeepers but don’t get in the way of downstream forks. Debian has some this in the Linux landscape for a long time, and it has enabled Ubuntu, Mint, etc. to flourish without compromising themselves.
> A proper code review usually takes something like one hour per 200-400 LOC and you should be spending at least that much time on code review alone.
Not only is this not enforceable (how do you enforce how long someone spent working on a codebase on their own local machine?) the metric is severely off which instantly makes me question the competence of the llama.cpp dev team. You can easily review 10-100x that in an hour, even if you're being super pedantic about it.
I also just ran _one_ of their files (with include deps) through Astra and it detected >100 vulnerabilities/correctness errors (with over 10 outright UB/memory corruption issues). It's actually outright shocking.
> question the competence of the llama.cpp dev team
Following them for years, they seem extremely well put-together, and have excellent judgment. They are using the same policy as Linux and Debian (in my words, the speed of light is human understanding and judgment). Whether it is reasonable is a different question from enforcement, which typically comes down to "this seems fishy, explain your reasoning".
As for code review, the rule of thumb I've used for decades is: it takes about as long to review and understand as it does to write. Your 100x metric is completely outside of anything I've seen in any hobby or professional project, ever.
I'd like to see specific files you scanned and specific vulnerabilities cited.
If you can thoroughly and accurately review 100 * 400 = 40,000 lines of CUDA kernel code per hour, I know roughly 1,000 people who would love to hire you right now.
What you have quoted is a sentence elaborating on the requirements listed in the document. This is provided to help you better understand why the requirements exist and the goal they are trying to accomplish.
The reason you SHOULD take that time to read the output is because you must read it to understand it.
And the way this is enforced is explicitly called out in the document (and again in more detail in the linked AGENTS.md): the maintainers may ask you to explain it.
Frankly, I don't believe you. I'm half decent at writing CUDA directly (a holdover from a project a few years ago and it is a nice skill to have), the degree to which these are optimized is unlike 99.9% of all other code out there and even a tiny slip-up is either going to kill your results, your performance or both and if you're lucky only in some edge case. Understanding this code is hard work. I made a couple of minor edits to some .cu files in llama.cpp yesterday because I have a pretty weird setup which they obviously did not anticipate and it took a couple of hours to get it 'just so'.
> You can easily review 10-100x that in an hour, even if you're being super pedantic about it.
I think spending 9-18 seconds per line of code is a good baseline for review. There are diffs where you mostly moved a lot that can go quicker. There are also diffs where I spend 20 minutes thinking about 5 lines of code.
> What you really mean is, the core team there doesn't want to lose control.
It's 100% this. They basically produce vague guidelines such that only the core maintainers are allowed to use LLMs, under the guise of "well of course we understand the code" and no one else is. It's also completely unenforceable, how are they going to prove whether someone understands the code or not? Even if they show sufficient evidence/understanding the maintainers can simply sabotage them and accuse them of using an LLM to explain the code. No one wins here.
The "mainstream" inference engines are notoriously slow to integrate this stuff, to an extent understandably given the complexity of ensuring numerical accuracy alongside supporting a wide array of systems and models. Part of it is that not everyone is willing to bring what they develop into a pull request because they vibe coded it and don't care to deal with whatever quality requirements the more well known inference engines have.
Hah, but of course the official documentation will state to launch it in "set it and forget it" mode by defining the ultimate goal. This way it's guaranteed to waste maximum amount of tokens.
On my local inference box I have a perpetual codex thread open in my llama.cpp checkout that I periodically ask to take a look at currently pending llama.cpp PRs, do some research on latest MTP, Dflash and other prediction or attention optimizations, do research on the latest model quants and finetunes, take a look at localLlama Reddit threads and just do essentially a sweep of the frontier.
Then it rebuilds latest llama.cpp, grabs the PRs it finds relevant to test against, and then it performs a benchmark and finalizes the upgrade and verifies what model, variant, or even a separate finetune that we should be running.
Occasionally, it performs its own optimizations and commits, which then gets superseded by pull requests and merged code that essentially validates the model's own optimization directionality.
Yeah we heavily leverage coding agents for optimizing our kernels. Since it's highly verifiable and takes time to measure we often leave multiple running and improving performance on different model architectures.
Definitely still helps to reference relevant academic work as well, or even just encouraging the agent to make bigger structural leaps, otherwise it will often get stuck working on low impact micro-optimizations.
I don't let it commit the optimization unless it is larger than 5% and no regressions. If it can combine two optimizations and get above 10%, it is allowed.
Yes, I was also surprised with this narrative because that's exactly how I have treated my drinking years, And I very rarely see this type of honest look back in the community. A lot of self-reflections are either self-flagellating or vilifying alcohol.
I find it very positive for myself to basically look back at any period in my life and be honest about it without re-purposing it into something that it wasn't.
> The network protocol used by Radicle does not give the confidentiality it was expected to give. Anyone who can observe the network path between two nodes can read the data they exchange as the data is sent in plain text.
Is this a... design choice? This feels like too egregious of an omission to be a regular vulnerability here.
The moat is custom inference hardware, lower power, higher throughput, etc.
reply