> and the remainder of VRAM is occupied by the KV cache and whatever experts still fit.
Those days engines try to optimize "whatever experts" part and can use remaining VRAM as cache, moving hot experts to GPU during inference (but less used experts keep executing on CPU, yea).
llama.cpp looks to be cloning openai's realtime websocket API with this audio model as its first use case? I guess this model is either the first speech-to-speech of its kind, OR it's the first good-enough one. I'm thinking I need to give this more testing!
Yes. All you need is a formal grammar that defines the syntax of the text being generated. The llama.cpp repo has a decent writeup on the way that grammar-constrained decoding works in general [0], and it includes a bit of a simplistic example for generating syntactically-correct C code [1]. No reason this couldn't be extended to other things -- I personally have run experiments with generating syntactically-correct SQL statements when doing text-to-sql [2], and I think it's pretty cool.
Exagerating? I setup llm endpoints on corporate level in a company on H200 gpus, that serves hundreds of software engineers, I have observed differences, even on different versions of nvidia drivers. I shared my day to day experience. It is more and more important if you have very limited VRAM like 24gb on a 4090 RTX desktop or RTX 5090 Laptop gpu. I set up and use llms on my personal devices also, when I can't use cloud models on classified company materials. Here is an example problem what I daily report and deal with.
> Setting up local decision / classification models is as easy as telling the agent you already use "set up laya on llama cpp locally https://github.com/ggml-org/llama.cpphttps://huggingface.co/ggml-org/Laya-GGUF" (or your inference engine and model of choice, walk away for a minutes, and come back to it running on your computer.
I mean, yes, that works at dev time; but going from zero to nonzero local inference components makes an agent implementation suddenly much more complex to ship to production / drastically narrows the set of hosts such an implementation can be deployed onto.
The extra funny thing about this is that the target audience is people who use agentic AI systems (system one thinking as a compliment to system two thinking). Setting up local decision / classification models is as easy as telling the agent you already use "set up laya on llama cpp locally https://github.com/ggml-org/llama.cpphttps://huggingface.co/ggml-org/Laya-GGUF" (or your inference engine and model of choice, walk away for a minutes, and come back to it running on your computer.
The main benefit of Jev seems to be broader knowledge? But local models (especially the ones actually trained specifically for decisions, rather than ones that "support" it like embeddinggemma) are probably good enough for what most people use it for. They also seem to be quite easy to fine tune (I haven't tried, but seems to be what people say).
These three are measured against the old model of Llama 2 7B, but it also means the numbers are without any per-model quirks that reduce compute and memory BW usage, thus more or less generically applicable.
I haven't used llama.cpp for image generation either but I recall an issue about it. Unfortunately I can't pinpoint it now and there is the older closed issue https://github.com/ggml-org/llama.cpp/issues/4408 so unless mtmd supports also multimodal outputs out of the box safe to assume output is still limited to text.
Not all architectures are supported by llama.cpp . The GGUF format encodes the NN in a standardized way, but then you need code that can use that NN structure.
I understand that llama.cpp could only output text, last time I checked (I do not know how to find a good source for that though).
Thanks for all of your exploration in public Simon.
Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.
> I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s
> Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable
__VK_ENABLE_DEVICE_GROUPS=1.
So when I said "a couple of nvidia engineers", I indeed meant Jeff.
VK_KHR_cooperative_matrix - embrace?
VK_NV_cooperative_matrix2 - extend?
I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
I haven't yet, though plan to when this model is released. The models that I've been daily driving (Gemma 4 26B, Qwen 3.8 27B) have fit nicely on my 3090. I think FreeToken only offers a perf increase for MoE models that you can't feasibly fit in VRAM.
Yeah I'm sometimes unsure how to get best perf out of llama.cpp, and honestly thought it already did what the FreeToken paper discusses, but from everything I've been able to find since llama.cpp has no dynamic expert cache for GPU. An RFC discusses adding such capability and there's impressive results some are claiming from a fork, but I had to stop reading the thread, reading all the LLM generated comments and summaries from people was making me dizzy.
There's quite a few tangential features that must be implemented correctly or risk affecting the LLM output in significant ways.
Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the extra linefeed was steering the model into making reasoning self corrections that only got worse with longer sessions (more details about this issue: https://github.com/ggml-org/llama.cpp/issues/24181#issuecomm...)
No inference engine is perfect, but I feel that llama.cpp is the most reliable way to run language models locally.
Just quantizing takes seconds-to-minutes, llama.cpp provides a nice tool[0]. Improving quality is then a matter of picking specific tensors to maintain at higher accuracy, checking on representative data, and repeating.
Didn't expect this to pop up here - please ignore for now. We are preparing official semantic versioning of llama.cpp and it's almost ready, but not quite.
Yeah, I think they are strays added by GitHub Actions, they don't line up with the actual version that's tracked inside the source code, which indicates v0.20.1:
Those days engines try to optimize "whatever experts" part and can use remaining VRAM as cache, moving hot experts to GPU during inference (but less used experts keep executing on CPU, yea).
e.g. llama.cpp recently added flag for that: https://github.com/ggml-org/llama.cpp/pull/29887