I believe there is constrained decoding available for Qwen 3.8 in vLLM. But mileage may vary: although you guarantee syntactically-correct tool calls, you'll want to observe the overall quality of the model with and without constrained decoding. In the case of Kimi, there is a clear recommendation from the creators to turn constrained decoding on. I don't think this is the case for Qwen 3.8.
We are also interested in this and may focus on it in a future post. They are two different perspectives: (1) What is the overall quality of LLM+serving stack+harness, and (2) How close does the behavior of our setup match the first-party provider's? In this post, we focused on the latter. Moonshot AI, the creators of Kimi, recommend constrained decoding to be on by default, and also serve Kimi that way. This implies that it helps overall performance, though it does not provide numbers for it. We want to serve Kimi as recommended, and ran into the challenges described in the post.
There is work that suggests that constrained decoding can negatively affect model quality, but this depends on many factors, among which the model in question [1, 2, 3]. Overall, it seems to me that there is a trend towards using constrained decoding. OpenAI's newer Responses API has constrained decoding turned on by default as well, similar to Moonshot AI's Kimi.
Overall, a fair assessment. I do want to place a footnote here that we did fix vLLM in the sense that its correct behavior for Kimi required code changes: it was impossible to fix it in the configuration. Later, the vLLM team fixed this problem upstream. Now, with the latest version of vLLM, the problem we describe can be addressed by setting the `--tool-strict-level` flag to `parameter`.
We currently focus on serving DeepSeek models on dedicated clusters for large customers. We do not offer this as a publicly available service at the moment, but we may consider doing so in the future.
I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
So if I understand it correctly, the tool calls are not actually done by the model?
In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?
Could something like that also be used in other contexts?
I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
> but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
Yes, this is what constrained decoding means. You constrain the model to only output from a set of "allowed" tokens (a.k.a. grammars).
So, if you use tool use with constrained decoding, you can be certain that the model will always choose from a list of approved tools (i.e. if tools=['read','write'], it cannot output 'execute', even if in other sessions it could, if execute was in the approved tools).
The tricky problem is that certain models work / have been trained in certain ways. And constraining them can lead to "invisible" problems. Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file).
> Could something like that also be used in other contexts?
Sure. You can build pretty cool grammars and constrain the model to do cool stuff, akin to "tool use" but more free hand. You could, for example, make it always emit a 10 point plan. For any prompt, it will give you a fixed syntax 10 point plan, always. And the plan will be constant with a plan_title and 10{plan_item, item_text}. But, as above, there's no guarantee that the individual items in the plan would make any sense... (i.e. step7 in a pasta making recipe would be "add glue").
Sorry, I'm not following. Constrained generation is something that helps you, the caller, to get back a message that looks a particular way. I'm having trouble seeing how this would work in an adversarial way. Perhaps you can elaborate?
A malicious inference engine can already return anything it wants, it doesn't need constrained decoding on the LLM side. If your client blindly runs commands in an unprotected environment, you're already pwned. They don't even need to run an LLM, they could just randomly add "tool_response:{'execute':'rm -fR *'} and that'd be it...
So the fact that got me curious is that one can steer the output, which, at least to my understanding of the tech, breaks the assumption that one can not do that.
So the thought is "assumptions are being broken here. Which ones? And what are the implications of that?"
Correct, here's an implementation example using the Claude SDK [0] but it's the same approach on other platforms: you tell the model what tools it has available, it sends back a response saying it wants to use a specific tool with given parameters, your code executes that tool then sends the output back to the model on the next turn.
Just trying to be helpful. I couldn’t see any other way to interpret the question below except as someone who doesn’t know how tool calls get done and who might therefore value a little insight. Didn’t mean to be patronising, apologies if I misinterpreted your question.
> So if I understand it correctly, the tool calls are not actually done by the model?
Exactly. The LLM is your next-token predictor, which output text and tool-calls. These calls get picked up by your client/harness (e.g., opencode) which executes them and sends output back to the LLM.
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
They are a special token that you can't just type out in text. It _renders_ as text, but cannot be input into the system as text. The model understands the difference between the textual toolcall token and the "special" one that's really just a spicy number.
(this is just made up tokenization but it's close enough to the real world)
If the user types , you get < (token id = 32), tool_call (41), > (102). It never decodes to , which is spicy number 12345, for example.
The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.
Seeing is a bit misleading because that's just a debug rendering if anything. The front-end sees a special number and knows it's a toolcall. The chat template uses that text form because it's easier than writing the spicy numbers there, but it also has a special parsing mode for those.
This is the very purpose of those tokens! The reason they're like that, is because you won't encounter it in random content, so during training, you can put them where you want them to end up at prediction time.
I imagine it would have to be something like that, except that I would have expected the "embedding" of the tokens to be final by that point. They would not have ever seen the tokens before, since they definitionally don't appear in any of the prose written on the Internet…?
It can happen that you would be able to inject them if there is a bug, same way sql injections work.
You have all the same pieces -- untrusted input on the left and the thing that accepts a stream of tokens on the right. In the middle could be anything, especially when vibecoded.
The first place I think I encountered it, was GPT-2's tokenizer, back when OpenAI did actual open source stuff, maybe around 2020?
I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.
The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.
The syntax was specifically chosen to avoid confusion with other tag formats. XML could show up in any user- or LLM-generated text and would mess up the message format. Some variations even used unicode symbols like "|" or "〈" to decrease the chance of confusion.
Also, back then, some people believed that it could prevent prompt injection, but history has shown that it was a futile attempt.
Would this technique work with inference optimization techniques like DFlash and MTP? In other words does it slow down things to the point where its just better to retry?
Yes, constrained decoding and speculative decoding work together, although they need to be carefully integrated and incorrect integrations give result in bugs (see point 3 under Section "Differences in Constrained-Decoding Behavior" in the blog post). Performance overhead does not generally seem to be a large problem. Of the speculative tokens, those that are not allowed by the grammar get discarded.