I believe there is constrained decoding available for Qwen 3.8 in vLLM. But mileage may vary: although you guarantee syntactically-correct tool calls, you'll want to observe the overall quality of the model with and without constrained decoding. In the case of Kimi, there is a clear recommendation from the creators to turn constrained decoding on. I don't think this is the case for Qwen 3.8.
I mainly use Paseo for the daemon features. I run it in a linux VM in my homelab and spawn all of my agent sessions on it. Exposed over Tailscale so I can connect to it from my desktop, laptop, and mobile phone and continue like nothing happened. A really great self-hosting experience.
I don’t know how this wrapper works, but if it is like any of the classifiers I’ve had Claude build off an LLM in the past, it grabs the probabilities of the tokens you are looking for, and then computes their relative probs against each other.
Even if the LLM thinks it’s made up D is the highest probability, that isn’t part of the set.
You never actually generate the prose, only the first pass, and grab the probabilities. It couldn’t ask for more details even if it wants to. It gets stopped before the first token renders.
I guess “A” by itself would be a seperate token but my point is the model might be trying to say something that begins with that letter rather than actually answering.
You’d need to use an approach that links the output to a closed set.
A is significant in it's own right. But most models follow instructions well enough when you prompt it, "Choose one of the following answers:", it will follow that 90% of the time. Use temperature tuning, and a LoRA, and you are 99% of the way there, just without the speed that Jev has.
Yesterday, just to prove to a friend that Jev isn't that "revolutionary" I extracted some image classification code that Claude had written for my private image organizer that used Qwen3-VL, and stopped the output at a single token, then used the probabilities. Input processing on my GPU was somewhere around 1000ms per image, so not too fast, but each question used the prompt cache, so followups were 100ms-ish.
That was my baseline of an untrained, non-optimized single pass classification. It would connect to my llama-server, and use the logprobs for the choices.
After that, I had Claude remove llama-server from the solution, and write it directly to the transformers, then I kept prompting it to profile and find more speed. Eventually my "Decision Engine" running locally on a trained 1B model got to just under 85% accuracy across my 500 validation prompts (images and text) not used or derived from the training set, and an 8MB image, with 10 questions with 5 choices per question, got down to just under 500ms. Pure text prompts and questions are below 100ms for 300tokens + 10 questions + 5 choices per question (average).
It did better on text than images, just because my training set included 90% text. I'll do more training and validation for images when I get home, but for now, I'm more than happy that I can get a local "decision engine" running in 4GB of VRAM and responding in under 30ms for most use cases I've had.
See the release notes: we now have experimental support for using any prefix shorter than the Linux default. We are aiming to eventually fully support (Tier 1) any prefix under 64 bytes long.
I’d imagine they’re live updating the library paths in the binary headers, so anything shorter or equal to what they’re using is a simple rewrite, but longer is more complex.
The present is built mostly on layers of the long-past :)
It makes me think of when Arch merged /usr/bin and sbin with /bin and sbin. Having them split made sense in a ton of scenarios that used to be very common, but increasingly the split was vestigial for most users.
Yea, my understanding is that the split between / and /usr used to be more or less: the drive they used for / ran out of space, so they mounted another drive as /usr. As a consequence, / became where you put stuff that was essential during early boot, while /usr was where you put everything else.
But these days, "early boot" is handled by initramfs and we've all got drives large enough to not need the split anyway.
The dedicated user isn't really required; it just expects that /home/linuxbrew/.linuxbrew exists. That can just be a symlink to your home directory. But yes, creating that symlink requires root on most systems. Brew itself doesn't require root or that pathname; you can put packages anywhere, but then many will have to be built from scratch since the pre-built packages don't work. And you need bubblewrap installed (which requires root) to use the sandbox, but again that is optional.
All that said, when you are running without a sandbox installing to a nonstandard location, not everything works consistently. I run this mode all the time. But it has been getting better.
The Pro has a nice backlight that is useful when reading at night. The only downside is the lack of a USB-C port, so you have to use their magnetic dongle to charge. Also, if you want one, I recommend buying from the official site to avoid getting one with the firmware locked (there may be a way to unlock it, but I'm not sure). Lastly, look at the competition: I'm happy with my X4 Pro, but there are a couple of others with different feature sets that are still compatible with Crosspoint and may be of interest if you want to use it for more than just reading.
Justs a heads up, people are comparing with the X4 and not the X4 classic. The classic is a brand new one released today, and it replaces the X4 on the official store.
It doesn't have a USB connector either, so it's all pogo connectors now
reply