Yeah, it’s surprising that this is the runtime model they went with as opposed to a hybrid edge/cloud model. I’d love to see a system 1 on edge, system 2 on cloud offering. They’re likely the best positioned to do this.
We’re building the inference platform for visual intelligence. We’re a deeply technical team of AI / computer-vision engineers (20+ years combined, MIT/CMU/NC State PhDs) who’ve shipped production ML infra across autonomous driving and LLMs.
We launched the VLM Run Gateway (https://vlm.run/gateway) in September, a unified API that serves open-weight VLMs, embodied VLAs, ViTs, served across a fleet of GPUs and clouds. That's exactly the infrastructure problem this role will own. We're already scaling to serve 100s of thousands of requests per day, so if this sounds exciting to you, read on.
Email us at hiring@vlm.run with your GitHub profile, papers, ML projects you’ve recently shipped (100+ GH stars only) - especially with Ray, k8s, GPUs, serverless. No AI text please, keep it short, shorter emails are more likely to get a response. No remote work, must be in the Bay Area (specify in email subject).
When you say Jev is optimized for it, I get that there’s no need for unnecessary decodes in an autoregressive fashion.
But both LLMs and Jev-like models would need to prefill, the only optimization Jev does differently is the decode which can be emulated by reading off logprobs.
We don’t know the param size of Jev, to determine the most comparable model, but if I had to guess it’s sub-100B.
FWIW you’re absolutely correct on how most developers are unaware of this as they’re mostly operating on the API level and not aware of server side (vLLM/SGLang) capabilities.
The aspect that I like the most is the typesafe API that introduces new probabilistic concepts that are more sound than json schema and constrained decoding with quasi-confidence scores. Developers were asking LLMs to also emit confidences which made absolutely no sense whatsoever.
Also used to block SSN and CC #s from logs, etc... as small and fast enough to do it. You don't want to call OpenAI GPT-6 and ask it to return your text with the SSN blanked out. I am sure people do though... (SSN is a bit simple, but all kinds of PPI in one model is more likely).
The nice thing about Jev is that people started taking about models that are not LLM text streams again.
> Classifiers (instead of LLMs) return results with confidence scores.
This is also bogus unless you are talking about Bayesian inference. No classifier can output CI for a single point estimate. In every ML theory textbooks worth their $, it's always stressed not to treat these sigmoid'ed or softmax'ed numbers as probabilities or confidence scores, there is no such thing as CI for point estimate.
This is an example implementation that was making the rounds recently. If you point whatever model you prefer at it, it’ll do a good job of explaining it.
No comment on this model itself, might be over fitted to the Jev benchmarks just to beat it.
I don’t know of one definitive resource but “constrained decoding”, regex/grammar-based decoding, json-schema decoding will give you a bunch of hits. Look up vLLM, SGLang and outlines’ implementations for more technical details.
We just did exactly this - added TypeSafe-compatible API (incl. websocket support) for various VLMs. Latency right now is <250ms, but will be able to get it to <150ms (p95).
Curious about this: do you expect that Claude science will also scoop interesting directions looking at anonymous usage, to publish posts like this before the original author (similar to the Navier Stokes debacle)?
There's no evidence that's what happened with navier stokes. By all appearances some employees heard a (somewhat inaccurate) rumor which led them to believe it was solvable and they proceeded to throw utterly absurd amounts of compute at the problem. The ethics of that are still questionable but for an entirely different reason than they were accused of.
p(y = next thinking+decision token | x = question) != p(y = next decision token | x = question)
The former is what LLMs are trained for, the latter is what Jev was likely trained on (likely used thinking alignment as an auxiliary loss, but not explicitly included in the probability calibration).
The author’s comments on vision providers is especially interesting. We saw that most providers don’t provide native video url support, have high-variability in vision performance (likely due to the fact that they’re serving different quantization levels behind the same model id).
If you’re building vision-native apps, there are so many footguns in vLLM/SGLang serving configurations, let alone the routing/orchestration in providers like OR, that leave the user more confused about the model’s capabilities.
text is mostly beaten to death, so you can expect good defaults to work for vllm. VLMs specifically are quite sensitive to quantization, especially if you want it to do fine-grained localization (time or spatial), and the vllm default params can be way off for your use-case.
For example, most vision models don't need 256K context length for modes like Qwen3.8-27B when all you care about is single-image captioning, so you can technically save on KV cache. Video reasoning does require that context length, so it's a different set of deployment parameters that need to be enabled.
All of this to say that the providers that offer these models, are simply using vLLM / SGLang, and mostly cater to the text inference use-case (coding, etc). Vision always seems to be a bit of an afterthought.
Got it, I'll test more deployment params when I benchmark models for vision tasks.
Do you have any tips for Gemma 4 31B in particular? I quite like the model but I feel like I'm underutilizing my rented GPU hard due to skill issues. Throughput should be way higher than this, right?
reply