But the article clearly describes how this SCM is just instrumental in their goal to build and serve better models. It's not some unrelated product they hope to sell.
Post-training of LLMs often destroys calibration. The objective is no longer to predict the next token, so it doesn't matter as much whether the probability you assign to next-token "A" is correct, as long as the overall answer is good.
So it's not too surprising that the most heavily post-trained frontier models are actually pretty bad at outputting calibrated decisions.
The biggest upside to using a zero-shot classification model is that it's actually zero-shot, i.e., you don't need to construct task-specific training sets.
If it needs to be fine-tuned, then why not fine-tune smaller and cheaper models that are only as big as the task requires?
I hear what you're saying. But this is not such a model. The article says that various internal teams had to fine-tune Clef to suit their specific needs. They're even rolling out a RL training product line for it.
Fair. Though in this case my actual classifier (dino v3 features fed into a very shallow DNN) works surprisingly well and is insanely faster than using an LLM -- so it really was the one shot performance I was hoping for.
That's right. There's only so much optimization that you can make to a transformer-based model and any tricks that Jev is employing, any open-source LLM can also employ.
You’re still hoping that the model will respect your with to reply with a single letter. With Jev, the model doesn’t even have a concept of replying with something else.
It would be great to see benchmarks for Jev that demonstrate the value of calibrated uncertainty.
For example, one could set a confidence threshold over which we trust the model decision, and otherwise reject. This provides a lever to trade-off accuracy and automation %.
Then we can ask questions like "What % of decisions can we automate to achieve 90% accuracy"?
100% agree. The key issue is that users think they are getting results from a specific model configuration and they are clearly not, which is fraudulent.
If OpenRouter wants to succeed as a business, they need to be auditing the providers they connect to (i.e. benchmarking) and removing fraudulent ones from their service.
I think you're being overly cynical here. First, I don't see any claim that is the world's best model for agentic coding. Second, it is absolutely the best model in terms of coding performance vs. dollar, and it's raw performance seems very close to the frontier.
How are you supporting the claim that GPT 5.6 is "far more token efficient" than Opus 5? Tokens equal, output is cheaper for Opus 5 ($25/1M) than GPT-5.6-Sol ($30/1M), and it seems to outperform slightly on agentic coding benchmarks.
It would still be the best model per dollar if the score was 2% lower instead of 0.1% lower. Would it be ok to still give it the highlight color then?
How big of a lie is too big? Especially when no lie needed to be told at all: many including myself would have noticed the tiny 0.1% deficit and been suitably impressed by the Opus 5 result.
I’ll admit this is a small deception by today’s standards. I’m one of those who believes in truth for truth’s sake.
reply