Normal LLM will do the classification on the text that is generated. Jev just returns the classification and confidence.
It has the advantage of speed and the confidence not being hallucinated.
But LLMs start to generalise on the pattern, rather than the classification that you want the more examples you have to train on.
LLMs start to break down as well the more classifications you have. Laya (Open source paper Jev is based on) even mentions that over 20 classifications and it starts to fail rapidly.
20 is around the level of sentiment analysis or minor intent routing. There are cheaper, smaller and easier ML models for that level of classification.
That is, if you force any llm to return json and a confidence it can also do that too and mostly likely it will he better at any one shot classification task than Jev.
LLMs have the great quality of knowing more due to the depth and richness of the training data. If Jev is trying to classify anything outside of its training data, it’s going to do a terrible job.
I fed into the hype at first. Testing Jev and Laya, they both suffer from the same issues as LLMs that stop them being useful beyond limited classifications.
I can't see any benefits that a typical ML classifier would not be better at.
Agreed. I tested Jev on OpenRouter this past weekend and it’s “okay” but a specific classifier is significantly better. It used to require skill to import sklearn (ok, not really), but now it’s literally one prompt and upload your Excel file or whatever and you can get your classifier out. It’ll run free, instant, more accurate.
This is predicated on you having training data already. I approach Jev more like Langchain -- you can prototype something new extremely fast and cheap, and if the use case works well enough, rip it out and build something bespoke. If it doesn't, you didn't spend a bunch of time curating a training dataset anyway.
I think the main argument would just be that because the model is general, you don't need to retrain it from scratch for a new problem - just tweak the input prompt. For a typical classifier there's a lot more hassle - collecting the data, training it yourself, retraining under distribution shift... In that sense Jev seems great for prototyping or small-scale use cases.
Counterargument: this works for quick prototyping, but for any serious business, you will eventually develop a benchmark/eval to track how well the general model is working, and once you have that dataset, you might as well train a specific model
Jev's bet is that if it works well enough for random use cases that nobody complains, then management won't feel a need to develop a benchmark/eval, and they won't need to employ all those data science guys.
I'd also add that they're hoping Jevon's Paradox also leads to a whole new segment of users who would have never reached for a classifier in the first place, given the barrier to entry.
Yes this is what I'm interested in. I think they might be right. I'm already finding myself thinking "well maybe a classifier would be useful here now that it's so easy to do...".
This probably just means that I could have been reaching for that tool more often already. But in practice I wasn't, and this has opened my eyes to the potential opportunities there.
And if you do get complaints or feedback on the classification, have a dev log into the user's account, tweak the Jev prompt a little until the issue goes away, and push it to production
Or not. And replace the generalist with the next generalist that gets you +15% on that benchmark for the same price, or gives you the same benchmark performance for half the price.
One advantage of using generalist models is that the generalists are improving - regardless of whether you're doing anything about it.
Yes, but the generalists are not routinely improving across all domains. The large labs are really focusing on agentic use, so I imagine that creative writing has deteriorated considering how distinctive Claude's writing style has become. Or I recently had an image-parsing task, and I was excited to try Qwen because I heard it had gotten a lot better at agentic tasks, but it failed my image-parsing benchmark.
There are focus areas, but capabilities improve across all domains - some slower than others. "Agentic use" is in itself a very general thing - because many tasks benefit from being able to leverage adaptive model-driven workflows.
Creative writing and Claude - amusing that you say that, given that Anthropic just went and tried to unfuck it in Opus 5.5 specifically. It is an example of a capability no one typically cares about, yes. No money in creative writing. But even there, we had gains in newer models.
I think here it's mostly that for normal business cases, we doesn't need to build one.
As a DevOps Engineer, I never once saw before the advantage of using a classifier. Now I see multiple parts of the stack where a better level of expressiveness will be useful (PR validations, Blue/Green validation, notification router for alerts, quick smoke tests, etc).
Nobody will give us the time and budget to build a custom classifier for these use cases, but a simple API call yes.
Using Jev as a plain classifier is the least interesting case. See robotic control, navigation, computer use examples, none of it possible with a classifier.
That's the point, they're classification in disguise. Agentic game engines/mods started doing this long ago due to the latency requirements (although they're typically using small BERT-like models that need to be finetuned, or low TTFT generative models and structured outputs). New or newly discovered use cases are great, sure.
It starts to break down once you go over 20 classifications. Which is very basic routing that can easily be done with typical ML models for cheaper and faster.
I guess I'm circling toward this view. The question is, are there things that are 1. worth doing, 2. for which jev (or jev-like systems) works well, and 3. are not worth the effort to train a custom classifier. Probably yes, but it seems like it might be a pretty narrow path. But a lot depends on #2. The trade-off between #1 and #3 is less stark the more successful one shot models are at handling use cases successfully.
Scripts and debugging, one-off log parsing or filtering.
I saw an article about 2+ years ago of a researcher using a small local AI strapped into excel to evaluate the abstract and intro of 10000 papers for "papers that research X in domain of Y", and let it loose.
jev is probably more capable avd faster than that workflow was, but saved one dude a few very grindy weeks for a litteratur review.
It's amusing how long it took, and much hype it gets for someone releasing the least revolutionary ML architecture in a new package. But i can see a fair few uses.
One of the biggest issues with LLMs is that they don't work well as a classifier. They tend to pick up on the patterns of the examples and not the intent of the examples (gets worse the more examples/intents).
I would have more contention with "Dynamic taint", but the paper doesn't appear to be AI slop at all.
As I understand the paper they are saying the reasoning/thinking you see is actually a translation of what is actually going on, and stuff can be lost in the translation. Similar to what was observed in j-space.
That would be on what the AI model generates, not what it is trained on.
The latter is where the contention is, and it's a valid argument. So much so that some companies are not using stolen information to build their models.
IBM for example indemnifies its models for its customers and has detailed information on where the sources came from to train them.
I'm not sure I understand this argument. If I go to the library every day for 10 years and learn everything there is to know about subject x I shouldn't be able to sell my skills to the world about it later because I didn't give the creators of the books I read any money?
Arguably LLM companies could have made large-scale deals with libraries and got the exact same knowledge (much, much more slowly). I wonder if people would have the same issues then? My guess is probably. Goes back to the meme that if libraries were proposed today there's no way they would ever be allowed.
> The latter is where the contention is, and it's a valid argument
It has been tried in court several ways already. Remember the lawsuit that forced Anthropic to use physical books? They tried to argue that the books couldn’t be trained on at all. It failed.
I feel people are too focused on US/China and don't pay attention to what is going on in the world.
ASML in the Netherlands for example was the only company in the world that makes EUV lithography machines, which all the major chip companies depend on. China recently reverse engineered their work to create machines since late 2025, but not sold commercially.
Ireland has chip production facilities.
EU might not be in the top 2, but it is not out of the running at all.
It has the advantage of speed and the confidence not being hallucinated.
But LLMs start to generalise on the pattern, rather than the classification that you want the more examples you have to train on.
LLMs start to break down as well the more classifications you have. Laya (Open source paper Jev is based on) even mentions that over 20 classifications and it starts to fail rapidly.
20 is around the level of sentiment analysis or minor intent routing. There are cheaper, smaller and easier ML models for that level of classification.
reply