I wouldn't be surprised if a good portion of these PRs are multiple agents stuck in a loop constantly overwriting each other changes, with no human involvement at all.
Previously there were two problems with NIH solutions:
- they have implementation cost.
- they are usually born from arrogance mixed with lack of understanding of existing solutions and their tradeoffs, and typically end up being a net negative.
Before LLMs, the cost is what primarily used to stop us from building them, it used to be a natural filter for bad ideas. Now that the cost is gone, it's nearly impossible for most teams to stay disciplined and not be building those.
Why wouldn't you build a software factory? If you succeed, the ROI is infinite. Who cares about immediate customer needs, we're building a utopia here. Once it's done, all customer needs will be solved. Just one more month, I promise.
It's often nearly impossible to know in advance how hard the task will be, so the model has to guess, and will inevitably miss. Just because you know how hard the task is, doesn't mean it's obvious. Often it's only obvious to you, because you have additional context that models are missing.
A bad guess can lead to either context rot (if subagent wasn't used), or slower execution (if subagent was used but it turned out to be a bad idea). Both are bad UX.
Now, we could improve accuracy by having the harness do a research on each task before executing... which leads to even slower execution, also bad UX.
> people that came up with a product people wanted
We've yet to see whether this is true, or is it just manufactured demand. There are dozens of Jev demos, but pretty much all of them are either cool but useless, or simply fake (i.e. harness doing 99% of the work).
But economics of Jev enable classifying high volume things. And I've tried all the open-weight Qwen based decision models on my Spark and none of them came closer to quality and suck at batch inference so they are doing something custom.
Silly question but is it right? I am in grad school and this semester we run multimodal data through ML and LLM-as-judge classifiers. Even plugging in ChatGPT with a relevant judge prompt produces relatively believable results. At the same time, I struggle to trust it to generalize. I’ve read that Jev does classification across many domains and is also powered by an LLM back-end? What separates it from my above approach outside of harness efforts and finetuning of the LLM? Thank you!
Measuring ARR based on 7 days of data is beyond stupid.
You can create a company with 2B shares and sell one share to your friend for $1000. Lo and behold, you own $2 trillion dollar company, leaving Elon behind.
Especially for a company still in their 15 minutes of fame that could very plausibly turn out to be a one hit wonder and a trivia question in two years.
Multiplying out revenue from the peak of their mini hype cycle while a dozen well financed competitors target them directly seems extremely optimistic.
1/3 of the Fortune 500 is likely measured by something like a single @company.com put a credit card in, against their company’s procurement and security policies.
It’s not to be unexpected for a new product and startup, but they lack SOC 2 certification, and the other things enterprises look for on information security.
I’d be surprised if many procurement divisions are approving typesafe AI, and I fear by the time they achieve it, existing vendors would’ve catched up.
What? The moment this was released we integrated decision models EVERYWHERE. If you’re not using it you’re basically like these people who think LLMs are useless lol
> Models we had before Jev had at most 2/3 of above qualities, but none of them were 3/3.
You're the one being deceptive here. Jev is trading accuracy, speed, and cost for generality. It's less accurate, slower and more expensive than trained classifiers. So it's still 2 out of 3, but with decimals. Maybe 2.2 out of 3 if I'm being charitable.
And the reason we didn't have that before is because nobody thought it's a good tradeoff.
When you say "trained classifiers", you are referring to models which are trained (or fine tuned) to work on one specific problem, right? That is the opposite of "general purpose".
Would Jev be more accurate in a specific task if it had been developed only for that task, as opposed to general purpose? Of course it would. So, sure, Jev is trading accuracy for generality. According to you "nobody thought it's a good tradeoff", which again is false, there was huge demand for a cheap and accurate general purpose classifier.
> work on one specific problem, right? That is the opposite of "general purpose".
A business doesn't need Jev for the sake of Jev. Most business are solving specific problems.
And fine-tuning got a lot cheaper these days - I've seen claims here on HN that ~500 examples is enough to beat Jev.
> "nobody thought it's a good tradeoff", which again is false, there was huge demand for a cheap and accurate general purpose classifier
There wasn't. The hope is that there was a latent demand, but we've yet to see if it's truly latent or just manufactured.
Noone is saying "hell yeah, finally we got a general purpose classifier, my business needed it so much". The typical message is "this seems cool, let me see where I can apply it".
The fact that name itself is a play on Jevons Paradox illustrates that there was no demand until Jev was released.
Yes, businesses are solving specific problems, but most businesses have more than 1 problem to solve. No, it is not economical to pay a data scientist to develop a custom model for each of your tiny problems. It is often much more economical to use a general purpose solution, like an LLM, or now, Jev.
At this point there's no need to pay a data scientist. You can literally ask Claude to do everything for you: extract real examples, classify them, post-train a model, and ship an API.
Now, it could be viable if your business has literally hundreds of problems thats require classification. I just haven't seen those.
I treat the fact that almost noone was doing that as evidence that decision models aren't that useful/groundbreaking. That, and the fact that every single demo I saw was either fake (e.g. playing games), contrived, or plain wrong (e.g. using Jev for compaction).
No, we're not at the point where you could ask Claude to do all of that, unless you have a super easy problem to begin with and/or you don't care about output quality. Feel free to link a counter example.
Maybe I'm missing something, but why couldn't it be generated by the model? In older classification tasks with transformers like BERT, you could absolutely obtain a confidence score.
The dice roll prediction is about the way the prompt is setup misunderstanding how Jev works (they treat the confidence score as a probability score, which it isn't).
If you instead give it a list of probability for each number and ask it whats the probability of each number, the result will be accurate.
That’s circular, sure. But if we think about potential real-world tasks where someone might, say, use it to classify on “does this advice correspond to our policy docs”, you’d absolutely push the answer (the policy docs) into its context..
What am I missing?
As in, neither LLMs nor Jev are truth engines. Truth comes from the provided context.. Plus weights.. kind of fuzzy, sorry I’m thinking out “loud”
You’d be much better off using a proper LLM here: more accurate, can provide justification, can be steered when mistakes are made. Jev is going to be a coin flipper with almost no knobs.
> I strongly doubt that a copycat that was put together within days after Jev’s release will be able to match it on a sun of its properties
But why? If the simplest way to achieve Jev's capabilities (accuracy, cost, latency) is by fine tuning a small model, what makes you think that this isn't exactly what Typesafe did?
And even if they did something different - what makes you think it was a good idea in the first place, given how easy their results were replicated without any "secret sauce"?
My point is that it’s not replicated. You replicate the accuracy, but not the other properties. The Jev-competitors only proved that you can get or beat the accuracy, nothing about the other properties. Especially the “zero hallucination” output and the (if it works how the documentation make it sound) prompt injection resistant architecture. You can’t get that with a fine tuned LLM.
I understand most major model providers support passing a JSON schema that is strictly followed in the output, accomplishing the same 'zero hallucination' and prompt injection resistance.
The only difference I am aware of is that probabilities are better calibrated with these decision models compared to regular LLMs which can output hallucinated numbers where your schema allows a number.
I’m not claiming you can’t get some of the properties in some ways with LLMs, just that Jev is the sum of its parts, not just one or two properties.
LLMs can produce structured output by limiting the next token based on a grammar, that ensures correctly formed output.
But (presumably, again I haven’t seen Jev’s insides) Jev doesn’t need to do that, it just has to output probabilities for each answer, and can do it natively without artificially limiting output tokens.
So jev is cheap, fast, never produces malformed output, its output always carries correct semantic meaning (but this doesn’t mean it always answers correctly), and it separates correct from prompt (which if it truly does that is the most exciting part).
Can you share a sample prompt demonstrating it? I’d be interested to see, especially since you’re saying it’s incredibly easy (20 seconds would mean it’s easier than on most normal LLMs even).
