That has always been the case with anything self-hosted. Otherwise you're putting a lot of faith in an upstream maintainer to actually put in the effort which they may or may not be doing.
They make it clear that their edge is in their synthetic data. Train your own jevs miss the point. And if you have that much data, you didnt need jev or these replacements in the first place.
HN's desire to pretend the data pipeline doesn't exist or isnt meaningful is silly.
I'd argue that a tiny fine-tuned Jev on my dataset that I can run for practically free is far more valuable if the idea is to make this a dependency in real work
People have been making their own calibrated classifiers and decision makers for a very long time. Mine outperform jev. But that's not the point of jev.
Is it though? I have found it much easier to fit/diagnose/improve simple classifiers on embeddings/hash vectorized features/etc than iterate on prompts, especially true when using the more modern LLM's (like gpt 5.6 Luna) where you can't even set the temperature to get any sense of determinism.
I do note though, that is infinitely easier to 'deploy' a Jev/LLM based solution than a data/model pipeline.
The comparison i'm making is against a training of encoder/small decoder model so I think we agree on most things. I dont think jev replaces the benefit of embeddings nor think they are mutually exclusive. All part of a handy utility belt.
What does Omarchy or Spotify have to do with anything I just said?
I am a software engineer, and using this software in my personal and professional work has lead me to a very different conclusion, but everyone is entitled to their opinion.
That's the usual response, along with "you can't compete with free". But, look at how much money the frontier models are printing, it is obvious that the bell curve of usefulness is still centered around them.
Let's also not forget that even the open models are not getting smaller, they are getting larger. Of course, you can distill them down into something that will fit on smaller compute, but at the end of the day, the data centers of compute, still play a huge role.
An easy counterargument is that the frontier models are swallowing all of the attention and money because they currently don't have a constraint of demonstrating that their value exceeds their cost. Take that away and it might turn out that cheaper models that are "good enough" and can be profitable are the more attractive route.
> they currently don't have a constraint of demonstrating that their value exceeds their cost
There is no loyalty in AI. I can switch to another model at near zero cost. They absolutely do have to demonstrate value. The concept of "good enough" is a misnomer because we're talking about putting these products into the hands of people who need to generate value from them.
They have to demonstrate value absolutely, but you're forgetting about the denominator of cost, and to a large degree so is the industry right now.
To take this argument to an absurd level, if Fable generating a feature cost $100,000 and Sonnet generating the same feature cost $1, it wouldn't be enough that Fable is marginally, significantly, or even dramatically better, because the cost difference is so outsized. For a variety of reasons, the true cost of tokens for frontier models is downplayed at multiple points (whether it's through AI companies running off funding, companies prioritizing AI adoption over AI costs, etc.)
If cost is not an issue at all, the best thing is always preferable regardless of cost. But in most cases, and AI is no exception, eventually cost will be an issue.
Open models keep getting bigger, but smaller open models also keep getting smarter. What I can do with an 8b used to require a 32b.
Also the decision makers who are signing off on things like ChatGPT Enterprise are at least 18 months behind the curve of what you can actually do with these things and how cheap they can be. They're still trying to figure out how to actually adopt the tech out of a sense of fomo, nevermind making nuanced decisions about hosting an open weights model. I see this firsthand in my own job.
I'm talking about adding facts to a model by modifying engrams or trying to bolster guardrails with J-washing, meanwhile they're still trying to figure out how to best prompt Copilot.
Give it a few years for everyone else to catch up, I'm barely able to catch my breath before there's some new development in the open source/weights space
> Open models keep getting bigger, but smaller open models also keep getting smarter.
Not only bigger, but smarter and more capable. From what I can tell, smaller are only getting smarter in very specific areas. There is a subtle difference there, that is extremely important.
> What I can do with an 8b used to require a 32b.
What exactly do you do with an 8b? I usually ask this question and either get no response or it is something that doesn't generate anything of value. So, please surprise me.
Smaller models have improved in every regard, not just domain specific challenges. You can see this in the benchmarks or by just testing one yourself side by side with an older release.
I'm using it at work in multiple data classification and redaction pipelines. It's replaced tedious manual labor and opened that staff up to focus on the parts of the work that requires their human intuition, rather than spending time on tedium.
Presumably part of the reason you don't get a response is your hostile approach to asking.
They have not improved in the one way that I use them today, which is coding. It is like talking to a halfwit, and I always end back up with codex.
Asking a direct question is not hostile. I'm glad to hear you've found a use for a smaller model that generates value. It gives me hope for the future, that said, I think we are still a long ways away from needing HPC in DC's.
Different people have different requirements. I use my local model for coding regularly. It doesn't help the conversation that things like quant, config, harness, and even prompting are equally as important to your outcomes as the choice of model.
I pay $200/month and I generate 10000000x more value of it than that. As you're saying, the actual barrier isn't just the model, it is the set up. As long as the walls are too high, $200 is going to dominate.
The internet was great before AOL, but the simplification is what brought the masses to it.
I have a shoebox sized computer (Framework Desktop) running Qwen 3.8 Flash Next. It has completely replaced my use of proprietary models in my personal life. 6 months ago I would have told you this was impossible. Based on the current trajectory I expect 6 months from now I'll have a Mythos class model at home. The best part is not having to concern myself with token cost has unlocked all kinds of experimentation and use cases. I have been pushing over a billion tokens per week for multiple weeks now, all for the $52/year it costs to keep this machine running 24/7
From the paste, it's pretty obvious that they're using Linux with a custom llama.cpp fork. The Framework Desktop uses the AMD Strix Halo chipset, which is a unified memory system with an integrated GPU. I think that it goes for around $3000 in these inflated times.
This tells me that the max throughput for the models I'm running on my hardware is lower than it actually is. Please allow us to tweak all the variables instead of locking me in to whatever rate you found by searching
reply