> So your flagship model must be fastest and most accurate.
That's rarely the way anything in life works though. Do you prioritize fastest OR most accurate? That's a question that depends on context, task difficulty, etc.
You can use a 17M parameter NLI model for simple stuff and get orders of magnitude faster decisions. But the intelligence of that model will be quite a bit weaker.
In some tasks you might be talking about very small differences in accuracy between that model and a 27B parameter LLM decision model for several orders of magnitude in speed. In other tasks, the 17M might be about as good as a random guess.
One of the biggest issues with using a general LLM like Sonnet is calibration. I think Jev really helped bring this to light. With Sonnet, you get an answer and that's it. It's really not that great for classification, especially tagging.
I've trained a lot of NLI models, which do similar things to decision models. A 1B parameter encoder is about the same capability as a 2B LLM, but if you have multiple tags to check, it's not as efficient because you can't cache the bidirectional attention. On the other hand, they do seem to do a bit better on calibration.
Decision models are much cheaper/faster to serve. For classification, you still need to do classic classification ML evaluation, but for many problem spaces you can get to ~90% of the power of a custom classifier model with an off the shelf decision model with just some prompt tuning.
Zero-shot models like Jev or others can still be difficult to prompt tune. A lot of times you are describing how you want things to be classified (called label verbalization), and miss nuances about how the model does interpretation of that.
I have started working on models that can both do a classification without explanation, but enable a "think" mode that briefly explains its decision. So if you want to keep track of failures and figure out why the model doesn't like them, then you have a direction to try fixing it.
I'd like to try wrapping this in some kind of GEPA-like process to automate tuning based on your labeled examples using the supervision of a larger reflection model.
> it's a 65B dense model and inference will be prohibitively slow on consumer grade hardware
Times are changing and with MTP and DFLASH, dense models are back in style. For qwen 3.8 27b, you can get great token generation rates with MTP and for quality, it's much better than a MoE of comparable size.
That helps, yes. But speculative decoding that requires a secondary/draft model means a trade off (smaller KV pool), so often times the only option left is MTP unless you have lots of VRAM. And as you already know, these options (MTP, DFlash, DSpark, etc) are also available for MoE models, too. If it's 3tk/s instead of 1tk/s, we can say it helps but not sure how many people would be excited about that. I'd be more excited about 150tk/s instead of 50tk/s!
I have reservations about AMD due to repeatedly poor Linux support. My 8700GE APU has display freezes and can't recover 1-2 times a week. For a chip that was released 2.5y ago, that's nuts.
Is that a known issue? I had something vaguely similar happen with a GPU that had intermittent hardware failure.
Granted their rocm support was abysmal several years back and they lost my trust as a result. But nvidia is hardly any better and intel doesn't seem to be a viable alternative ...
> Did an AI tell the United States to bomb that girls’ school in Iran, which killed 156 civilians?
Yes, because certainly the people who are supposed to be in charge bear no responsibility. This article does not make very good arguments. If the US was attacked by a foreign weapon based on AI, nobody would accept that argument.
> So you’re spending billions of dollars to make big machines that might guide us into socialism?
> No! Socialism is when the government collects the wealth of the entire population and spreads it out evenly so everyone is provided for.
Sigh... ah yes, the great tradition of lying about what Socialism is in order to make your unsupported argument.
This is where I landed as well. If your company takes funding, then unless you're new and green, you should already know where it's heading. I cringe when I get contacted by recruiters touting their big funding rounds and valuations, as though it should mean something positive to me.
It also seems increasingly focused on b2b, which is probably representative of greater imbalances in the economy. What consumers can buy is becoming a slowing trickle down in itself.
The weird thing is that we learned this lesson in the dot com era, and then there was a reactionary shift to self-funding, and now we've forgotten; I guess that's just the nature of human generations.
The really odd, undelivered thing about LLMs is that if they are as productive as they seem, software people simply should not need VCs. Because you can self-fund a massive team of AI developers, you should be able to get your product ready for market in absolutely no time, and you should more or less be able to roll it out with nothing more than a tiny amount of seed capital and a suburban garage.
A bill with opposition from the president brought by a minority party that currently has no majority power in any area of government... I think this is probably a better thing to do than a bill at the moment.
After midterms, they would still need significant public support to pressure against a veto regardless. So what you call puffery, is probably a very necessary first step.
Well that's your incorrect opinion and you're welcome to it. It's wrong though.
Juneteenth national holiday, Emmett Till Anti-lynching act, Inflation reduction act to name a few. Plenty of budget resolutions are also made that eventually become law.
That's rarely the way anything in life works though. Do you prioritize fastest OR most accurate? That's a question that depends on context, task difficulty, etc.
You can use a 17M parameter NLI model for simple stuff and get orders of magnitude faster decisions. But the intelligence of that model will be quite a bit weaker.
In some tasks you might be talking about very small differences in accuracy between that model and a 27B parameter LLM decision model for several orders of magnitude in speed. In other tasks, the 17M might be about as good as a random guess.
reply