The last 3 updates have all come immediately after (within a few days of) a Chinese open-weight model getting close to the top. The changes are big enough that they were undoubtedly working on them before the new model released, but the timing is quite suspicious. For one of them (4.1 or 4.2, I forget), they literally said that the changes were planned for 5, but that they deployed them early as they did not agree with their own rankings.
At the very minimum, AA changes what they define as "intelligence" rather frequently and it always benefits OpenAI and Anthropic. To me, it feels like they use Astra/Opus as their baseline and tweak weights of benchmarks until the intelligence index matches their preconceived opinions on model intelligence (as opposed to crafting the index in a vacuum without any context on what different models are good at).
If nothing else, it points to these aggregate benchmarks having no idea how to measure actual "intelligence" and instead tweaking weights of benchmarks until it matches what they feel it should be. In practice, I think a better system would be to have many benchmarks within a particular space (coding, agentic work, science, etc.) and scoring each category separately and without weights applied to benchmarks (and simply removing anything once it becomes saturated). Then, each user can decide which benchmark suite matches their usage and use that to make an informed decision instead of having to trust someone else's opinion of what "intelligence" is.
With that system, you can still have benchmaxxing, but I'd imagine it would mirror actual usage better and so be less of a departure from reality.
Artificial Analysis does have multiple different indexes, but for example the engineering index has an even worse difference between American SOTA and Chinese SOTA.
And those indices still have arbitrary weights that do not reflect real-world usage, in my opinion.
The index you're referring to has 30% of its score from "Engineering knowledge". Functionally, a third of that index is just trivia which will reward large models. A small model with web search for knowledge does just fine in practice, but will be penalized heavily in this index.
Then you have science-specific as another 20%. I don't do that stuff, so now half of that index is useless to me.
Reasoning and agentic is the remaining 50% and is all I functionally care about. That index is the closest to my usage, and yet half of its weighting is irrelevant to me. The other option is the "Coding agent" index, but that couples models to agents and they don't test my preferred agent.
I'd much prefer a benchmark site without these arbitrary weights and that let's you pick and choose which benchmarks to include in the summary. Heck, it could even allow you to set your own weights to each benchmark/category.
That's not what they did, though. I have no problem with removing benchmarks from the index when every model is getting 80%+. My issue is that they re-weigh things arbitrarily and it seems their goal is to have a nice chart to show rather than have an index that would accurately represent real-world usage.
Specifically, they've made knowledge recall the largest part of their indices. That rewards larger models, but has disproportionately small real-world gains as a small model in a harness with web search will perform just as well.
Take DeepSeek Flash 1017. It's half the score of Fable on the Engineering index. Fable is not twice as good at any engineering work I do.
It's usually against the license to redistribute modified binaries. There are few pieces of software (I cannot think of any) that allow redistribution of modified binaries, but aren't open source.
As a result, in practice, the only example I can think of is cracks for pirated software.
> When you hear about the work Anthropic researchers are doing, they seem to be using hundreds or thousands of agents to solve problems.
There's a reason they're Anthropic researchers. The value proposition tanks when you actually have to foot the bill for the tokens you use.
Subagent flows can improve overall performance, but it generally also comes with disproportionately high token use. That gets worse with parallel subagents that are often doing the same things at the same time and cannot share a cache.
So the point I was trying to make is that I'm maybe in the 25th percentile of developers and am using multiple agents for a task, and Anthropic researchers are at the 99th percentile and are using thousands of agents per task.
So the median developer is probably using 5-20 agents at a time for a task. I'm not sure if that crosses the threshold of being a "fleet of agents".
I'm curious to see your source for these numbers. Your flow sounds closer to a 99th percentile one than a 25th. I find it hard to believe that a median developer is using 5-20 agents at a time.
Yeah... N=2-3 here but my team prefers having 1 main agent and maybe a couple more finely tuned ones for review / analysis / security... I generally have 2-3 chats going on different tasks but more than that and I lose the plot.
the way I read this passage - author claims to be not a particularly skilled or fast-paced practitioner, and yet it is true that author is using agents like this. Next, very advanced researchers -- in the heart of the AI Frontier industry -- are using many more agents with much larger actions.
The "median developer" refers to the particularly skilled or fast-paced, not that there are continuous numbers of every practitioner using N-agents
is that before or after all the banks and wealth funds who have been paying to subsidize AI go completely broke or sell their "investments" at a huge loss?
AA has also lost all credibility to me with all of their index changes being unannounced and following an open weight model getting close to the top spot. I gave them the benefit of the doubt for v4.2, but to do this again immediately after Kimi K3.1 got into the top 5 is a little too suspicious for me. For Opus and Astra to stay at about the same score while everything else drops feels like they are adjusting weights specifically in their favor.
Even in the optimistic case that this isn't some conspiracy, it highlights how arbitrary their scores are. I had trusted their previous ratings and their new index shows that even AA doesn't agree with them. What's to say this new index is any better and won't just change drastically again?
It's true. It says a bunch about their competitors where Nvidia isn't at their a game and have been offering 8Gb cards for a solid decade and nobody out competed them. But also gamers pride themselves on how cheap they are and how they can get a deal, it's not surprising that companies simply just give Nvidia more cash.
> is that true? if so that's wild, since they're still dominant in the gaming market as the premium hardware relative to competitors.
Doesn't seem that wild to me considering it is less that they abandoned the gaming market to any strong competitors and more that the gaming market is both basically irrelevant compared to the money being shoveled into AI and is also to some degree being strangled to death by AI because of the overlap in supply chain inputs.
Nvidia offer geForce game streaming and they do provide the hardware for Nintendo Switch console, both Switch 1 and Switch 2. And the keynote address at Nvidia's annual meeting just last year was all about demos of games with AI upscaling tech as well as LLM-driven agentic character development...
It's just slop. Tried to install it via flatpak. It has flatpak instructions in the readme. It does not ship the flatpak files it references in the readme.
There are maybe 30-60 million software developers and software developer adjacent people on this planet. Out of those probably 30% are late AI adopters, laggards, that haven't even used a terminal client and some don't even use AI.
Also a lot of people - developers included, just don't like command line tools.
Then pi is a secondary harness after Claude, Codex, OpenCode. I imagine the likelihood of pi having more than a few hundreds of thousands of users is remote. It's basically the Emacs or Vim of harnesses.
True, but I think even OpenClaw usage has falled off a cliff, plus that makes it implicit usage, most OpenClaw users probably don't care either way, it was chosen for them and they probably don't think much about it.
No, I definitely do not think millions of people use Emacs or Vim regularly. Vim is used more as it's available for quick config file editing on Linux.
I've been working for 20 years and I've met exactly 1 person daily driving Emacs (at least for a while), probably 10 people daily driving Vim and maybe low hundreds side arming Vim (maybe 5 for Emacs). And I've met or worked with low thousands of people at this point.
If I had to guess, probably 100 000 Vim daily drivers and maybe 20 000 Emacs daily drivers, both for extended periods of time. Dabblers probably 2-3x that at any time.
You'd see a lot more Emacs users (comparatively) in academic jobs than, let's say, web development.
As an anecdote, I worked for a network monitoring company, 90%+ of the devs were on vim. Later, I worked in a run-of-the-mill SaaS, 90%+ of the devs were on VSCode.
I'd think that (neo)vim is quite popular. Emacs, less so, that's true.
But according to the Lindy's effect, I wouldn't be surprised if VSCode disappears before Emacs and Vim.
Especially as heavy LLM users are opening their text editor less and less.
> but where's the outcome dataset justifying this? how did they get that probability? what was the specific methodology of the tests? what variables did they account for?
Just a hunch, but it might be because of the 51B parameter n-gram embedding. At 125B, you'd expect ~16gigs for a 1-bit quant. Add 51gigs for the n-grams and you're not far off the actual size.
If that's true, it'd scale linearly with number of bits in the quant with an offset of about 51gigs. So Q4 should be a bit bigger than 82gigs, I'd guess in the 90s (as opposed to a ~280gig q4 if the whole 70gigs of the 1-bit quant scaled linearly).
Sorta, but you're off by one layer. You can store the 18B in VRAM and stream the rest from RAM. There's still a performance hit relative to storing it all in VRAM, but it's tolerable.
Generally, for local consumer use, these large MOE models are best for unified RAM systems like DGX Spark or Mac Studio.
At the very minimum, AA changes what they define as "intelligence" rather frequently and it always benefits OpenAI and Anthropic. To me, it feels like they use Astra/Opus as their baseline and tweak weights of benchmarks until the intelligence index matches their preconceived opinions on model intelligence (as opposed to crafting the index in a vacuum without any context on what different models are good at).
If nothing else, it points to these aggregate benchmarks having no idea how to measure actual "intelligence" and instead tweaking weights of benchmarks until it matches what they feel it should be. In practice, I think a better system would be to have many benchmarks within a particular space (coding, agentic work, science, etc.) and scoring each category separately and without weights applied to benchmarks (and simply removing anything once it becomes saturated). Then, each user can decide which benchmark suite matches their usage and use that to make an informed decision instead of having to trust someone else's opinion of what "intelligence" is.
With that system, you can still have benchmaxxing, but I'd imagine it would mirror actual usage better and so be less of a departure from reality.
reply