Rarely seems to make them look good whenever I drop in to the site. Far more expensive for little gain. Even within their own effort levels it looks bad, Fable xhigh is 50% more in cost than high for zero improvement across a range of tasks.
A tiny bump in intelligence score hardly seems worth it for cost, and honestly most people's tasks are not cutting edge work.
Current benches on there seem to make Sol look the best, at a certain point the subscription subsidy tap will turn off and people will be in for a rough time paying full price.
I don't think there are any that can be trusted. There's too much riding on these benchmark results so the labs find ways to game them either by benchmaxing or by influencing the format of the benchmark.
I think you mean that benchmaxxing has turned into a business model, not for the Chinese models (maybe them too, of course), but for the AI labs, and artificial-analysis is not remotely the worst offender.
Right now the big labs are buying benchmark tasks from "benchmark labs" (or whatever they call themselves) for absurd amounts of money, and making sure they perform well on them.
The incentives this generates are terrible.
The big AI labs will only buy benchmark tasks that the frontier models perform poorly on (because they have trouble finding any public ones). So the tasks are being mass-written now directly based on the models they're supposed to train. Obviously the math says you shouldn't do this. If there is a causal connection between the benchmark and the model, you lose the normal guarantee that improving benchmark scores will improve model.
In fact improving such benchmarks may make the model worse. To understand that let's just give a ridiculous example. I create tasks like "1 3 5 11 13 17, give next number". The next number is 301. These are the house numbers on some random street in Brussels, Belgium (I'm guessing the big jump is because they "merged" 2 streets, and just added 300 to the numbers of the second part).
If the model trains on this, performance will actually decrease because obviously this is not how you should reason.
BUT making tasks under this "impossible to reason towards" part is of course how you make tasks that might even be reasonable, and of course frontier models are going to suck on tasks like this. So they sell!
BUT they suck! Finding houses is actually going to work a lot better if you ignore this fact until it becomes unavoidable in one out of 1000+ streets.
And yet this is what benchmark improvement has moved to. If you look at the terminal-bench 4.0 tasks you will see what I mean. Doing better on those tasks ... does not actually make a better model BUT you have to actually go read those tasks to realize that. A lot of the new tasks largely demand you make weird jumps in your reasoning not based on understanding the underlying theory. They rely just being able to predict what goes wrong in certain cases when theory would suggest it should work, and mostly it even does work but for some reason not acceptably. They reward not doing the full work, realizing what goes wrong and correcting, without understanding why the normal reasoning won't work and correcting without knowing why. I find it very hard to believe this makes a model more intelligent. It rewards knowing the encyclopedia by hard, so to speak.
But it certainly makes it look better. Whereas the obvious, coding improvements, do not make models look better anymore.
AA must be doing something right. Ever since they released the new benchmark, you see open weight China model fans complaining about it under every release thread. Probably because it’s hard to benchmaxxed against.
But admitting their model is weaker is a no go. So conspiracy theory here we go
For what it's worth, in our own metrics and benchmarks the intelligence is a pretty good match, but the cost per task is wildly out of whack. We see the open weight models like GLM-5.3-flash or Deepseek 4.1 Flash 1017 use a fraction of their tokens/cost of the benchmarks, which changes the cost-benefit equation a lot.
> Probably because it’s hard to benchmaxxed against.
It's mainly just a composite of existing benchmarks. It would be very easy to benchmax if, say, AA shared with a preferred lab which benchmarks it plans to upweight/downweight in the next version. Just speaking hypothetically, of course.
Hence the constant switching around and running the benchmark themselves. They have been pretty transparent about which benchmarks get added or removed with every release.
I dont think the target audience for the blog is abstract algebra grad students
However would be good to add that the solution can be structural(like in NoPE), then you dont need explicit representations and it just falls out of your architecture.
When I see a title "Understanding..." I assume it will be about understanding theory not implementation or the theory of implementation. When I see a title "Surveying..." or "Understanding Implementation..." I assume it will be about surveying the implementation of that theory. There was just a title mismatch so I felt it was prudent to fill in the gap in understanding the blog did not cover.
Also, I expect the target audience of the blog is ML researchers, since he goes into some of the math, and umm... the kind of ML researcher that is not at least aware of the basics of abstract algebra is closer to an alchemist than a researcher.
If you set up mass surveillance and control before this happens, which is ongoing in about every developed nation at full speed, you can control this.
Since you cannot produce value through labor, you have no political power. Marx points out that apart from providing meaning and income, work provides political power in the sense of you being able to strike and set demands on your employer. However, given the automation of informational roles, there will be such a demand for physical roles(before robotics) that they will not be able to strike.
So you will be in a situation where you need to rely on the state for sustenance, which will probably mean that you will have requirements on you being fully surveilled and following whatever rules the state wants for you to stay alive.
AI doomers are often in a bubble where technology makes up their entire world. The digital portion of the entire world GDP is about 21%. Of this 21%, 25% is big data, of this 25%, about 40% is LLM/AI-driven, so about 2% of the total GDP.
You have 80% of the world doing manifacturing, farming, real estate, etc. The AI doomers do not see this, and think that this flips overnight with their dystopia.
Most likely you are doing to have very slow and boring years of transition with lowered informational-work salaries, etc.
If your idea of doom is just the economy, I'm put in mind of:
Sen. Richard Blumenthal (D-CT):
I alluded in my opening remarks to the jobs issue, the economic effects on employment. I think you have said in fact, and I'm gonna quote, development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity. End quote. You may have had in mind the effect on, on jobs, which is really my biggest nightmare in the long term. Let me ask you what your biggest nightmare is, and whether you share that concern,
Farming and real estate is not where doom risk comes from.
What percentage of the Bhopal economy was the Union Carbide plant? What percentage of Pripyat's economy was Chernobyl?
If all that happens with AI is it slowly displacing humans in economic roles, that's not by itself doom: although some bad outcomes exist there, it is also a prerequisite for UBI to be stable, which is about as un-doomy as I can get.
Neither of your examples ended humanity. Most people in rural Brazil without any kind of digitalization or connection never had to care or were affected by Chernobyl.
The same way, even if AI somehow hacks everything if it poisons large cities somehow with a biochemical attack, those people will go on living their lives as before and humanity will be fine.
The sole point is to show that severity of damage is not limited to the percentage of the economy of that sector.
If you can only change your mind about the potential for risk after everyone is dead, I'd say that's "leaving it too late".
> The same way, even if AI somehow hacks everything if it poisons large cities somehow with a biochemical attack, those people will go on living their lives as before and humanity will be fine.
And what if the AI is simply used to support (e.g. it's bought to run PR campaigns) the fossil fuel industry to the point of us buring all the reserves? Simply used to support the beef farmers who want to clear-cut rainforest for ranches?
Those two things combined, alone from support of any other industry, is enough to mess up not only the climate, not only global biodiversity, but is enough to imperil (but not necessarily destroy) human civilization.
Beef and fossil fuels are a tiny percentage of the global economy, perhaps 2% combined. There's a lot of room in the other 98% for outsized dangers to turn "imperil" into species-wide hoist by our own petard.
Very good breakdown, I really like the author and all of his other blogs as well. But "Do we solve alignment" is probably not a binary outcome. Is it alignment if the model satisfies most human desires and wants, but at the same time uses behavioral ques and mass surveillance to do so?
There is a swath of partial alignment scenarios with for example a paperclipper that minimizes deaths in the world by constraining where we can go and who we can see. Is that alignment? Risk-averse people would claim yes, they can have their little bubble of safety and its their utopia. Others might feel constrained.
All of this is pointless if you dont have consumer purchasing power. Or unless the only thing getting transported are materials for compute, resource extraction and power.
Absolute garbage. Completely obvious that this interpretability research was done by someone who likes proteins and the LLMs pushed them into feeling like this was the best analogy and discovery since sliced bread.
Maybe if things weren’t relentlessly sucky in exactly the same way every god damn time, you wouldn’t have to hear the same critiques over and over again.
They have altered their benchmark multiple times in the last 4 weeks (v5 in https://artificialanalysis.ai/articles/artificial-analysis-i...) and the last 5.0.1 change https://artificialanalysis.ai/methodology/intelligence-bench... pushed up Anthropic's models a bit, you can see that in v5.0.0 at the first link, Opus 5.5 has a lower score than in it gets in v5.0.1, and v5.0.1 was released just before Fable dropped.
Some part of me believes that the benchmark mix was put together to favor Anthropic's models.
reply