For sure, myself and colleague saw this nerfing. He said - new model coming. It was incredibly obvious. I also started getting a lot of "How's Claude doing today" messages at the same time, which is presumably to get feedback on the A/B
But the claim here is "the models are nerfed before a release", not "they A/B test quants / efforts". If it were A/B testing, it doesn't really make sense to correlate it to release windows at all.
They might want to see how people react to a new model depending on whether they've been nerfed.
If the nerfing is lower effort (seems to be, presumably why they hide the CoT) then they may be wondering if they can get away permanently with lowering effort on the new release provided it gives a wee improvement.
I strongly believe it will be shit like this that will have people abandon them. They're being way too cocky for a company that pretty much didn't exist 2 years ago
Its incredibly easy to roll out a compute instance to X% of users, or route traffic between compute instances
so not everyone will have the same experience. If I was saving on costs that would be the easiest way to do it without deploying, without any nefarious intent. The persons in charge of that likely don't even know about how individual users are impacted
At what point does this become false advertising, or even fraud? Imagine paying AWS for a high-performance server, but 5% of customers who order one quietly get a slower, cheaper server and still pay the full price. That's a serious claim and makes me skeptical Anthropic is doing something like that without more evidence.
I think they could easily reduce several parameters to slightly reduce cost (something like "it's the same model, a few less passes, but quality is still 95%+").
Or switch to a faster, cheaper cache mechanism or something that's almost as good, but much cheaper to run, in the name of improving efficiency and keeping up with demand. This switch to cheaper mechanisms is what it feels like to me.
It's a problem for us. When they nerf the model our productivity really drops (due to the specific use case and process we have). We've started using OpenAI and will gradually use it more and more if this continues.
It would be nice to know how they select. I max out my $200 subscription every week and I was selected. On Saturday, the Opus on my account that happened to be supervising a running live process, became unusable.
It would be trivial for them to opt-out accounts that are doing benchmarking from a planned regression.
I really really really do not trust companies on the verge of IPOing from goosing their numbers. Even if the lawyers are up their ass, the social pressures are more than enough to compel the behavior. For every good person willing to do the right thing even at the cost of a delayed career progression, there are 20 people willing to do anything not even for a dollar but for an "atta boy!" from the boss.
That's a major reason to use local models. Or open weights models on a trusted cloud provider. Even if they are way less capable, they're more predictable
In the context of public relations they would not need to ID every benchmark only the most credible or most public benchmarks many of which are openly available on github and whose problem-set could be fingerprinted.
You're absolutely right that it is a game of whack-a-mole and eventually we may have leading private + credible + popular benchmarks that ultimately dissuade the behavior, if it can be dissuaded.
Look up eval awareness. There are all sorts of subtle signals an LLM could use to know if something is shaped like a benchmark or not. But that's not a bad thing, every animal behaves differently in play vs real-life.
Yes the model can do that, but it would not be "trivial" (as GGP had said) for Anthropic to opt out accounts performing private benchmarks from this theoretical model downgrade people think is happening.
The most plausible approach would be for Anthropic to train some separate classifiers to detect benchmarks. I'm not saying it's completely impossible but IMO this is conspiracy theory levels of effort.
"Nerf tracking" is adversarial and exploratory. It's not something Anthropic wants anyone to do, and it's not something that the "nerf tracking" designers are necessarily doing in a foolproof way. Don't assume too much from them either way.
Sometimes you just have insufficient evidence to be sure of something either way, and nobody gets to win the argument or shut somebody else down.
In this case, some have reported analytical work[0] that goes beyond "gut feel" anecodotal evidence, but even that's not going to conclusive and may not be trustworthy at all.
These nerf trackers do not work. I wish people would stop posting them.
They don't detect a quantized model until the quantization gets down to 4 bits or less.
Humans absolutely do detect quantization well before 4 bits...
Try it yourself if you want. Try a small local model, both quantized and not, and I would be very surprised if it isn't immediately obvious which is which.
Not really. There’s no honest sense in which quantum mechanics is interpolated from the text of Euclid’s elements, to demonstrate the point with a very extreme example. Many mathematical breakthroughs of the past seem to have involved the observation of semantically interesting concepts, beyond the syntax of known theories.
(To onlookers this particular post makes no claim about AI’s capacity to make the same observations.)
reply