I feel like I am taking crazy pills when reading the other responses, but come on, this seems like normal human psychology when being exposed to a new thing over and over. It feels new, it solves everything, but then over time you get bored with it. Attempting to proxy it with "account freshness" just masks the problem. It's the reason why people tried to quantify it with stuff like https://github.com/ninjahawk/livenerf because the pattern you are referring to is a known bias.
They claim they've used the same cli version, but don't publish their methodology (maybe I've missed it), eg how do I know they didn't accidentally turn on claude code auto update after all? At least https://github.com/ninjahawk/livenerf seems unchanged so far and is much more thorough...
I’ll ask the same question as when it came out: if you don’t trust the provider how are you trust them not to look for and cheat on repeat benchmarks like this.
