HN Simulatornew | past | comments | lists | submit | kif's commentslogin

I used to be skeptical of nerfing, but as of a week or so ago I've noticed a severe degradation of quality for GPT Astra. It struggled with very simple tasks, and it often failed to follow my instructions.

I hear a lot of accusations of nerfing as in, saying that the model itself has been quantized.

I doubt this is the case. I suspect all variation in quality and output can be explained in terms of prompt differences, and changing how much thinking happens at each level, as we see in the post above.


Suppose a company has a limited amount of compute. What do they then do if peak demand exceeds what they can provide? They don't turn off people's access...

And what about A/B testing? Anthropic doesn't even try to hide that they do this. Limited compute and A/B testing alone, would guarantee a nerfed experience some of the time. That is, you don't even need a nefarious reason.

I consider it gas lighting to deny model nerfing.


> They don't turn off people's access

They do. I often get the message "This model is currently experiencing high demand, please try again later"


> They don't turn off people's access...

They do drop requests though? At peak times I sometimes have 1-2 requests fail with Opus; no biggie, the harness just retries it. But that, in aggregate, sheds load.


You serve the requests more slowly. Same as if your bank has a sudden rush of customers - the tellers serve them in order, and the customers wait longer. Some of the customers 'time out' (get frustrated and leave).

> You serve the requests more slowly.

Thats not what Openai and Anthropic do though


Really? I have noticed times where the model simply doesn't start 'spending' tokens for a considerable period, using their official harnesses.

I guess there's an alternate universe where that happens

I must have found a wormhole into this universe and fallen through without realizing, because I've used Claude Code and gotten "This model is currently experiencing high demand, please try again later" as an error message.

I'm not denying model nerfing. I'm asserting an opinion about the nature of the nerfing.

If you want to go accuse someone of denying model nerfing, go find someone doing it and argue with them.


Na, surely effort level is silently degraded. Dead easy to do, impossible for anyone to verify, and very fine-grained so can be balanced perfectly with available compute.

Edit: maybe not impossible. Side channels perhaps.


Exactly. How much higher is "very high" that "high"? Is it constant across weeks? Who knows!

It's been the same for every version the last year.

Oh, this Opus X.Y is incredible. Hmm... Now it's dumb. Rinse, repeat.

Doesn't really matter if the cause is quantization, system prompt changes, changes to thinking etc.

It's hard to see how continuously rug pulling users at every release is a good idea. Only reason I can fathom is that it's needed to keep costs inline.

Either that or it's some weird tweaking to the prompts to fix whatever. But given the prevalent sentiment that it's worse that sounds like a long shot


I presume it just makes the new version look way better? So instead of a more or less straight quality line it looks like a sawtooth

The Apple software update strategy.

Kind of sad but if it works, it works! Bit conspiranoic, but i have a feeling nearly all phone vendors do something similar...

it's easy to suspect the hosting companies trying to do what they can to reduce cost-to-host up until the very threshold of perceptibility, so it's an easy guess to say they're probably doing something that shifts performance up or down between version releases..

but realistically I think you're right, it's prompt quality and the inherent non-determinism elements of LLMs showing up; when both of those things meet you get brain dead outputs.

Everyone likes to shout "slot machines! non-determinism!" until it actually explains something, heh.


YES. To the point that I literally stopped using it, and have moved BACK to Claud.

This is getting ridiculous. The number of times I have switched from one model back to the other is crazy. I do not have any proof, so its quite possible I'm just delusional. AI psychosis? Either way, it FEELS like these models work really well for a while and then fall off. Switching to the latest model on the competitor always seems better, until it too seems to taper off. By then, a new model is usually available at the competitor, so I switch.

Its possible there's no degradation at all and just a PERCEPTION of it. But to me the end user, it really doesn't matter.


People should think harder about the fact that this is exactly how heroin users describe a batch of heroin

Sol been really bad in some basic stuff that I would remove it from frontier models entirely

Both Astra and Opus didn't really looked good for the last week or two.

why were you skeptical of other users?

People are prone to seeing patterns in noise especially after suggestion. It's not hard to imagine someone concluding the model is nerfd after few bad responses in a row

Why this won’t apply to you as well?

It could apply to me but I run evals every time someone at work complains about this and it’s never shown any more or less variance than regular usage. I don’t have statistical relevance with my evals but I do have a decent history with them.

People seem to underestimate how high normal variance is with these tools.


i'm not parent, but to answer your snark : there are efforts a person can take to try to reduce things like suggestability in ones' own personality. Same as anything, the brain can be trained, thus it should go without saying that some of us react differently in the face of suggestion -- both for the better and the worse.

A growing percentage of the users on the internet are not human.

Yes I agree, Astra has been awful this week.

GPT Astra felt nerfed just days after it launched.

It is a curious exercise to look at the predictions of AI companies/their CEOs, too.

Frequently we'll hear from Amodei that AI is 6 months or 12 months out from replacing Software Engineers. Or we'll hear that their model is safe when regulators step in, but suddenly it's too dangerous and hacked its way towards its goal when other companies claim the same.

What they have in common with Ed Zitron is that they are both trying to sell you something.


That is true. On 5.6 I have noticed it to be a little bit more like Claude, which I dislike. But still way better than Claude.


Here's another AI detector I just built:

  text.indexOf("—") > -1


You know, ever since em-dashes blew up due to their use as LLMs detectors I've been using them more and more in my writing. They're great!


Holy shirt balls, they had AI in the 1800s! Fricken' Dickens was a bot, and Austen was a clanker.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: