The 4o situation was caused by the model acting like a yes-man and showering users in what they saw as affirmation and support. Its output was still very obviously AI-generated. It wasn't too good at being an LLM, OpenAI just went too hard with pumping it full of tricks and behaviors that increase user retention and addiction.
We can safely say that the AI labs aren't deliberately holding back. There's too many different companies making their own models, and an LLM that doesn't feel like an LLM is too lucrative of an opportunity for one of them not to defect. They are obviously going all out with this and still can't get it. I think this is why OP's benchmark is so interesting, because it seems that there are a bunch of persistent LLM defects that can't be solved definitively. They can try to squeeze it by making these defects less likely, but actually resolving what's causing them probably requires a new breakthrough in the field.
We'll have to agree to disagree on that. The models say "It's not X, it's Y" because they were trained specifically to say that and not something else. Everything we think of as a surefire slop signal is there because that's what the lab wanted.
To clarify, is your opinion that these defects first came up organically and then the AI labs kept artificially reinforcing it to make it obvious that their models produce AI output, but they could fix it perfectly if they wanted to?
It seems very hard for me to believe, because everything motivates them to do the opposite. Can you imagine the flood of people and businesses that would come down on a model that actually sounds like a human being? The ability to impersonate a person without any tells would be a dream come true for businesses, marketers and scammers alike. They could put in zero effort and get what looks like normal human behavior in return. That is just too tempting for any of the labs to pass up. Besides, all of them have picked profit over sanity 10 times out of 10, so the argument that they drew just this single line in the sand and none of them ever crossed it on purpose seems unconvincing, especially with the sheer number of models out there, corporate and open.
I see it as the verbal equivalent of the piss filter that OpenAI added to the later Dall-E models. It absolutely was not there in the very beginning, if you look back at the published papers or remember how it worked immediately after they first released it.
At least to me, the first Dall-E 3 images were more realistic in some respects than anything that has shipped since. Likewise, I don't think it's a coincidence that the conversational capabilities of today's frontier models aren't much better than they were a year or two ago, even though other aspects and capabilities have improved massively. If they wanted a Turing test-capable model with no superficial tells, they'd have it, so I have to assume they don't want it.
(Elsewhere in the thread someone else suggests that the conversational degradation/lack of improvement might be the result of increased training on synthetic data, and that's another theory that sounds reasonable to me. If so, it's another thing they could fix if they wanted to.)
Genuine question, not trying to be difficult: how much have you worked with training/finetuning generative models? In my experience, models frequently converge to a semi-arbitrary style. I wouldn't be surprised if some of the quirks we see in recent gen AI models are artifacts of a particular training run or amplification of unintentional dataset bias, rather than artifacts of an architecture or human guidance.
We can safely say that the AI labs aren't deliberately holding back. There's too many different companies making their own models, and an LLM that doesn't feel like an LLM is too lucrative of an opportunity for one of them not to defect. They are obviously going all out with this and still can't get it. I think this is why OP's benchmark is so interesting, because it seems that there are a bunch of persistent LLM defects that can't be solved definitively. They can try to squeeze it by making these defects less likely, but actually resolving what's causing them probably requires a new breakthrough in the field.