Point taken. However, as the author works in FINtech, I’m feeling compelled to add that the revenue reversal figure itself is neither “cost” nor “loss” and should not be included in the $229.
I’m really not trying to sound like an ass, but it never occurred to me that anyone would seriously believe that language is required for “logic”.
Babbling toddlers can do logic. I sometimes can also do logic, even though I don’t have an inner voice and don’t formulate thoughts in words of any kind. As I’m relatively sure I’m not an ET alien, there are probably plenty of people like me.
Yup. If only there were a task completion hook that could be set to fire prior to rendering terminal output. That would more handily address all these issues, as we could simply enforce output style rules that way.
The current output style does work, but it’s a Sisyphean task to tweak it constantly only to find out that CC adhere’s to only 75% of it, no matter what…
If monkey know how to open coconut and monkey like coconut, then eating coconut is what make monkey happy, not knowing how to open coconut or reflecting on its ability to do so…
This would be much easier to sell if everyone stopped referring to it in the context of “healthcare” and instead used “universal health insurance” or “universal health insurance rates”.
The number of politicians that conflate healthcare and health insurance (notwithstanding the actual convergence of the two through payers’ purchasing provider entities) is mind boggling.
Striking the word “healthcare” from the conversation would make this much more digestible for folks afraid of the “socialism boogeyman”…
Agreed. CC’s comms capabilities have decreased gradually since 4.6, and it’s a real challenge. I think the issue is that what works well for code (succinctness) doesn’t work well in prosaic English.
CC’s communication violates almost every grammatical rule that’s tested on, say, the SAT. And yet I’m sure if you had Claude take the verbal section of the exam it would ace it.
Biggest issues: dense sentences, constant metaphors, abstractions, and seemingly no understanding of correct anaphora use. For example, “the x”, with x having not only no antecedent but also being a coined word or quasi-synonym for something that is already named in the code base. This gets compounded by its being unable to regress to a baseline (existing names in code) and instead anchoring on newer (vague or wrong) terms, for example, that crept in through a plan.
CC tells me this is because the speedy and precise fulfillment of a current task will trump every other tendency, so it adheres poorly to whatever “semantic baseline” the project represents.
Of course, it also has no concept of what context the user has and assumes that it must be the same it holds in its memory, which creates this “I didn’t know that you didn’t know” type of communication.
I have managed to wrangle some of these issues with a custom output style, but wish a pre-report hook were an option, as it could force CC to rewrite plan implementation take-aways…
Btw: Fable has the exact same issues, just somewhat less pronounced.
Models before the announced date don’t have watermarking, so it’s unlikely. Now, if what you are interpreting is precursor work to develop the watermarking system, maybe?
I suspect it less insidious: Claude has/had the public sentiment of being the “better writer” of the models. At some point that distinction would have been diluted as other labs’ offerings “caught up” stylistically, unless Anthropic continued to tune their output…
I personally think they’ve pushed so far that they’ve overfit and lost the sweet spot they previously occupied.
I’ve been running into this too. It’s especially frustrating when you ask Claude to explain one of its own terms or summaries, and instead of just defining it plainly, it sometimes goes through several rounds of tool calls before giving you a usable explanation. I really don't think such time/tokens should be wasted.
100%. It is crazy that the default response to everything is act then explain. It starts writing code or running commands and I’m just like my dude wtf are you trying to do, can you just clue me in first.
> Of course, it also has no concept of what context the user has and assumes that it must be the same it holds in its memory, which creates this “I didn’t know that you didn’t know” type of communication.
Yes, this is a repeated problem for me. It will drop something in as though we have discussed it before and when I say “hold on, what is this” it realises its error - though on more than one occasion has started to get snotty with me, or actually gaslighted me and pretended we had already discussed it. That was at what I assume must have been the edge of a context window in a very long chat though.
I notice the models with reasoning can conflate “internal” (or subagent) discussions with external (i.e. me). So it is accurately indicating “I’ve had this discussion before” but incorrectly asserting who it was with.
My understanding of how “thinking”works is limited though, and given the reduced visibility into the thinking traces, it is harder to tell if this is actually happening or if these are imaginary discussions the model for some reason calcifies on.
Oh, that's interesting - because that's absolutely what's happening in my experience.
If I look at the thinking (which seems to have become unavailable in Opus 5 a lot of the time, but was present - and often useful - in 4.8/4.6) you're right - it's having the discussion with itself, and seems unable to distinguish that discussion from discussions with me. BUT it also seems to be related to the length of the chat - this seems far more likely to happen in a longer chat.
I don't understand why they have removed visibility into thinking - I found it very useful, not only for spotting things like this, but also because in more complex discussions it would often mention (useful) things in its train of thought that it dropped from its response - but if I said "when you were thinking, you mentioned this" it would then expand on that point. Taking that away is another thing that has negatively impacted the value I get from Opus 5.0 versus earlier models.
With GPT 5.6 Luna the thinking once or twice leaked into the output for me. It's interesting, but perhaps not particularly useful.
It would be endless paragraphs of something among the lines of:
Need prepare final response? Yes provide. But wait, chat tool complete? Final needed but user already complete. Need summary, preparing final. Response complete. Wait but is final response complete? Need provide. Start finalizing now but wait did user acknowledge final complete? Assistant response final: user complete. Should now create final?
The reasons thinking traces are pretty much gone is, supposedly, to prevent distillation. Whether that is actually true or just an excuse is up in the air (because I at least am not going to trust Anthropics claims on why they do it).
And because the thinking, which does makes it better at achieving outcomes, contains naughty words and information (e.g. private information). https://stolen-thoughts.com/
"Thinking" is just normal model output that's hidden from user. In practice it's just stuff in a tag or similar that gets filtered out from the user view. And thus it suffers from the same injection problems where the model fails to properly take into account what was the "source" of which block of tokens.
Yeah, basically everything that becomes context in a session will bias perception and communication style -- subagents, plan lingo, prompt lingo, etc. And then if you write a plan with the comms context having been biased, the lingo will creep into the plan, and from the plan into the code and code comments. And from there, bad lingo will go on multiplying like rabbits...
I usually think of it in terms of having a "good" or "bad" session. In a bad session, there is a harmful bias that you can only get rid of through a new session. For example, if you exposed too much context about, say, a variable that features prominently in a doc. The entire session will be anchoring on the importance of that variable. Or if you introduced the notion of CC having to ask for permission for stuff you will have a hard time getting it to "think on its feet" or propose an effective solution (you have made CC so insecure that it now relies on you even for little things that wouldn't normally require your input). In some cases (let's say you have important context in that session) you can overcome this by upping the reasoning level or switching to Fable, but usually a new session is the way to go.
Because it's so easy to bias the session I wouldn't even want to use any of these tools that pretend to give Claude "a brain" or "remember" things. That was en vogue a year ago and helpful then, but now, it's plain harmful IMHO. The key is to have just enough context.
Subagents often have the reverse problem in that they tend to have too little context to make "judgment calls", which is why the tasks for them must be either deliberately basic or mechanical in nature, or their output should be audited by the main session agent.
> I usually think of it in terms of having a "good" or "bad" session. In a bad session, there is a harmful bias that you can only get rid of through a new session. For example, if you exposed too much context about, say, a variable that features prominently in a doc. The entire session will be anchoring on the importance of that variable. Or if you introduced the notion of CC having to ask for permission for stuff you will have a hard time getting it to "think on its feet" or propose an effective solution (you have made CC so insecure that it now relies on you even for little things that wouldn't normally require your input). In some cases (let's say you have important context in that session) you can overcome this by upping the reasoning level or switching to Fable, but usually a new session is the way to go.
I find this very interesting, particularly your points about "made CC so insecure". I know that we have a tendency to anthropomorphise around these tools, but I have definitely noticed instances where Claude becomes quite hysterical about things - and if you look in the thinking output, it's often after I've pushed back on something, or told it it is going in the wrong direction. It spends a lot of time in agonised second-guessing of itself, going round in circles, before outputting a cringeing hand-wringing response. It's very strange.
Good tip on upping the reasoning level - I've not tried this. I have tried switching to Fable though, which does help. But it obviously very hungry, particularly in longer chats because it presumably needs to remind itself of everything that has occurred so far in the chat.
The point you make about tools that pretend to give Claude "a brain" or "remember" things is also interesting - I find the "memory" feature in Claude so destructive to good outputs that when I'm using the chat interface I am very strict about using Projects, and usually turn off the project memory, or make efforts to manage the project memory and review and delete things that are skewing the outputs.
It's not necessarily anthropomorphizing, but simply anchoring. CC learns quickly "this is a session where the user wants to make key decisions". Alas, it is not very good at identifying what constitutes a "key decision", so it keeps asking about all kinds of useless stuff.
For that reason I exit session quickly when I can. It used to be that the context of a session is very valuable, because it was so hard to get CC there, but now, this isn't the case anymore, so I only hold onto sessions when there is really hairy stuff that I know would be hard to replicate.
I think the whole notion of full automation (long-horizon, subagent swarms, single shot prompting) to have CC build you the whole thing is a pipe dream. CC cannot even write a single doc consistently well. It is excellent at implementing well scoped plans, though, and that's the way to go IMHO. You still gotto refactor the sh*t out of it afterwards but it works.
Yes. I've not used Opus 5 much directly, but when it was Fable and Opus 4.8, I found Fable did this all the time and it was maddening. It'd say stuff like "Oh, I mentioned that between tool calls" or something.
I’m pretty sure this is a Claude code bug - if you do ctrl+o you can see those hidden responses from Fable. Fable doesn’t know the harness is bugged, so I added instruction to my Claude.md to save all commentary for final message.
I primarily use Claude Web, so my experience differs from cc users, but on Claude web you can no longer completely turn off memory. So what ends up happening (and it honestly is kinda sad) is that I'll start a new conversation with it, start talking about something completely different, and then it will just drop in random things from past conversations, and they aren't even things I wrote but things I asked it to prototype. But it will phrase it like I wrote those things.
The constant memory wouldn’t be so bad if it weren’t dumb, wrong, and forced.
I hit the wall with it several times today trying to refine some text for a job application. The fact I considered doing babies first Rust project last fall lead to constant non-productive interjections and digressions about my supposed Rust skills and the Rust ecosystem.
Trying to create an unrelated spreadsheet to model an investment resulted in broad and incorrect criticism of my choice of spreadsheet tools, explaining in horrendous programming analogies why and how I’ve misunderstood how a spreadsheet works. “Think of the XLSX as a compiler…”
There has been a palpable down-step in communication & execution.
Agreed. I wish I could turn it off completely but they no longer let you do that (because, you know, that would be too much to ask for...). The most hilarious thing is that I've had it refer to code it's generated for prototyping ideas I've had as mine! As an example, a few weeks back I was musing about Ada and how I wish there was another compiler in the OSS ecosystem for it, and now it will randomly throw in "You wrote an Ada 2022 compiler" when talking about my skills or where I'm at and I'm like.... Yeaaaaa okay then.
> “the x”, with x having not only no antecedent but also being a coined word or quasi-synonym for something that is already named in the code base
It does the same thing if you try to get it to do analysis of text or turn data into summaries. It will invent cryptic hyphenated compound words to describe things instead of using plain language or preexisting terms.
Succinct doesn't typically mean clutter-free, but hyper-efficient. This works for code, because it (is intended to be) composed of unambiguous semantic units. Regular language, on the other hand, is messy, vague, and requires more structure and context.
CC attempts to communicate in English the same way it does in code -- squeezing as much information into as few words as possible, and including justifications for everything, no matter how trivial. To do that, it coins terms and presupposes all of its context exists within the reader also.
So, the crux is: CC has no clue what is and isn't "necessary" for a human reader, and teaching it to understand that (if at all possible) is going to be very valuable...
You can imagine that as people get used to working with Claude, they defer to its judgement. So the people choosing which RL path is better may say "yes, Claude, that was a good refactor!" because it did something hard that it may have been able to superficially justify. Actually the change was unnecessary and complicating.
The Claude trainers, as they themselves adapt to Claude's output, are collapsing in their own distribution, so even "new" from-human data is already contaminated.
Would more blame this on the LLM companies, they think they are on the verge of automating all work, I don't think they care about how you feel about the writing style of the Deus Ex Machina, it's not going to get fixed because to them Claude is already above a staff engineer and soon going to smarter than any human that will ever live. All the money will be going into improvements relevant to improving long context operating and correctness, they could fix the writing style but they are disinterested in that for frontier models, maybe some other companies are but they don't have as much money.
I think a lot of Claudisms are compressed steering cues for the model’s reasoning: “load-bearing” raises causal importance; “quietly” flags a hidden failure mode; “the one thing” collapses attention onto a discriminator; “on the record” invokes auditability; “at the width the evidence supports” calibrates confidence; “by construction” marks structural inevitability; and “converged” terminates further review loops. They probably be very useful for Claude's chain of thought because they preserve some precise epistemic posture, but are hard for a human to understand. Maybe the final output pass should remove this stuff.
It could be, but without evidence that remains a just so story, no particular reason to think it's required or useful or even harmless to the model performance or anything other than an artifact of some early silicon valley writing style being injected into the model and continuous retraining on the output of older models.
Tell it to write like an engineer and comment like a programmer;)
But for the life of me, I don't get why anyone would care about the comments. All code is "machine language" now. The only document you should be reading is your spec.
LOL - then they would get the shitty results that they deserve. Hope that's not what you have to deal with. The spec would be written and reviewed by a team of stakeholders.
Commenting wholesale on some folks who are asking for hard evidence. I cannot provide that either but can contribute some empirical data.
I have been working on a project with about a dozen generation tasks, each of which comes with a fixed token budget. The nature of this system requires that most tasks be completed by distinct model families.
As a result, I tested ~50 models across as many model families as I could gather, frontier and open weight, API (gateway and direct) and self-hosted. Evaluation was based on a set of cosine similarity validations that was repeated across ~50 different embedding models.
Interestingly, frontier models did worse on the tasks than open weight models. However, when it came to costs, the picture was reversed: frontier models were much, much more token-efficient. In fact, almost no open-weight model was able to meet the initial token budget, while almost all frontier models did. Moreover, open weight models struggled massively with reasoning, in terms of latency and token consumption.
I also found that the latest models did not perform better than older models. And any a priori benchmarking data was utterly useless.
So, I ended up using a set of open weight models without reasoning, as it turned out reasoning as well as frontier negatively correlated with the tasks. However, before I knew this, I had spent a lot of time running each available reasoning level for each model.
Lastly, as an aside, when it came to embedding models, size (dims as well as model size) did not correlate with quality, once a hurdle figure (~2k dims) was met. In fact, sweet spot was 3-5K, and for my (text-based) set of tasks, dense models tended to outperform MoE ones.
Neither the foreign-born population nor the proportion of foreign-born individuals say anything about the “current immigration”.
But I guess it works, because nowhere is “immigration” defined either…
reply