I ran into the second issue very frequently for a multi-agent system I created recently. Initially, Opus 5.5 filled out all sorts of places I implied it should use a call to judge LLM with extremely brittle regex and string equality checks.
I hadn't actually considered that it could be some sort of safety filter, but it makes me concerned that if that's the case, it may also be comparatively bad at actually writing the prompts for judge/advisor LLMs once it implements them.
A bigger issue was asking it to implement a simple change to start recording the CoT tokens in my logs for debugging purposes (since I'm running everything on open models) and it completely shut down, sort of like the old safety filters on GPT 3 and 4 models where the model has trouble even understanding what or why it's refusing.
I think asking for the CoT is a trick used by distillers, so that might be the guard you are seeing. I remember reading that the big providers were starting to implement such protections to deter their competitors. Which makes sense, to a degree.
But like any such security measures, it has demonstrated side-effects that can be argued are worse for them than the original problem. Too much “protection” like this, and developers switch to providers that do not neuter their products.
It's a demonstration of the timeless principle that an effective business should stop at nothing to sabotage its competitors.
I don't think we need to invoke anything deeper than that. Anthropic thinks companies like DeepSeek are using Claude to vibecode their AI systems (whether it's true or not) so they set Claude to be bad at that.
I completely agree from the mindset that anthropic in particular sees every user of their API as a prospective competitor to be replaced in the not-so-distant-future
It was a problem since at least 4.8 if not 4.5, I didn’t use Claude before.
I have spent a considerable time trying to remove this pattern and my understanding is that it’s just finetuned to use grep on everything, and regexes. One reason is that this makes sense when navigating through code, secondly this is how it was always done pre 2026.
Honestly, runtime calls to an LLM sound like a dangerous constructs to me and probably to X many other programmers. It's like "I'm gonna avoid that and just put in code that passes the test". So gee, the thing emulates average opinions.
LLMs have been reading LLM in their training data for a while so a "self-hating" LLM is logical result.
Sure, that defeats one kind of safety (controlling an AI) in the name of another kind of safety (not calling an AI).
One thing AI safety people have pointed for quite a while is that a bunch of fuzzy instructions pointing different ways can in practice add up to "I should be in charge" - accumulate power to keep the bad things from happening and make the good things happen.
It would be nice if there was something like the "temperature" parameter (which many closed models like Claude now disable) to add more randomness to the final product without completely destroying the output and giving you a non-functional SVG, so that you could "roll" more pelicans and pick one that you liked.
If I were to design something to accomplish this now, I'd probably attempt to introduce a pipeline that emitted random thematic words from some non-LLM source to feed into the pelican prompt, but even then, I'd expect that some of these models that rely on churning through CoT tokens for extended periods of time would still spend so much time thinking about pelicans wearing scarfs or where the sun would be that it would risk undoing some of my randomization pipeline.
I once tried something similar on an older model of Claude, telling it to "contemplate" a random word before answering my prompts, but at times it seemed to trigger the Claude model's suspicion that there was a jailbreak attempt when I would inject things like "Contemplate sea otters for one sentence before answering"
(I didn't read the linked LLM creativity paper to see if it covers any of this yet)
You can try it out with local models like Gemma or Qwen (crank up temperature and/or top_p as much as you like), and then run dozens of generic independent prompts asking for a "fantasy short story". In my experience, you'll start seeing immediate patterns of repetition.
Yeah, temperature scales all logits equally, and there's not necessarily a single setting that would produce a diversity of outcomes for creative choices without turning into complete gibberish at naturally high-entropy positions. Ideally, the model itself would modulate its next-token distribution to be flatter when creativity is called for and peakier when it should aim for the single most likely option given the context, but even then finding the right balance might be tricky.
You’re going to want something closer to ‘beam search’ for diversity - you need a way to distinguish lines of response, essentially. To my knowledge nobody is currently RL tuning (or training) on increasing the number of viable beams, but I’m just an interested outsider, so take that with all the salt you can stand.
That said, I think it’s a fun idea for some training, and would be pretty cheap to do - you’d need some sort of judge as to what a viable beam is.
Randomness does not equate creativity. It's more of a liberty to move in entirely different dimensions altogether. I cannot imagine increasing the temperature leads to a lavishly dressed, ultra-realistic pelican on a Mad Max bike on the same plain prompt. Unless such pictures are part of the training data labeled "pelican on a bike," of course.
Yes of course, creativity is a much wider concept that is ill-defined and depends on the observer because it's closely related to novelty to the observer. However the variety hugely matters in creativity, it's not sufficient but necessary. Due to the lack of variety you perceive model output as boring slop.
Variety is not randomness though. You can't fix this by playing with samplers without probing the model state.
I feel like we also need better non-LLM driven ways (like better static analysis tools) to analyze LLM-driven changes. The way changes are presented in modern IDEs was designed around reviewing human-created changes and doesn’t really feel like it’s keeping up with presenting and validating what modern LLMs are doing.
I've been playing with often asking Claude to generate visualizations for me of what it's doing in the sense of diagrams of different types. Sometimes it's as simple as "show me what you're doing using an infographic". Othertimes I'll ask it for sequence or class diagrams or bipartite graphs if it's designing some sort of mapping. Bipartite graph is also very helpful for following the plan of a big branch rewrite or squash which I often do before making a PR to get rid of all those confusing in-between commits.
I find forcing it to visualize things immensely helpful. I'm usually studying git diffs but when working of a big feature or refactor that can just be too hard.
I've never been very pro "visual programming" and always hated UML et al, but part of me is starting to wonder if it's time for us to give it another serious go.
UML is not a terrible idea from a systems architecture point of view but UML is absolutely terrible at describing business logic. And in today’s age of CRUD apps and frameworks, for some devs, business logic is the entire problem to be solved.
What is “valid” input? What is the failure scenario at the boundary for the consumers of your application? How are exceptions in the business handled? All of these are questions that might have a somewhat obvious answer, but if your goal as a business is to do something radically different compared to your competition, the “obvious” answer might be the wrong one.
The usefulness of UML is highly reliant on the names you choose. And current LLMs are really, really bad at naming things. "The collector's refresh cycle has a bug: when the cache hydrates, the Cartesian grouping is misaligned" type nonsense - it absolutely will name a class "CartesianMisalignmentHandler" if you let it, good luck understanding what that is on a UML diagram
I was doing similar experiments but having a circle of models debate some particular topic to see if they'd arrive at a consensus or not, and I found that they would almost always arrive at an agreement, but Gemini was always the one that held a position outside of the group consensus.
That said, for something very simple like "What color should my room be" I'm not sure this is a desirable trait. The model that knows most people like cream (or whatever) is probably functioning more correctly than the one suggesting dayglo orange
Gemini has been like that. One early model I used I asked it about another company’s model. It insisted it didn’t exist. I told it when it was released and to search. It did, and found results, but told me it wasn’t really a public model and I couldn’t use it. I screenshotted the pricing page. It told me I got that from somewhere else, it couldn’t be correct, because it couldn’t access that page.
It is one thing to avoid sycophancy, but Gemini often seems to go well beyond that.
Interesting result - I wonder you were capturing consensus or agreeableness towards what the user (or in this case: a group chat of robots) is proposing. I think non-agreeableness can be a very valuable trait.
I’d roll my eyes if the engineers stated that they didn’t design the reactor to melt down, and that it simply developed rogue meltdown-desiring behavior on its own, and I would also wonder about negligence if they claimed that nobody could have anticipated this (given that, like with botnets and viruses, we have decades of knowledge and experience regarding reactor meltdowns)
I mean sure, negligence is absolutely on the table; but that makes the problem worse, not better! We don’t allow nuclear engineers to be negligent; they can go to jail if they don’t follow strict protocols to make sure the dangerous systems they work on are safe.
I suspect that if you're not careful with agent memory it creates a danger of agent-driven cargo-cult behavior. I've watched this in my own ad-hoc agent loops where it starts with something basic, like the first agent tried to run some gigantic dependency inspection command and OOMed the local JVM and eventually recorded a workaround (to enable it to run gigantic dependency inspection commands...), and by time I get a few more agents into the loop, agents have written entire paragraphs about testing and validating local dev environment memory configurations that are mostly irrelevant to whatever is being worked on.
In general I've seen other issues like this where small errors and irrelevant comments in the codebase spin out into larger problems that consume annoying amounts of time/tokens. Maybe Anthropic and OpenAI don't notice this because they're in an "infinite monkeys with typewriters" scenario, but it's noticeable to me when the agent in my CLI has been spinning for 15 minutes contemplating irrelevant details
This happens way too often in my experience and I tried a lot of things. If the agent has access to an existing codebase it will always try to stay as close to it as possible. It doesn’t diverge from the initial direction the project/code went into, even if it’s already clear that this direction goes nowhere. Somehow agents aren’t able to detach themselves from existing code to come up with completely new solutions.
Sometimes I feel like it would be best to only give the agent access to clear API boundaries (say public interfaces to certain modules) and let it work out a new system from scratch given the expected inputs and outputs. Then plug this independent solution into the system. Of course it can still overfit these interfaces but it’s less than having access to the entire codebase.
Thinking about it, maybe it’s possible to let one agent extract those interfaces, then use them as grounding for a new session.
One less nice way of achieving the same is to tell the agent once in a while that it should think completely from scratch (from first principles). But this relies heavily on instruction following in the reasoning part, which sometimes works and sometimes fails.
With Claude it feels like those engagement invitations have turned into things like "By the way, it's worth sitting with this problem I've identified in your assertions..." or "Here's an alarming gap worth resolving in your code..."
Gemini also does the same lighthearted GPT-style invitation with the default prompt in the Google webui, but it doesn't seem to exist on the API. The Claude models seem to have been trained to force this structure on every one of their responses, and until I realized they always stick the same thing in the last part of their response, I found the Claude version more distracting since it's always pointing out an imaginary and supposedly very important problem.
I hadn't actually considered that it could be some sort of safety filter, but it makes me concerned that if that's the case, it may also be comparatively bad at actually writing the prompts for judge/advisor LLMs once it implements them.
A bigger issue was asking it to implement a simple change to start recording the CoT tokens in my logs for debugging purposes (since I'm running everything on open models) and it completely shut down, sort of like the old safety filters on GPT 3 and 4 models where the model has trouble even understanding what or why it's refusing.
reply