That is bending the barrel to align the scope. What’s interesting is that the authors mention the cause of the failed calls, “…a model may generate a tool or field name that it has seen during the training, even if it does not exist in the current production environment. For example, it may generate search_web when the available tool is web_search, or use city_name when the active schema expects location”, but then failed to see that it’s also the solution. If you call it “web_search” but it calls it as “search_web” then just rename your function to “search_web” and let the problem resolve itself.

Things get more interesting if you don’t give it a call syntax and just see what it does based on the name alone.

I’m in, whaddya got?

And then you can use Claude-Magnum to fix the issues.

And then you can use Claude-Oeuvre to fix the issues.

And then you can use GPT-Cosmos to fix the issues.

And then you can use Claude-Omnibus to fix the issues.

And then you can use Claude-Logos to fix the issues.

And I can make a bowl of cereal with a teaspoon of milk.

Pity you think that.

I think my comment was a touch deeper than you caught. Jam doesn’t have a strictly required amount of sugar, it’s not defined as having X amount, so saying “you can make it with less” is two people talking about how much milk makes a bowl of cereal. Both are correct, it’s jam either way and it’s a bowl of cereal either way. Think about it.

You're right, good point.

They didn’t say anything of the sort.

I just use the noun in the question. The name of my pet is pet, my favorite movie is movie, and I attended High School. Add your own standard prefix if you want it impossible to guess. Nobody will know you attended “BLARG High School.”

You’re using the subjective definition of “quality”, as in the shade of blue it chooses for “Build a website”, or the character names for “Tell me a story.” In those cases it’s likely still subjectively “high quality”, depending on who you ask.

What the article is discussing, and what many people are concerned about, is something that you might be missing in your understanding: they’re not actually random. In fact, they would be entirely useless for real work if every token was randomly selected based on all possible outputs. It’s not, even at temperature 1.0. It’s based on the training corpus and once you have your tool names, syntax, and prompting style aligned with the training data then they become incredibly deterministic in the areas that matter, such as tool calling and parameters. I build my toolset by testing thousands of names, syntax, return format, and other aspects until I find a convention that produces the exact correct call, 100% the exact same every time, regardless of context length. Those decisions are per model and what works with Opus 4.7 won’t necessarily work on 4.8, and neither version will work with a local model or GPT.

That’s only possible because the massive training corpus is the guiding principle behind the choices. Providing a file reading function called “Read_The_File” will fail, either on the first call or somewhere down the line, because that name is not associated with the concept. Your instructions are trying to override 500 trillion tokens from training and it will cause perplexity to manifest as wrong tool calls, wrong syntax, “oops deleted prod”, “Claude lost the plot again”, “WTF?!”, and probably nearly every frustration you’ve encountered and determined to be “they nerfed Claude” or “it’s a dumbass.”

For those that are aware of it, that knowledge lets people tweak and tune the prompts/tools accordingly.

You may not put that effort into your system, perhaps because you’re unaware of it, don’t use it in a way that requires it, or you’ve just taken the failures caused by perplexity as something that’s inherent in the framework, but for people that build precision infrastructure around them it’s potentially devastating news. Watermarking, which is based on whatever tokens, threshold, cutoff, and triggers some guy at a desk decided, will necessarily alter that entire system.

Weird, the same stupid crap happens with Windows. I’ve had to learn the exact N character search to find the things I use often; any more or less and the result with that name is buried. It’s like Microsoft and Apple are just pip installing same broken search package.

On Windows I use PowerToys' Alt+Space command bar which does a much better job. In fact, it's the only search I use on Windows.

I switched to Everything for search a while ago and haven’t looked back. For the start menu I replace it with Open Shell, which seems to find apps much more reliably.

Good info, I’ll check it out. Thanks!

Always turn off searching the internet as much as possible on both. Helps a ton. I think I set both windows and iOS to only search local applications and settings

I don't find this to be the case at all.

You’re right about the harness being the issue. It’s really down to giving it functions specific to your use case that will let it surgically read/modify files, rather than needing to consume entire project folders. I built my own and for Python files some of the most helpful functions I provide are equivalent to:

inspect_function(filename, function, class)

replace_function()

call_graph()

And a few other convenient ones. Beyond that it’s trickery like if a function returns more than N lines I omit the result and auto-reply “Your function call was too verbose.” Typically that’s stuff like recursively listing every file in a repo to “see what it’s working with” or similar. When it emits the next call in response to it I clip the previous attempt (and my response) off the conversation and attach the new call/result as if that’s what it did in the first place. I also log that event so if the same type of thing happens often enough I’ll create a special function to address it, or modify an established one so it’s not tempted to do it again.

Language models don’t know what they know, they know what has been said. Even if you give it an entire Python environment it won’t reach for AST, but if you give it a function called Python_AST() it’ll use it every time.

I’ve never seen an off-the-shelf harness that approached it that way.

Style requests tend to get washed away by context. I’m not sure how you’d do it with Claude Code or whatever (I use the API directly), but you might try fiddling with giving it a function called “response” or “message_user” with a mandatory parameter like “output=plaintext”.

I’m basing that idea on custom functions I give Claude where I designate the style in the required parameters. Example:

“””

@pyrepl(code_golf=True, output=CSV)

//END

“””

The “//END” is a special terminator I use for halting the response. Claude runs code between the tags. The difference in code is incredible. No banners, comments, fluff, print(“=“*70), or any other nonsense, and the code is tight and compact. The parameters do nothing, Claude just outputs them as part of required syntax and it dictates the style purely because it output them.