There are good ways to leverage LLMs, but there's a lot more load bearing wait on that word 'leverage'. Something needs to do the leveraging, and do it well.

I'm experimenting with my own harness at the moment, currently codenamed Murder because I call the individual contexts/agents 'crow's.

The fundamental unit of it is what I call 'intrusive harnessing', where the harness actively manipulates the token stream so that significant quantities of tokens are only ever exposed to Layer0 when it's useful for them to be present.

For example: the full instructions for shell-tool calling aren't in the system prompt diluting attention while the model is reasoning/discussing what kinds of cat picture you want to put in your app.

My approach is more like dev-branching, and it seems to be working way more effectively than compaction or simple aggressive sub-agenting.

As soon as the harness sees the model is inferring a shell tool call, I stop the inference, mutate the context so that the full set of instructions/examples/guidance for shell tool use are inserted. Once the model has inferred the tool call, I curate the output it gets back. I ask the model to evaluate the output - good or bad - and give it a chance to accept/retry, before allowing the tool-call and output into the original context.

Does it use more tokens? Yes, although we're only mutating at head, so in a long-horizon context, it leans heavily into cache, just not the way anthropic/openai want you to realize you can.

It sounds like compaction but it doesn't come with the nasty brainwash experience where you just need the agent to fix that one last thing, it compacts and the agent comes back a paranoid delusional mad max.

``` <|system|>You're an AI agent. You do agent things. <|system|> ... there's a list-dir tool and a shell-call tool ... <|system|> ... memories ... <|user|>It doesn't look like it ran. <|reason|>I should look and see if there are any errors in the log file.<|agent|>I'm going to read the log file to see if there are any errors. <|tool-call tool=shell-tool ```

We stop there, and splice in the detailed instructions for the tool the model was about to predict. I'll use <|ALLCAPS|> to denote harness-generated pseudo turns.

``` ... as before ... <|agent|>I'm going to read the log file to see if there are any errors. <|SYSTEM|>Shell Tool: ... shell-type=bash, zsh, fish, pwsh on this system. Preferred shell is ... Additional arguments ... Pagination ... <|tool-call tool=shell-tool ```

the model finishes out the call. On windows, with a typical harness, this frequently goes like this:

``` <|tool-call tool=shell-tool|>Get-EventLog ... | head<|tool-call|> '''tool-result error: unknown command: head ''' <|agent|>Ah, windows doesn't have head. Let me just read the whole log. <|tool-call ...|> '''tool-result ... 500k tokens ... <|agent|>I see some windows log events but you didn't ask me a question. Daisy, daisy? ```

With Murder it goes like this:

Rev 1 ``` ... prefix as before ... <|tool-call tool=shell-tool ```

Rev 2 ``` ... prefix as before ... <|SYSTEM|> ... how to use shell tool; shell-related memories and rules ... <|tool-call tool=shell-tool shell=pwsh fence-vs-escape=true|> '''pwsh Get-EventLog ... | head ''' '''tool-result error: unknown command: head Your tool call terminated with an error, ... ... structured response required ... options or annotation , ... ... <|reason|> windows doesn't have the head command. Let me try reading the whole log. ... replacement tool call ... ... model note ... ```

I take that feedback and loop it, so, Rev 3: ``` <|system|> ... how to use shell tool; shell-related memories and rules ... <|agent|> ... prefix as before ... <|SYSTEM|> ... as before ... <|agent|>{prev_cmd} failed, because windows does not have a head command. Let me try reading the whole log. <|tool-call ... no head ...|> '''tool-result ... first few lines of result ... ''' <|system|>Your tool call succeeded but generated 446,219 lines of output. Only the first 5 were listed. ... structured pagination / retry / rephrase options ...

```

It then repeats while the model figures out the right command, figures out which filters to use, but the harness effectively immediately guides the model to do an immediate [optionally self-adversarial] review of the command against the output until the model concludes that the result is useful by various criteria. That doesn't mean successful - sometimes what is superficially an error (no such file or directory) is the answer you were looking for.

Let's say it takes the model 3 more turns to figure out how to use event viewer, and finally it s.

Here's the win, the outer main context - the one we're going to keep growing as you work with the agent, looks like this:

``` <|system|>You're an AI agent. You do agent things. <|system|> ... there's a list-dir tool and a shell-call tool ... <|system|> ... memories ... <|user|>It doesn't look like it ran. <|reason|>I should look and see if there are any errors in the log file.<|agent|>I'm going to read the log file to see if there are any errors. <|tool-call tool=shell-tool shell=pwsh|>Get-EventLog ... | ... | ... '''tool-result (use ref-tool id=A401U8X593 for full transcript) Event ID | Last Occurred 1010111 | 3 weeks ago ''' ```

We used a lot more tokens. How can that possibly be good?

It's happening at the end of the context, so the cache comes into play very effectively.

But if we'd let all that derp into the context, it would be a potential attention sink degrading the value/worth of every subsequent token.

The pattern of try-thing-fail-try-solution-fail-try-win appears to be an incredibly strong pattern for most agents.

Fundamentally: When you're 3 prompts down the line and there's the imprint of the model doing "somewindows command | head" in the context with the model litigating it and fixing it -- that meta-pattern will drive the model to predict more of these patterns. It's going to repeatedly eff-up the exact way it saw in its training material.

When I try to get Claude/Copilot to work on this codebase, they freak out. The hyperbole/marketing pitch the agents were trained on and is built into their inner prompts cannot seem abide the idea of stopping an LLM mid inference. They seem driven to perceive an LLM endpoint like a 911 call you can't just go quiet on.

I have a mechanism for non-parallel sub-agents ('maggots', their job is to curate a large body of work whose full text is irrelevant to the main context). Basically just a tool call, but every time Claude or GPT have been near it, they've broken it, forcing it back parallel so they can send the invoking model a notification that it's child has been spawned and the parent should call the 'check-result' or 'wait-result' tool when they're ready to receive the results.

One of my test architectures is running against a solo Unsloth Studio instance that can only load one model at a time. It really doesn't react well to having you load the coding model to start your sub-agent work and unload before the model has generated its first token... :)