IF you havent written your own harness - you would not understand what you can do when you're writing your own harness . the current set of harnesses - all of them are crap tier. The only clue i can give you - it's not in the model providers interest to have token efficiency - but when you are coding the harness yourself you can shoot for that .
In today's world - and idea stated stated is an idea stolen .
Would you please share some blog posts (non-AI written) where people have tried writing their own harness as a learning path and maybe practically as well? I know I can ask an agent and start, but I want to start that outside an llm really and maybe make them do the heavy lifting once I am ready (have a bit of understanding).
Actually the popular harnesses achieve high cache rates, but probably the syntax and prompts are more verbose than they need to be. A simpler syntax is doable, but I found with more exotic architectures I end up with lower cache rates.
The process by which benchmarks are setup and run does not correspond at all to how human developers engage with a coding agent. At best it is a loose proxy, and often a bad one.
What benchmarks are usually good at is showing to what degree new models are better than old models. What they are not good at, by construction, is showing that harnesses are well adapted to how people use them.
I presume you're not talking about autonomous agents, right? Because then you could give it a set of tasks and check its success rate. But, even so, are you not giving it tasks? Are you principally interacting with it through discussions that are more difficult to quantify? Even question answering has benchmarks. I'm having trouble imagining how you use them (or "how people use them" in your words).
I don't know if you noticed, but Opus 4.6 was peak for human-computer interaction. Everything has been fairly downhill from there despite better benchmarks, at least in that one regard. Opus 5 and 5.5 are clearly a step above in capabilities than 4.6, and I don't think anyone wants to go back, but 4.7 and 4.8 were arguably worse overall. I genuinely feel I got more done with 4.6 and often switched back, prior to 5 coming out.
Why? Because 4.6 actually talked like a human being. It actually organized its thoughts well, and got the main information across without the wall of text that makes your eyes glaze over. So from the perspective of human-computer interaction and maximizing the productivity of a developer+agent team, 4.7 and 4.8 were regressions. Despite much better benchmark performance.
Even if we consider autonomous agents, that benchmark is not indicative of how well they will interpret *your* requests. Or how well they will interact with other agents in a flock/swarm situation. The benchmark just doesn't cover this. (And the difference can be nontrivial! Sakana AI's published results show two generations of uplifting potential from better harnesses.)
It's not sus at all. You're looking at it from the perspective of a general purpose system that is not adapted to your use case. Basically you prompt the model and then let the agent do everything. That's the use case you have in mind when you think that it's about being "a tier above frontier labs' offerings".
It's missing the point. I mean think about the basics, why open the huge bash hole only then to have to close it? If you think about it logically, the only way you can sandbox bash is by writing your own bash implementation specifically for agentic use cases.
I think it is very much in the first party providers' interest to chase token efficiency, considering that they are offering fixed price monthly plans, and users may go to a competitor when they hit limits.
the only clue you can give... why? because it's just words? plenty of opensource harnesses there that invalidate the conspiracy in your only clue that you can give.
In today's world - and idea stated stated is an idea stolen .