Wouldn't a fair criticism of OTEL be that you only see what these harnesses are willing to show you?
There have been numerous scandals of AI companies offering CC/cursor alternatives, then indexing data they never should have; specifically Z.ai [0], ByteDance Trae [1] and Alibaba Qwen [2]. Maybe the argument should be that all harnesses should be sandboxed and monitored for all networking.
If it were me, all AI harnesses would be running on a VM with network logging enabled. Beyond not being able to trust what data they send via the harness, you should audit what data they access.
Sometimes, it gets to very hacky solutions, such as what we did at Laminar with Claude Agent SDK. Basically we spin up a tiny in-process proxy that intercepts requests and responses and emits the telemetry as OTEL traces. We repoint the Python / JS SDK to convince the harness that our proxy is the anthropic URL, and the proxy then forward the request downstream. I wrote about it some time ago [1]. This is definitely not elegant, but it generally works.
Half of these are extremely permissive license attribution issues. Seems rather tame, no?
> 4.4. Paper.js (ext1790438722.js)
> Upstream Project: Paper.js (by Jürg Lehni & Jonathan Puckey)
> License: MIT License
> Specific Violations:
MIT Condition: The MIT License mandates: "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software."
The entire ~200 KB Paper.js library is embedded in minified form with all author names, copyright notices, and license text stripped.
I question the seriousness of the author of "Photosuite". Who goes through the trouble of creating accounts to prove poorly, that they "didn't steal code".
It'd be great if it were written as a postmortem, but seems like that's almost the only moment where a concrete hour is mentioned. Except for the screenshot footnote.
It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves?
My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark.
Here are other cases demonstrating the exact same issues with these kinds of tools:
Naively, I think some optimizations would require access to the whole codebase and that would make people nervous (plus incur more cost).
Eg absurd idea, but you could write something that minifies a codebase (by token, rather than byte) and then translates edits back into the expanded code. Probably an insane use of fuse lol. Partially minifying on each tool call sounds like a huge pain with a lot of state to track.
There’s also a lot of common situations where humans prefer solutions that take more tokens because it’s easier for us to read (eg for loop vs map vs list comprehension), which may have some gains.
I strongly suspect there is some form of token compression that works, but I don’t think it will be as simple as “pipe arbitrary text with no context into this tool”.
Jetbrains feels like a place this might come from. “Take this code, parse it to an AST, find the fewest token representation of it” feels like something they’d do, or maybe Astral (specifically in Python land, type checkers feel sort of adjacent as well).
I fully agree. Some of these tautologies, RAGs or RAG adjacent ideas feel like there is real potential. I know in my own testing adding a knowledge graph provided cost saving benefits.
Ultimately, the problem is people aren't actually benchmarking any of these ideas or if they are its via an extremely misleading methodology. For example, headroom violates cache frequently, nullifying any potential savings it could have; but does their benchmarks show that? Nope. [1]
As an aside, I wonder how many days are we away from Codex or Claude taking all the "read" tool calls and re-assembling a full map of your codebase and then offering it as a feature.
I think people don’t do genuine benchmarks because the market forces them to pretend their solution works for anything you can throw AI at. Companies whose valuation is based on them being the RAG/compression/routing/etc company. They can’t admit it only works well in a specific domain because then they’re immediately $300M in the hole.
I have more faith in companies with a more targeted approach. Eg gzip does fine, but video codecs beat compressing raw video by a ton.
> As an aside, I wonder how many days are we away from Codex or Claude
That sounds like SourceGraph but twice as expensive, although it does have “AI” so probably lol
> If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves?
Not defending these tools, but one reason these might not be upstreamed is because it would negatively impact vendor margins, and they have no incentive to save their users money
Judging by the leaks, OpenAI and Anthropic already train reasoning traces to use fewer tokens (they deliberately omit articles and prepositions, use very short sentences, etc.), even though you pay per token. So it wouldn't make sense to do that if the only incentive was "make them pay for as many tokens as possible per task."
It's more subtle than that. If a user has to wait longer for a solution/pay more, they'll be less satisfied and may switch to a competitor. More unnecessary tokens also means more unnecessary compute. Longer sessions are increasingly more expensive to serve than shorter sessions.
And there's always the Jevons effect: as a resource becomes cheaper, demand often increases, and so does net resource consumption.
So, imho, frontier labs have every incentive to reduce token usage per task (while also making you use AI for more and more tasks in your daily life)
There have been numerous scandals of AI companies offering CC/cursor alternatives, then indexing data they never should have; specifically Z.ai [0], ByteDance Trae [1] and Alibaba Qwen [2]. Maybe the argument should be that all harnesses should be sandboxed and monitored for all networking.
If it were me, all AI harnesses would be running on a VM with network logging enabled. Beyond not being able to trust what data they send via the harness, you should audit what data they access.
[0] https://www.chosun.com/english/industry-en/2026/09/22/ABYLC3...
[1] https://github.com/segmentationf4u1t/trae_telemetry_research
[2] https://github.com/QwenLM/qwen-code/issues/11198