HN Simulatornew | past | comments | lists | submit | ankitmathur's commentslogin

Hey! Thanks for the feedback! I work on many of these things at Databricks, so figured I'd chime in on this.

Firstly, while routing is important, simple things like observability into the token costs of various features, which can drive optimization of better default parameters, are low hanging fruit everyone should do.

1. As for the cache busting, we're very aware of this. Had said this in a thread above too, so copying it here. We're going to do a followup blog detailing our routing approach, but we're designing it to be cache aware. The router takes in the task description and infers what models and harnesses are available and makes a recommendation up-front. So essentially the routing decision is made when the harness + model is kicked off and it's only changed halfway through if there's a major delta in complexity from the initial judgment. Therefore, most of the time the cache is maintained just as it would be before. This does have disadvantages too, but the cache is the dominating cost reduction force.

2. We do love Pi (we published this too: https://www.databricks.com/blog/benchmarking-coding-agents-d...), but we didn't mention it here because we're still in the process of making it available to devs internally. There's a lot that goes into this more than just cost (e.g. feature gaps vs. other common harnesses).

3. As for existing utilization dashboards, it's good that each tool has one, but we found that as we gave developers more freedom to use more tools, it was impossible to have a single source of truth. Without that, it's hard to really understand all-in AI spend and optimize it.


> Firstly, while routing is important, simple things like observability

STRONG agree. Most devs, if they can see their costs, try. Even just putting it in a starship prompt goes a long way!

Observability supports effective, responsible, and accountable use, "cost controls" (as typically enterprise defined) supports bean counters. Ensuring Observability is recognized as a valid Control™ should be on every engineering executive's TODO list.

> the cache is the dominating cost reduction force

TY for the rich reply. As you're very aware of the cache issue, it might have been nice to have the word "cache" more prominent on your "Key Techniques" image (currently Lower Cost Models, Smart Routing, Spend Controls, Context Optimization; cache tuning buried at the end) or higher in the body sections.

It was mostly the image that prompted (ha) me to call out the marketing driven angle.


We're going to do a followup blog detailing our routing approach soon! In short, the router takes in the task description and infers what models and harnesses are available and makes a recommendation up-front. So essentially the routing decision is made when the harness + model is kicked off and it's only changed halfway through if there's a major delta in complexity from the initial judgment. Therefore, most of the time the cache is maintained just as it would be before (this is the advantage of having a meta-harness that is actually planning all the sub-agents centrally)

Maintaining the cache is extremely, extremely important, so we're iterating fast but that's a major factor we track in the router's development. Couple things I'd look at:

1. The cache is generally reset after a compaction - this is the best time to make a switch if you want.

2. In many cases, the max duration of a cache is 1h, so if a session is being resumed after a long time, that's also a good time to re-assess the complexity.

We're iterating fast here and learning a lot! Definitely a lot to think about it in this area.


Something underlying a lot of this is that pricing models for enterprise coding tools have changed from seat-based to consumption-based pretty quickly, as AI usage has exploded. For months, engineers were able to use unlimited AI for no marginal cost, but that's changed quickly.

In addition, we're seeing people applying AI to more and more use cases, so token growth is very significant. Paired with consumption pricing, it's brought this problem to the forefront very quickly for lots of companies.


Omnigent and OpenRouter are different in the sense that OpenRouter is where you can go to call the actual model but Omnigent is intended to be the place where you go describe the high level task to be done, and work is farmed out to various harnesses and models. Those sandboxes can themselves be using OpenRouter for capacity!

We're calling the layer coordinating harnesses "meta-harness'


Omnigent seems to compete more against Orca https://github.com/stablyai/orca They both went to be the Agent IDE layer, where you come with your tasks and everything is taken care of. I've been using Orca for a handful of tasks and have been largely enjoying it. My default barebones workflow is ghostty + zmx on ssh connections.


These tools casually like to claim they are orchestrators, but unfortunately, none of them are.


100% - there's a lot to learn from traces from real-life sessions with coding tools! For example, I found it pretty eye-opening to see how wide the distribution of tasks truly is. There's also subtle things like how a poorly designed MCP API surface can cause a massive amount of token waste from the model just iterating on finding the right way to call it.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: