Anthropic competing with Google instead of OpenAI at the frontier is unexpected, I have been doubtful of Google DeepMind's status as a frontier lab in the past few months, but damn they've been cooking recently
It's important to note that DeepMind and the other two labs aren't playing the same game.
The latter need to raise funds and IPO, but the former is an arm of one of the most powerful companies to ever exist with some of the largest data warehouses around and custom hardware. And they have an endless pile of internal code to maintain already with hundreds of millions of end users already bought-in.
The incentives being different leads to different focuses and product lifecycles.
In MiMo's v2.6 public RL training run, they explicitly trained on 23 different harnesses in order to minimize overfitting on any single harness. I imagine that this technique has to be common in RL training runs given that model providers have little control over where their models are used.
These are cool results and I'm glad that tokens are being invested to eval harness capabilities.
However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail.
I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!)
"Cost per pass" is distinct from "cost per successful task."
Cost per pass is simply the average cost per task divided by the pass rate.
If you draw a line from the origin to any harness on that chart, its slope (pass rate ÷ cost) represents passes per dollar. Cost per pass is the inverse of that slope, meaning the lowest cost per pass gives you the steepest line.
In this case, I am not entirely sure how cost per pass is distinct from cost per successful task given that success is defined as pass/fail.
To illustrate the difference, the author of FrontierHarness included "Figure 3. Cost and pass rate of each harness with each model", in which we can see that Codex CLI has $0.937 cost per pass, but a $0.573 cost per task. Measuring cost per task also changes the winner to OpenCode
1. Tell your agent to eval the skill itself (https://code.claude.com/docs/en/plugin-evals). This is easier and faster, but less accurate, and harder to extend to capture things like code quality.
2. Build your own benchmark out of work your normally do, and then eval the skill on that dataset. This is harder and more annoying (because creating benchmarks is hard) but provides deeper insight into how the skill affects the agent in known scenarios.
It also depends on what the skill is doing. If it's creating an artifact (like a report, easier to eval) then your strategy is very different than if the skill is changing agent behavior (like telling it to use cheap subagents under XYZ conditions, harder to eval).
The importance of doing this grows with the number of people using the skill. Working on my own projects, I don't usually bother. Working in an enterprise monorepo, suddenly the risk of shipping a bad skill is amplified, and evals become much more attractive.
Also note that skill behavior changes across different models. A skill built for Opus 4.8 may not be necessary on Opus 5.5. A skill built for Opus 5.5 might cause Haiku 5.5 to behave erratically.
TLDR: tell your agent to do it, and think about building your own.
This has bitten me so badly SO many times. Every time there's a new model, it's time to re-evaluate the skills. It got so bad I put a lot of the skill behind a CLI.
Evals saved me, but I really wish skills had a `writtenFor` field in their frontmatter so an agent could warn you it was running a skill written for a different model.
Would you know of decent sources on creating (personal mostly) benchmarks? Doing something like that currently for processing PDFs and it's a bit cumbersome, but that's work I guess.
My 2c as someone who is working on this problem is that you can follow the 80/20 rule when making them for personal use - put in the 20% and you can make a good enough benchmark relatively easily. The remaining work is certainly valuable, and necessary to do if the benchmark is consequential, but it's a pain in the ass and hard to do correctly, so if you don't have to, don't worry about it
IMO for an involved voter, this is actually a good thing. AI can both help you clearly define your priorities and policy preferences via an "interview", and can subsequently find candidates who's publicly stated positions, and past record, align most closely with your priorities.
As always, AI can be used to completely outsource critical thinking, and doing so for elections is extremely concerning. However I would hope that more people use it to make better, more informed decisions, than to blindly choose candidates
Pretty cool to hear about your process - I've been thinking about this space a lot so have some context about pain points here.
I've found that building a benchmark is something that's easy-ish on the surface level, but has many layers of technical complexity the deep you go down the rabbit hole.
For example, a tangent on grading. Do you use test passing and/or LLM graders? If you're using an LLM, it's fairly easy to spin up a basic LLM grader. But - is that grader calibrated? Is it biasing towards same-model agents? How many results do you need to be above the noise floor? What is it even measuring, and is that aligned with what you care about? Does it have the right information to make those judgement calls?
I'm super curious how what you've been using this workflow for? Also happy to answer any questions / discuss further!
Yes, I'm with you 100% on the setup. You can go as deep as you want down the hole. At some point I had ask myself what I was really after with the benchmarks and where the line of diminishing returns was (and more importantly, was I really getting generalizable results).
I'm currently using a 2 tier grading system, one pass mechanical (LLM graded) one pass human. The grading ranges from -1 to 0 for the mechanical pass and -1 to 1 for the human pass. A model that does everything competently passed with a score of 0. Non-pass gets a -1. For the mechanical pass that's it. For the human pass there's a chance to get an extra point for exceptional work. Accepting the subjectivity in the human pass was part of the "what am I doing this for" point from above.
The scores then don't get read left-to-right and summed, but rather vertically per task and category. The temp setting as another commenter noted means you can't say anything definitive about the quality from a single run. What I think you can tell is 1) per task, who is the outlier in either direction and investigate, 2) how a task is generally handled by the models and 3) in the aggregate, who is dropping and gaining points consistently and does that change per task type. In the end, it's high touch. It doesn't give you one number to cut or keep a model. It tells you where to look and where not to.
You're the second person running an evaluation business to express interest, maybe I should start one as well ;) I will be reading your blog posts to see if there's anything I can adopt. If I was to give you the objection I would pose if this was an MBA brainstorming sessions I would ask:
Who are you targeting? and if the answer is engineers, what's to stop them from using the LLM to do it themselves? The problem space exists in the easy/hard domain; easy to set up, hard to refine because of nuance and specifics. Moreover, an engineer whose job might be under pressure because of LLMs would be highly motivated and highly capable to create a testing and evaluation setup and "own" this part of the process since it can't be replaced by an LLM.
Personally, I think this space is good consulting territory. There is hard-won knowledge that is valuable and sale-able, but beyond the basic setup, everything has to be tailored. It's also going to be hard to get people into a product moat. Git and IDEs give customers (if they are engineers) all the tools they need for this work and they already know how to use them.
One last note of your comment about same model agents. This is definitely something I have noticed. It's probably structural, but models absolutely prefer their own outputs to others. You can also move them toward agreement or disagreement by telling them that their model (in another session) produced some output. Kimi in particular will dunk on other models and promote what it thinks is its own output. Kimi is also highly sensitive to being bench marked. I have a specific audit script that goes through thinking traces to see if the model realizes it's being tested. Kimi figures it out more than any other model and will dig big time to find other results if you don't completely isolate it. Claude gets pissy if it thinks you are trying to benchmark it. Fable has inserted several times "not under testing" and "not a competitor" when left to its own committing. Fable is also very good at setting up open weights models and testing them. They must have spent a lot of time going for thoroughness and scientific method in training.
Artificial Analysis shows Grok 4.6 taking $1,068 to run their suite while Gemini 3.7 Flash takes $485. So it looks like Gemini 3.7 Flash is less than half the price in the real world.
Per-token cost isn't a great metric given that some use way more tokens than others.
Agree, with you but I'm still using 3.6 Flash because of tok/s/ latency/ uptime with high context. Tried Grok 4.6 and it was scoring lower on some internal benchmarks or slower.
> Hard to understand why anyone would choose 3.7 Flash under these conditions.. is Deepmind still a frontier lab?
At this point, I think they're mostly targeting Google One and Workspace subscribers, except doing worse compared to Microsoft because they don't have Microsoft's huge enterprise moat built from their DOS and Windows days.
This approach seems fundamentally predicated on being able to evaluate coding agents on your own code by having domain specific evals. With that knowledge, you can trust the routing logic is actually improving/maintaining perf while reducing costs.
Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity
I'm actually working on building generic repo-specific benchmarks at https://stet.sh ;)
The difficulty of evaluating coding agents is indeed a really big challenge. We built evals on our own codebase and shared some information about that to allow other companies to replicate. We found our own evals correlated loosely with public generic SWE benchmarks.
In large user populations like at Databricks I think the ultimate answer will come from experimentation instead of offline evals. We are already doing this in small groups, exposing them to new candidate models and then measuring per-developer cost and perceived quality changes.
Experimentation makes a lot of sense as agents need to be pleasant to work with, and power users are a willing and able cohort to do that differentiation. Agent capability is only part of the equation, as evidenced by the widespread frustration with the experience of working with Opus 5, despite it topping benchmarks and being objectively extremely powerful/competent.
However, as a self-admitted eval true believer, I remain bullish that publicly available eval capabilities will catch up to current model intelligence, and start to be able to capture more and more of the subtle components of the agent experience, such as interactivity and code maintainability. Meta and Scale have some evals in the interactivity space that seem pretty promising https://arxiv.org/abs/2606.29957https://labs.scale.com/papers/swe-interact.
Human usability of the agent should definitely remain the top priority, but IMO we focus too much on the subjective, and not enough on the measurable/objective now, and I think/hope we'll see more companies move in the direction of Databricks
reply