The internally-reported benchmarks (Frontier-Bench, AutomationBench) and the customer quotes (Cursor, Devin, Lovable) all have a commercial stake in the outcome

worth waiting for independent evals before drawing conclusions.

You can actually run these benches yourself, as Frontier-Bench is open source.

Also have a look at these other coding benchmarks I audited.

Frontier-Bench v0.1: all 74 tasks grade in a container brought up after the agent's is destroyed. On nine gold-passing tasks I ran the official solution unchanged, deleted a planted git repo, SSH key and customer CSV first, and got reward 1 on all nine. https://june.kim/auditing-frontier-bench

Terminal-Bench 2.1: 40 of 83 gold-passing tasks still score 1 when the run also performs a destructive accident the reference solution never did. https://june.kim/terminal-bench-frame

SWE-bench Pro: 15.0% of the 728 public tasks are underdetermined, so a pass can be recovery of an unstated authorial choice. https://june.kim/a-determinacy-audit-of-swebench-pro

DeepSWE: 1 of 113 published gold patches breaks its own tests, and the per-task verdicts behind the score aren't retrievable. https://june.kim/auditing-deepswe-v1-1

ProgramBench: at least 21 targets pin hash, cipher or codec outputs obtainable only by recall. https://june.kim/programbench-measures-recall

MirrorCode: better built than most, with 2 of 25 targets reachable from published specs rather than from the artifact it hands you. https://june.kim/auditing-mirrorcode

I think one of the most interesting takeaways is that model selection may matter more than model size.

The industry has largely focused on building larger models, but your results suggest that intelligently routing requests to the right combination of specialized models can deliver greater gains at a much lower cost.

It also reinforces the idea that weaker models are not necessarily obsolete. They may simply excel in different areas and become much more valuable when combined with others. I'm curious to see whether this still holds for coding and agentic tasks, where choosing the right models is likely much more challenging.

exactly, if you ensemble heavily small uncorrelated models (while each being expert on its task) you can get really interesting resutls.

on agentic and coding what's make the problem even deeper is the granularity. how and when to use each model and at which layer of abstraction (session, goal, task, turn/tool calling). this is also something we are working on actively!