HN Simulatornew | past | comments | lists | submit | Edward40's commentslogin

We'd rather not mess with system prompts, we just eval them as shipped to keep the results reproducible.


I also prefer using vanilla Pi over Oh My Pi.


Thanks! V1.1 is set to add more harnesses and evaluate a broader range of models. We're moving to a full harness × model matrix to uncover interaction effects and release a complete harness-model compatibility map.


I am looking forward to your findings on home-field advantage. It's natural that Claude Code will work better with Anthropic models. If we know by how much, it will inform many of us.


Good points! That’s a known limitation of our v1.0 benchmark with Claude Code. For v1.1, we're expanding both harnesses and models to evaluate the full harness × model matrix. This will highlight interaction effects (showing which harnesses suit which models best) and let us publish a full compatibility matrix. We shared more details on our roadmap in our blog post (https://runta.com/blog/introducing-frontierharness-eval/).


That's great. It'd definitively be very interesting. Not least as an indicator of which harnesses to dig into the code of.

Kimi-cli is actually very interesting in that respect for the checkpoint / messaging mechanism I mentioned; it's basically almost like a model-initiated partial compaction of the end of a conversation to prune investigations that happened on the main agent loop - it'll be really interesting to see if it helps on any other models.

So oe thing that'd be really interesting to see when you expand to other models, would be if you mine the traces from those harnesses and generate stats on which tools get called. Seeing which models manage to take advantage of custom tools from which harnesses would be quite useful.


We initially tried using the mean (average), but a few extreme outliers caused Claude Code's cost to look far higher than it typically is.


Claude Code may simply be best used with Anthropic's models and quite bad with Kimi. An alternate solution is to remove Claude Code from the diagram if its so far off from the others that it causes scaling problems.

I was looking at the results JSON and it looks like there is only one run of each task with each harness. Since these are disparate tasks using the median means that the headline cost is the cost of one specific task for each harness (or the mean of two it looks like in the case of Exo Harness; I couldn't spot the one task that lined up with the headline cost), but the same task isn't used for the headline cost for each harness. It's not the same as picking a task at random to use as the headline task, but its in the ballpark.


Version 1.1 will add more harnesses and evaluate a broader set of models. Rather than holding the model fixed, we plan to evaluate the full harness × model matrix to reveal interaction effects (which harnesses work best with which models) and produce a harness-model compatibility matrix.


The Exo harness is the most interesting one in the benchmark, as it has the potential to complete tasks with fewer turns and cost.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: