Note: I work at AA. One of the things we are most focused on is getting the best spread of private/public benchmarks in order to:
- Protect benchmarks against excessive hillclimbing which we believe to be beyond overall model improvement (GLM 5.3 gained 16 points over GLM 5.2 in Tau-Banking in only 2 months, near SOTA, but had a more moderate gain in our private AA-Briefcase evaluation where it was still significantly behind the frontier)
- Still make use of strong public benchmarks (e.g. TerminalBench 4.0)
As the frequency of model releases increases, we will also be moving faster [1] to ensure our Index stays up to date to reflect both (1) how people are currently using models and (2) what are the strongest signal benchmarks which show these capabilities.
But isn't there still a meaningful spread of difficulty, risk tolerance, etc. in decision model relevant tasks? Especially at high volumes I think there would still be value in being able to make intelligence/cost/speed/calibration tradeoffs between models
This is based on what we measure from the API - we see different endpoints depending on how they are served can have different cache hit rates (cache hit rate is defined as the % of cacheable tokens that actually come back as cache hits)
With max reasoning, Luna is actually less than half the actual cost to run compared to DeepSeek V4 Flash 0731 with updated prices (based on Artificial Analysis Cost per Task)
As the frequency of model releases increases, we will also be moving faster [1] to ensure our Index stays up to date to reflect both (1) how people are currently using models and (2) what are the strongest signal benchmarks which show these capabilities.
[1] Index release history: https://artificialanalysis.ai/methodology/intelligence-bench...
reply