I am always interested in this comparison since it shows how quickly you can go back and forth with a model, capturing both tokens/task and tokens/second. The new models seem to be meaningfully better here.
Note that the graph is misleading for open models since it's only measuring their official providers and that's rarely the fastest.
For example, the official provider for Kimi K3.1 Pro is dead last for speed on OpenRouter. You could run it more than 5x faster for the same cost elsewhere, which should push it comfortably into the pareto quadrant.
Why does AA choose ~$3.1 per task to be the breaking point for "most attractive quadrant" highlight? Surely the most attractive section is just being close to the entire frontier?
The choice of X-axis maximum and Y-axis maximum dictates the cutoff point of the quadrant. With a maximum of $80 per task, the quadrant is $3.1 and lower. If some new model gets released and costs more than $80 per task, the "most attractive" cutoff moves up from $3.1 since they have to move the X-axis max. It's completely arbitrary.
It is suggesting a hard cutoff since it says "most" attractive, a superlative, instead of "more" attractive. And it is not pareto optimal, because if the models get really good and are scoring 10x more for 10x cheaper that green region will no longer be pareto optimal.
It combines nonograms with crosswords, requiring both logic and word knowledge to solve. I am still working on the interface and tutorial, as well as the puzzle difficulty levels.
Or the Grey men in Momo (Michael Ende) who offer time savings through efficient appliances, while stealing the time made free through their use. Their main for is Momo, who encourages her fellow children to play for the sake of playing, which terrifies them.
It shows the time to complete a task versus intelligence index, which is much more important for my use-case as I do quick back and forths usually. Someone on here suggested it to me a while ago. It also captures speed and token efficiency nicely, since both of these decrease time per task.
We already have UTC, which is almost always used as the legal basis for all local timezone definitions. In many industries (such as in many aviation and maritime operations), UTC is used almost exclusively to synchronize things worldwide and to avoid any local timezone confusion that might cause problems.
So using UTC works great in situations where participants span multiple timezones. But I think the intuition of maintaining a local clock for local things, with solar noon kept somewhere approximately around clock noon, is still useful, since many day-to-day scheduling concerns remain entirely within one's local timezone. The key is that participants must be aware of their local timezone's offset from UTC. Not having to change their clocks twice a year would hopefully also make this easier.
This looks great! I also think speed should be part of the metric (i.e. how long does the model take to actually solve a task). For me, I prefer to run expensive models such as Sol on light reasoning, which usually gives me good answers with quick responses.
For my style of coding (quick back-and-forths and corrections) it makes a big difference if a model comes back in 1-2 minutes compared to 5-10, and I am happy to pay a bit extra for that.
I agree on both: indeed a nice article, and I'd like to see a chart with speed as x axis.
As someone who only needs AI for a couple of tasks per day, I don't really care how much it costs, especially when subscriptions are subsidized. I want to filter by speed (eg, max task time < 1 min) and then choose the intelligence I need for the task. This will surface models like Gemma4 31B (xhigh) running on Cerebras and GPT Sol (med) fast mode. Using these models feels great and are affordable for infrequent tasks.
I think this is the most import graph on their page. I wish they would let us filter by intelligence, or pass rate, and then see this graph. This is the tradeoff that actually matters, cost/token or tok/s can be very misleading (take glm-5.3-flash as an example).
The more dimensions you take into consideration, the larger the proportion of the Pareto frontier becomes across all distributions, and making it harder to choose.
Every day, players answer 5 questions that require estimating some quantity, such as distance, year, cost etc. Then you can compare against answers of other players and the real answer.
Working on finding the right balance of questions that are both fun and also help understand facts about the world.
I am always interested in this comparison since it shows how quickly you can go back and forth with a model, capturing both tokens/task and tokens/second. The new models seem to be meaningfully better here.
reply