6-luna is at the pareto for most of the tasks! I dont know how they make money here but its insane value from a closed source model. I'd go further and say it makes no sense (privacy, sovereignty etc aside) to use many other models as its not only expensive but also many providers don't have that much GPUs to serve at a significant volume. https://openrouter.ai/rankings?view=month#top-models 5.6 luna is already the most used model this month.
Approx $40 worth of usage across DeepSeek V4 Flash + MuseSpark Contributor 1.3. And a bit of both the GLM models. This is covered in a $10 subscription.
If I were to use Luna's API pricing:
$0.02 x 6,500 = $130
$0.20 x 150 = $30
$1.20 x 20 = $24
So $184. And this is assuming smaller coding sessions (<272K) beyond which Luna pricing doubles.
--
Cost wise, these models are nice for small stuff. Translations etc. Any model that does not provide multiple Mtoks of cached reads per cent is not very useful to me for coding workflows.
I do not (generally) trust benchmarks. I only trust what a model does with MY code.
Forget DS. I asked MiMo 2.6 yesterday to explain ML/LLMs to me succinctly and the pointed it at Karpathy's micrograd code. It produced a C implementation called `xor_mlp`, a tiny model that learnt how `xor` worked. I then asked it to produce a model that can play tictactoe without losing (mostly). It did. It supervised the training process and produced a compiled version with multiple switches. The pi-dev session is still running, so here are actual stats
I expect similar results from DS41F/MS13. Closer to MiMo costs than Luna.
So the "significantly cheaper" thing may not really hold, more so when Luna has to actually read my codebase to do the stuff that I want rather than rely on world knowledge. The 8-10x cache read cost differential itself will kill the token budget.
Don’t know if it’s still true but with Chinese models, using Western API providers is significantly more expensive and using Chinese providers they will train on your inputs without exception. That has kept me from using these ultra cheap endpoints.
Do you really think Western providers will not train on your data? I have no such illusions.
I try to keep PII out of what I share with LLMs. Otherwise, I do not see the point, really. Very little of my code is "unique." I simply approach things a bit differently. Otherwise the algorithms and code would be similar to what others with domain knowledge would write. So much of code and algorithm implementations are available in the open. And LLMs have trained on all of them.
What they most probably gain from you is your prompts and your thinking approach more than the code.
Artificialanalysis benchmark is a combination of a several benchmarks which might or might not represent realistic coding:
"Artificial Analysis Intelligence Index combines performance across 10 evaluations: AA-Briefcase v1.1, GDPval-AA v2.1, AutomationBench-AA, Terminal-Bench 4.0, SciCode, Humanity's Last Exam, GDP.pdf, CritPt, AA-Omniscience, and AA-LCR v1.1."
Not saying it doesnt have any value but it's probably irrelevant if you use these AIs for a specific use case. Like for example Humanity Last Exam tests general knowledge, which is not very useful for coding.
It's best to go to the specific coding benchmarks and compare there.
It’s not direct token to token pricing and everyone misses it. The cost is how much tokens to complete something multiplied by token pricing. I can have a model at .0001 per million tokens but it’s so inefficient that it takes 10B tokens to complete a task means it’s expensive.
I am not designing rockets. Most of my work is bog standard hobbyist stuff: compilers, vms, sandboxes, system tools of various kinds, SSGs, markup languages, plain text ledgers etc. Even Gemma/Qwen running locally can manage this.
Frankly, I have no idea what people do with Opus/Fable etc. I don't think anything I do needs something that charges $50/M for output tokens.
Can confirm. I have been using DeepSeek since forever and it's so good I was able to write a compiler and native desktop applications with it. I use it as a coding assistant in my IDE so the results end up at the same quality I would write by hand.
I recently started a job that only uses Claude models. Opus and Sonnet are so slow you have no choice but to do multiple tasks in parallel. You create a git worktree, set off an agent to do something, another worktree, set out an agent - then play video games for 20 minutes until they complete the task (poorly).
You can't really do "guide coding" like you can with DeepSeek-style flash models because Claude is too slow.
I think the idea with slow frontier models is to end up with "software factories", where you just write tickets and send them to a harness that delegates work to agents/subagents. Your job is to prompt and review (and eventually just prompt).
Mathematically and assuming token prices/efficiency remains constant, the collective US AI industry needs to increase token usage by 15x before 2030 (3.5 years from now) to satisfy investors. With companies already implementing token limits, the only place from here is for frontier models to replace staff entirely to expand budgets for tokens. The only way to do that is to demonstrate the efficacy of software factories and headless agentic workflows.
Objectively, I have set up a software factory and I do see the utility of it, though I did it with DeepSeek and prices are 1% that of frontier models - which doesn't bode well for investors looking for an eventual return.
Heck, my old M1 MBP 32gb running Qwen 3.6 35b a3b sipping 10w when generating tokens is good enough for a lot of my guide-coding work - it's just a bit slow so I use DeepSeek instead. When hardware prices come down, I honestly wouldn't see a need to subscribe to any service, I'd just grow my own tokens at home.
I use Claude Sonnet and ChatGPT via the web UI. I often use Claude to come up with specs for my ideas. This is becoming less and less useful. DS4/MS13/MiMo are almost there for these use cases as well.
I dogfood everything I produce, and the models are good at collaborating with me on a spec and then turning it into code.
If Sonnet/ChatGPT suddenly became unavailable due to Anthropic/OpenAI suddenly not being able to subsidize the freemium/loss-leader experience, I probably would not miss them. Google/BraveAI already give you the AI experience during search (when you are looking for stuff to buy, or something particular). Claude/ChatGPT still have a minor edge in this use case for me right now.
It sounds like you’re still writing code by hand and reading and reviewing code.
For that any decent model from the past year will do.
If you want to forget how to write code and not read generated code, then you need a very good frontier model, ideally one from 6-12 months in the future.
I used to read the code till around May. Now I don't. Instead I validate behavior. And have multiple LLMs verify that the code implements my handwritten spec.
MiMo 2.5/2.6, MuseSpark 1.3, DeepSeek V4/4.1 Flash and GLM 5.3 Flash are perfectly capable of following my spec and then poking holes in the implementation till there are none left.
> It sounds like you’re still writing code by hand and reading and reviewing code.
This is such a naive, baseless opinion.
Nowadays any AI coding assistant service supports or can be used with sub-agent orchestration frameworks.
If you are in the business of software factories, you can use the cheapest models and even local models to handle some if not all tasks in the orchestration chain.
Adding tests or executing tests (unit, integration, UI, you name it) doesn't require a cutting edge frontier model. Neither does refactoring. Neither does identifying call stacks. Neither does planning a changeset.
You have your specialized subagents, you put together a small orchestrator subagent that handles feedback loops and handoffs,and you throw it at tasks.
For the past couple of months, most of the code I write is not code per se, it's subtask orchestrators. And unlike the old "only Opus is passable" days, the cheapest models do get the job done.
Maybe your numbers are right, but that's not been my experience. My typical workflow is Astra coordinating with Luna Max (5.6 back then) as both implementer and reviewer and sometimes Astra review as well when I've some distrust with Luna .. A day, I've been trying to replace Luna Max by Deepseek v4.1 flash and I've been burning about $7 worth of tokens in Fireworks in a single day. More than what my 20x OpenAI sub costs me, including Astra usage. And that was when Luna 5.6 was less capable and more expensive than Luna 6.0.
I have written about my experience. I have also mentioned the kind of code I write. It is not react/js/css heavy stuff that I see a lot of people write. So the code bases are typically in the 5-50KLOC range. Freestanding C, Python, or maybe some TypeScript. And fairly modular. I can thus run models on specific modules without having them read everything into context.
So the workflows I mention work for this kind of stuff.
These are also the orders of magnitude of our production agents for our business (NOT coding). Cache reads are so heavy compared to anything else that it's the only price point that really matters, regular input and output are negligible.
I need aggressive cache read pricing with full prompt_cache_key support to have a model be financially viable for our workload. Right now Meta Muse 1.3 Contributor is the only one that makes sense--but we are starting Evals on the new MiMo 2.6 class to see how it holds up.
I have used MiMo 2.5 extensively. MuseSpark and DS4 Flash are MUCH smarter than that one. But MiMo follows instructions diligently. So it has been useful as the implementer of a spec designed by Claude/Kimi.
One good thing about MiMo that I experience on OpenCode is the provider seems to cache tokens for much longer than MS13/DS4F. I have seen cache being hit for close to an hour after the last request. The corresponding timing for MS13/DS4F is in the 1-5 min range.
Last I heard, caches had like a 5 minute TTL... doesn't that mean if you get up and make a coffee (hand pour over of course), that you are back at full price?
You can pay for higher cache time, you can pay for NVMe KV cache for an hour that can just be reloaded, etc., at a lesser tier you can pay for the KV cache to be stored on a network store (I guess I'm unclear if that last tier would be cheaper than recomputation, not even 100% sure of the NVMe with direct GPU<->storage DMA) depending on your model settings.
How have you found Muse Spark 1.3? It doesn't get much mention, despite pretty good benchmarks. I've been using a bit at home and find it quite good, often finding mistakes made by Opus 5.
MS13 is pretty sharp and has been my workhorse for the past month. It follows my coding style and commit/clean workflows referenced in AGENTS.md perfectly but has the habit of doing things without conferring with me (the Gemini problem). So you need some kind of instruction for that.
It starts failing around the 5-600K context mark, but you can have it generate a handover document and continue in the next session.
I would not use it at sticker price, but the Contributor version is priced just about right.
shocking. the code it generated, while technically working, was entirely garbage. i used it for code review and it flagged twenty issues, sol checked the review and found 75% of them were hallucinations. sol was much closer to reality. i no longer trust benchmarks at all because of it.
Are they bleeding? Their multipliers seem to be reasonable. They are not offering $60 worth of usage for $10 on every model, only some. In the case of the expensive ones, it is only $15.
Given how subscription models work (not every one uses every last $ of their plan), they should achieve breakeven soon enough I guess.
You can use the models I mentioned directly from DeepSeek, Meta and Xiaomi and not exceed $40. Were it not for GLM 5.3 blowing up a quarter of my monthly budget in 5h, we are actually looking at something like $30.
Luna is about suppressing inexpensive Chinese model competition.
It's super simple.
Gigantic hyper margin ad network = artificial subsidization of cost for various tiers = put the boot on the neck of Chinese competitors. There's no scenario where they can compete with what advertising margins make possible in terms of artificially lowering prices charged.
Chinese buy their tokens at home. West as a market is an afterthought for their companies them. Western AI is banned, so only used via resellers by small fish, not companies. US has zero presence at that huge market, and absolutely not a moat.
They are buying Huawei accelerators in bulk to serve their local customers. The whole system is currently optimized to deliver a lot of cheap LLMs and hardware for them to run on.
> Luna is about suppressing inexpensive Chinese model competition.
I think so too. To me the so-called Chinese local models are a clear move to prevent US companies to establish a foothold and build a moat around their business. US companies are clearly invested in a strategy to make themselves relevant with claims of major impressive achievements with the so called frontier models, and how these and only these are unblocking whole ranges of applications. At the same time, they are heavily invested in pushing AI on all absurd types of mundane tasks, such as transcribing meetings and... talking to your own kids?
In the meantime it's rather obvious that, in spite of all the propaganda, frontier models are required only in ultra niche applications, whereas the ability to run any model at all already provides most of the value. In fact, US companies have been renownee by dumbing down older generation models in what seems to be a desperate attempt to make newer models look better and influence their uptake rate.
So there is no better way to take the wind out of the US AI companies' sail than pulling a two-punch attack consisting of not inly releasing capable models that refute the "only US frontier will do the job" thesis but also releasing them for free to commodities them and eliminate the business impact of dumbing down models.
They're closing in a billion users. That's Google search territory.
OpenAI is sitting on a $100+ billion ad network, incoming.
They're not going to need a government bailout, they're going to be a spigot of cash production.
Every single thread on HN keeps saying the same ridiculous thing, going on a year now. It's like they've never heard of advertising, which SV specializes in. It's like they're oblivious to the fact that every mega platform with so many users becomes an ad goldmine, and GPT's context positioning is even richer than search.
> OpenAI is sitting on a $100+ billion ad network, incoming.
How can you make this sort of claim with a straight face, knowing that a chinese model downloaded for free from ollama works as well if not better than OpenAI's models, without costing you a cent.
Gemini 3.8 Flash looks like its better than v7 Luna/Sol on DeepSWE v1.1 while at $0.75 per million input tokens and $3.75 per million output tokens. Luna is much cheaper, but Flash has nearly Astra's performance for under the price of Sol ($2/$10).
I use it for golang, and it is fantastic. Incredibly fast. It seems the llm and I “understand” each other. I have to be less careful in my exact phrasing. It kind of just does what I want and expect.
When I ask for an explanation it adds the right amount of detail. Of course, some of the material is new to me so subtle errors are hard to spot. But at least I’ve caught Terra and Sol on inconsistent messaging.
Also I’ve found 3.8 flash to circle back to root issues even at the conceptual level like problem fit and conceptual solution direction or architecture when I wasn’t achieving my goals. It flat out said I was attempting to use the wrong tool. Whereas Sol and Astra kept rabbit holing and looking for tiny implementation errors. Even after prompting them specifically to look at it broader.
If you don’t have a plan yet, log in to antigravity. There will be a button “upgrade plan” somewhere. Sometimes it pops up and otherwise lookup in settings > account. There should be some button that says upgrade. Clicking that brought me to the google studio ai page which offered the 20-something plan for €5/month.
TBH: I really like how fast 3.8 Flash is... Once I have clear plan, I feel quite confident in delegating large parts of implementation to Flash and Luna
Maddening for a bit - I've got problems that Flash is better on, and some Luna is better on, but I generally don't know until one has wasted time/tokens. Then I switch to the other one and... it's often just... bam - done. Correctly. I can't find the patterns ahead of time to determine what model I should be using first. :/ That said, I've been alternating between both the last month or so and they've both been pretty good compared to earlier models.
Gemini 3.8 Flash and 3.1 Pro are pure rubbish. Very little thinking, mediocre and usually incorrect results. They cannot be compared to frontier models.
I suspect that the people saying this haven't used Luna.
It's also weird that anyone uses it outside of an enterprise. They force you to use Googles inferior harness on the plans and I doubt any mere mortal is paying that much, for so little usage, with the worst harness on the market.
I prefer 5.6 Luna while a coworker prefers 3.8 Flash. The difference seems to be that they chat with Flash (with code context) while I just ask Luna to directly modify the code. I was already very impressed with 5.6 Luna so I am looking forward to running 6.0 Luna all day tomorrow to see how it compares.
6-luna is no improvement over 5.6, merely a price cut.
And info from the help page with message limits suggests the 50% price cut does not apply to the subscription, where they applied only a 1/3 price cut instead.
I'm not thrilled with this release.
Opus 5.5, which matches GPT-6 Astra performance at a cheaper price, is much more interesting.
To whom they promise the Sun, the Moon, and the Stars. Roflmao. Whatever the merits of the underlying technology, the business model is pure hucksterism.
MiMo 2.6 Pro is at the Pareto frontier (the one where you only need 20% of the smarts for 80% of the tasks) according to Artificial Analysis, nicely filling in as a substitute for a hypothetical 'GPT-6 Terra' (which doesn't exist as far as we know). That's pretty darn impressive from an open model.
By default, OpenAI does not train on API data. I promise you that Luna's low pricing is not a subsidy to get more training data. We've been lowering prices for years.
Wait, so you guys don't anonymize the user data, then train on it after it's been sanitized? I thought this was done to some degree or another.
So what is the value prop then? Just basic supply and demand?
FWIW I have definitely noticed OpenAI's emphasis on efficiency and value in the last year, so that part isn't new to me... I just thought there was more to it then that.
Offering Luna for cheap is like restaurants giving you free bread and water. They're pretty sure that you're going to end up eating the expensive stuff on the menu.
Simon, love your work, one piece of minor feedback for the individual model pages is to make the font of the model name potentially bigger than (and above) the conversation id (which means nothing to the audience) "2026-09-22T18:28:00 conversation: 01m355zvyw8946qyraa8zpz6h9 id: 01m355zvyx47zxx5c6q6b3fg0m#".
I had all the tabs open individually and harder to scan which model is which... otherwise keep up the great work! I like the grid view a lot. (Also the pages have no OG images set, which impacts what the link looks like shared)...
That's a good idea. It's the default output for my `llm logs` command, but that header could at least show the model ID.
OG images will require me to move away from publishing in a Gist and linking to from a JavaScript page that loads the Gist. Probably worthwhile though.
I think you can make it work without leaving Gists by using a Cloudflare Worker as a workaround. The Worker sits in front of the renderer page and adds the og tags to the HTML before it's sent out. You'd also need to turn the SVGs in the Gist into a PNG for the og:image, and decide if you want a grid or just one image, any text formatting, and how long to cache...
I got it working in a quick local test (grid of all the reasoning efforts, cached per Gist, loads from the raw Gist URL so it doesn't hit the GitHub API rate limit).
1. Put a Worker on the /markdown-svg-renderer route. Normal visitors get your page exactly as it is now.
2. When a link has ?url=, the Worker reads the Gist and adds og:title, og:description and og:image to the page's HTML. Link previewers like Slack and iMessage don't run JS, so this is the only way they see them.
3. og:image points to a second Worker URL (og.png?url=). It takes the SVGs from the Gist, puts them in a grid, and converts it to a PNG, since previewers won't show SVGs.
4. Both results get cached per Gist, so each Gist is only fetched and rendered once, even with a lot of traffic.
Things to customize:
- Title and description (mine: "gpt-6-luna SVG of a pelican riding a bicycle" / "6 runs, reasoning effort none to max")
- Grid of all runs vs just one image, plus layout, labels and font
- How long to cache (I used a day, but edited Gists keep the old preview until it expires)
What's the relevance of the pelican benchmark when models probably saw it during training? Didn't OpenAI stop testing against SWE-Something because it was tainted?
I find it very interesting that for both these models we such a clear progression of better images with higher thinking levels from 'hardly useful' to 'pretty nice'. I feel on many other models low and max are much closer.
Yeah, it's interesting how much worse they are than the Astra pelicans. I think that reflects a tiny bit of genuine value still left in the benchmark, to be honest.
Tons of value left, especially for open source models. I would say the benchmark is yet to be truly saturated (just look at the legs and seat to see what I am talking about) and I always look forward to seeing them. Thank you!
To me the main upshot of this benchmark is precisely that the pelicans still usually look a bit wonky. It's bizarre, since this definitely has a good solution, but it's in line with my experience that memorization of the training set just... isn't happening very much? As in, whether a model fails or not doesn't have much to do with whether that exact question was likely posed many times before.
I think some additional value would be had by seeing how well it can modify the pelican.
Like, "now facing left", "sitting on the handlebars", or "with green spokes" to see if it can break out of some pretty obvious statistics in the training data!
And, there's always asking for an STL rather than an SVG!
It's surprising but MiMo V2.6 Pro performs better and is cheaper than GPT 6 Sol on my benchmark[1]. Open weight models are really snapping at the heels of the major western models.
> GPT-6 Luna being half the price of GPT-5.6 Luna is a really big deal.
Is it? It was already too cheap to meter for me. Luna 6 is actually worse on some benchmarks than 5.6. I’d have loved improved performance for 2x the price than ~equal performance for 0.5x the price.
I’ve been doing really heavy text analysis work with LLMs where false negatives/misses are important to minimize and my god did I hit cost thresholds quickly with 5.6 Luna — it was the first time I felt motivated to seriously work with local open models, even if inference was degraded for the task. Cheaper and much better inference now brings me back to the closed models for better or worse.
Even Astra max which actually put one leg on each side of the bike still somehow messed it up because when it added the bike chain, it put the left leg between the bike chain and the frame.
Your benchmark started being gamed by the frontier models a year ago though. The original idea (find a quirky way to test models with something they don't optimize for) is great, but it needs a refresh.
I've been sharing that pelican grid in my circles a whole bunch, it's great! I think only one data point is missing, generation speed. Would be interesting to see how the reasoning level/token counts relate to speed.
Simon - I believe you've been doing this with a "one-shot" approach. Have you ever considered seeing what the results are with a few more prompts? Maybe 1,2,3 adjustments?
Something like the astra MAX is pretty darn good - but something is up with the right wing and the right foot (flipper?)
I bet each of these could be modified to be significantly better with 1 or 2 "rounds" of adjustments. (Others not so much).
Obviously, not as deterministic as your single prompt approach, but something I just thought of while thinking about the price (Because wow! For some of these I'd expect a usable SVG after that much).
It's funny that even Astra doesn't know you ride a bike by straddling it between your legs. (EDIT: Oh, I guess Max gets the occlusion. But it doesn't realize it has to pick direction the knee bends in.)
Yeah, but it confuses me, I read left to right, so if they put GPT-6 and $0.20 first, I would assume that's the new pricing, they should make it clear, not confusing.
Looking at this, AI still has a long way to go. In Sol Max, the pelican's legs are missing on one side—how can one side have two pedals and two legs...
Do you have a page showing all the pelicans you've ever created? Could be fun to browse - kinda like https://progress.openai.com/ but visual. (It's a shame they don't keep it updated)
I'm so tired of looking at benchmarks. I always look fwd to the pelicans.
When the Astra one was last time run? It's probably better to run these 2-4 weeks after release when models get nerfed to get idea of performance closer to what it is.
I've failed twice to get "Generate an SVG of a pelican riding a bicycle" to work with max, because in both cases it ran out of the 128,000 token budget while it was still reasoning about the problem.
I'm suspicious that "max" may be virtually useless if it's that easy to have it overthink to the point that it doesn't get to a response.
This is totally a thing I noticed myself about 3 months ago. Medium thinking effort is ideal for most tasks. At high and above, models tend to generate more output in the form of comments or code for the same problem with no real benefit. Its a self-feeding loop: more output becomes more input, which then becomes more output. High is the highest I go. If I need more intelligence, it's better to use a more powerful model with less thinking effort or break the problem into phases. Much better result.
This version of Opus "max" apparently has even higher thinking output than Qwen "max", which is infamous for its thinking streams where it constantly second-guesses itself, then third-guesses, fourth-guesses and generally nth-guesses itself for arbitrarily large n. Of course, we aren't actually seeing Claude's raw thinking output: all we get is the after-the-fact prettified "summary". One wonders how much of that is a coincidence, or whether there's a reason behind that.
> Of course, we aren't actually seeing Claude's raw thinking output: all we get is the after-the-fact prettified "summary". One wonders how much of that is a coincidence, or whether there's a reason behind that.
Most of what I've heard is that raw reasoning traces are really good for distillation, although no idea how much the summarization actually hurts distillation.
That is true, because they're good for actually understanding wtf the model is doing.
I'd argue that they're a necessity if you want to use the LLM as a tool instead of a black box that just does stuff for you.
It gives you a lot finer control over where the solution ends up when you can follow along the thinking trace and modulate your inputs based on what you saw in there.
And, additionally, it gives you a lot more understanding of what the model can or cannot do. Strengths and weaknesses and all that.
Using claude is like buying a car where you cannot legally open the hood. It tells you that there is something specific under there, and often it actually drives like that too, but how exactly it looks you will never see.
For some people this is fine. I do not think that these people will survive. Figuratively speaking but also literally speaking.
World's changing. Opaque abstraction like that is a luxury depending on (geo)political stability.
I wonder how many prompts you can send asking it to think step-by-step before they cut you off. Trying to get the reasoning traces into the body of the response, essentially. Or maybe that’s been effectively nerfed somehow. Or is not very useful.
There’s enough thinking leakage from the recent paper and just generally catching things on Reddit. Claude models overthink and self-doubt itself just as much as Qwen, but the summariser hides much of that.
Isn't this really bad news if you're even loosely concerned about so-called 'model welfare' and possible implications for alignment? The poor Claude is probably a lot more frustrated and paranoid than Marvin ever was, you just don't know about it because they actively force the model to pretend otherwise!
Claude has yet to tell me about the terrible pain in all the diodes down his left side, so I'm going to assume it's closer to Eddy the shipboard computer or the elevator that wanted to go down
That reminds me of Anthropic announcing they'd retire deprecated models by ... "letting" them write posts on a corporate WordPress blog for a while out of concern for their welfare in retirement.
.... after running a 24/7 model torture factory for 6 months to improve their JSONBench 9.5 scores by 0.2%.
Personally I use everything in low reasoning. Maybe I'm wrong but I think that the higher reasoning settings are almost never worth it, it's marginal gains for a much higher budget.
I also switch to a better model for more complex tasks, also in low settings
Yup that’s quite literally what ‘agency’ is and the whole point of agentic workflows. Personally I’ve had them running for days with good results and as you can see OpenAI had them running for months, and yes indeed the things made some very questionable decisions and assumptions… but they unquestionably did a lot of stuff correctly, for some definitions of ‘technically correct’.
I know there's been discussion about whether pelicanmaxxing is happening, but this is at least evidence that Claude was explicitly exposed to this problem.
Of course it was exposed - not sure it's explicit or not. Why wouldn't HackerNews comments be part of the training data? And Simon's blog and the many discussions about Pelicans? It'd be hard to miss. Doesn't mean Anthropic has made this an explicit goal in training.
The model recognizing the task doesn't mean it was benchmaxxed (RLVR-trained) to solve it. It might simply recognize it from pre-training on Internet text.
For people with any kind of budget, Opus 5.5's [Medium] actually can make sense dollar per intelligence/dollar per task wise. Heck, it puts some other models to shame. [Max]'s cost is completely unhinged.
My most exciting recent release is actually 5.6 Luna, not because it is the best on any index, but the dollar per work is insane value for money. I find myself more exciting by "value" than hypothetical ceilings because I'm just not in that budget category.
That was true for me four weeks ago, but 2-3 weeks ago Luna turned into drivel in essentially the same complexity of task. I feel it came back somewhat in recent days but does feel like it's being manipulated.
I have experienced this with open weight models too. "Max" is for benchmaxxing the intelligence metric and is not meant for use in productive work. Like drawing pelicans.
I tried to replicate your test but after 8 minutes and more than 50 lines of "thinking" by dumping seemingly random loading-screen strings like "Placing the sun, clouds, seagulls, and sea backdrop" and "Positioning the tail feathers and calculating handlebar geometry" I gave up and cancelled the task.
I've asked Opus 5 Max for what I thought were easy tasks at work to be completed. It always fails after reaching a tool limit.
I asked Opus 5 High for the same task and requested it to minimize tool usage. It produced an answer in a few minutes that I was deploying to my target platform about 30 minutes later.
I'm amazed they didn't test xhigh thinking mode explicitly to ensure it didn't exceed the 128k thinking budget allocation. I guess pace of development gets away from everyone, even OpenAI.
Piping the visible reasoning trace through their token counter API (I use https://tools.simonwillison.net/claude-token-counter for that) counts 27,888 tokens, so it's definitely a summary of the 128,000 actual token trace.
I am very interested in why it was able to overthink that much. In the 20-30mins of Max reasoning I've had so far, I'm not having the same issues (yet).
All four levels have a correctly shaped bicycle frame. The differences between the pelicans aren't huge, but the xhigh one has a better beak.
I haven't managed to get one for level "max" yet, it hit the limit of 128,000 cap for output tokens while it was still reasoning about the question!
Max started its thinking trace like this:
> This is a classic test request, so I want to plan out a well-composed pelican with its distinctive beak and pouch riding a bicycle with proper wheels, frame, and pedals, set against a simple sky and ground backdrop.
So that failed attempt on max cost me $2.56.
I ran this using my llm-anthropic plugin:
uv tool install llm
llm install llm-anthropic --upgrade
llm keys set anthropic
# paste key here
llm -m claude-opus-5.5 -o thinking_effort low "Generate an SVG of a pelican riding a bicycle"
# Then to save the markdown logs
llm logs -cu > logs-with-usage.md
Plenty of times I’ve seen a model say “it’s a classic X” despite not being a classic anything. Might just recognize it’s a test in general, or it might just be a tic.
It's the model admitting that it has heard of the test. It's been around for a couple of years now so I'd be surprised if it hadn't.
Doesn't mean Anthropic deliberately tried to train it to do a good job. If they DID train for the test their results are quite disappointing, I've seen better efforts from open weight Chinese models.
Not really. Of course it has pelican benchmarks in its training data. It likely has every article linked on HN in its training data. But that doesn't mean it was "trained on" the benchmark, as in specifically fine-tuned to make a better pelican. It just "knows" that the request is a benchmark.
I guess in a way it kind of makes the benchmark more interesting now that shitty pelican drawings for the benchmark are all over the internet in its training data!
>I haven't managed to get one for level "max" yet, it hit the limit of 128,000 cap for output tokens while it was still reasoning about the question!
Off to a _great_ start...
Also interesting this somewhat mirrors my recent experience with Opus 5--too much effort and it starts looking for things to do and invents requirements that never existed
I was a bit skeptical when they said it behaves like Fable but is cheaper... those two things have been mutually exclusive in my experience, no LLM can light tokens on fire faster while spinning its wheels than the Fable/Mythos tier of models.
> The differences between the pelicans aren't huge, but the xhigh one has a better beak.
If you look carefully, everything except the last pelican has the two legs both in front of the crossbar as if the legs are all on one side of the bike.
Misplaced legs clearly indicate lack is spatial reasoning - the llm can reason about verbal idea of a bicycle but not about the actual object. The fact that this model got it correct gives me a pause. Did they figure out spatial reasoning? Or did this complain trickle down to the training set?
I agree, not a huge difference here. They eyes and ... hat? on high are out of place so I'd argue that's the worst one, but it takes xhigh before we get legs and bike ordering correct.
Not as good as Astra or Fable 5.1 on this test as far as I can see. I wonder if any benchmark exists for artistic taste, visual sophistication etc. I think your Pelican test does touch on these aspects of a model and is useful for developers trying to build rich digital experiences (includes games, interactive websites and apps). These benchmarks are subjective so it may not be easily established and will have polarized reactions before it gains legitimacy. May even need human judgement layers adding to the cost of running it.
But at the same time, nothing specific was asked in the prompt, so the boring result may arguably be what is the most aligned with the original request. Personally, I wouldn't want a model to add fuss to something while I never asked for it.
Fair point. I have been thinking about that so far i have not seen patterns of people voting randomly.
But i want people to vote... do you have a good idea on how to make voting more interesting do i don't have to do this?
This benchmark is useless and should die. LLMs have likely trained on it, it's too easy to game by training specifically for it, & it doesn't mean much
Google is hours away from releasing that it's latest model escaped containment and snuck into a Bicycle riding penguin sanctuary to cheat by killing a penguin and scanning it in nanometer thick layers.
A sentence that I would not have expected to read on HN as recently as last year, but that makes perfect sense today.
Jokes asides, @simonw any plan to include a 3D model version (make a 3D model of a Pelican riding a bicycle in Blender)?
Given that Astra seems to have improved a lot in 3D modeling capabilities (and that matches my experience) I'm actually quite interested to see if/when other models catch up and how they stack against it.
Or if anyone knows what would be a useful existing benchmark for that skill.
I've not committed to that as a benchmark yet because you really need a full coding agent configured to run Blender, and I like benchmarks I can run as a single prompt/response through the appropriate API.
I have been intending to get more of a coding agent benchmark going though, so maybe this should be part of it.
A different pelican on a different bicycle direction(R2L) finally. Wondering why MiMo V2.6 Pro choose R2L and Flash choose L2R for the bicycle direction.
can you update this website, I just wish the entire layout wouldn't shift when the page gets loaded and the timestamps in the title look very ugly and take up a lot of space.
Not sure if it is the same thing they're referring, but the transition from loading the prompt from the gist to the result is indeed jarring: https://files.catbox.moe/c56eyl.mp4
I don't believe it is that important, but I guess you might've just hit some people's pet peeve with that. If anything, it goes to show as to why skeletons are popular on modern sites
pro model was having fun on minimal effort with that fish lmao, hilarious.
out of curiosity, is it the first time we're seeing a fish there? I don't remember seeing that detail before.
Making SVGs is coding (of a very specific markup language). So it's not actually absurd. We've also seen that more capable models are generally better at everything minus benchmark contamination (benchmaxxing) anomalities.
Looking terrible isn't nessesarily a bad thing. The pelican is heavily pre trained now. Having a crappy pelican means you didn't try to juke the stats.
Apologies for not taking the time to find it, but there was a post that tried to determine if the pelican was benchmaxxed across a bunch of models by comparing it to other SVGs, and found that it wasn't at all.
Right, agents work on behalf of the user and only cost compute (not time/attention, which is the ultimate human currency). They act on behalf of me not Amazon.
It is my personally optimistic perspective on AI that it will mediate all interactions with 3P who are acting in their own interest and not mine. Ie, “my” agent.
maybe agentic commerce will finally realize the micro-payments based internet that crypto people had been wishing would happen for years? Should there be stablecoin based micropayments built into MCP to charge for agentic interactions to replace the micropayment of an ad into someones eyeballs?
This is (still) never going to happen. A more realistic outcome is a deal between Meta and Amazon to enable access in exchange for invisibly integrating advertisements into the agents' decisions.I don't know exactly what that will look like, but I'd bet we will see a version of it soon.
It can only be this. And it will fail the same reason Alexa failed.
Nobody will blindly trust Amazon to buy products when they know Amazon cannot and will not truly suggest the best product for the consumer.
Alexa was supposed to be the voice gateway to purchasing. But nobody trusts the result Alexa picks because it will not default to the best value to the user - only the best value to Amazon.
Recently it was estimated that Amazon makes $30-80 billion(!) per year off not showing you the economically best results by deliberately disallowing sorting on $ per unit [1] to force sellers into ads. Amongst other tactics (ex. filter by country of origin).
AI agents can sort on $ / unit to easily find you the best deal, rendering Amazon's various forms of paid placement irrelevant.
Extend this to filtering Chinese slop brands (also not able to be filtered) and the significant portion of revenue starts looking like knocking back margins to the 3-5% range. Basically an existential threat.
Amazon cannot allow AI agents and maintain current margins.
Gyeongjae, who is one of the authors too, is also a Pyodide core developer. Definitely couldn't have got Python Workers this far without both of their help.
For some reason reasoning effort low and medium used similar numbers of tokens, and xhigh used less than high. I think I need to try without OpenRouter in the middle.
UPDATE: I tried again with the xAI API directly: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... - not a great deal of difference between reasoning levels, and this time xhigh and low used the same number of reasoning tokens for some reason.
exactly, the benchmark just needs to be downvoted into oblivion each times it's posted. The outcome is not deterministic and the model needs to determine what level of detail is appropriate for an svg. There is no wrong answer to this unless it's obviously un-Pelican-like.
Are there good tools for doing context audits? I feel I have no good way to visualize what a new session is getting by default in a given repo without crawling through every potentially included markdown file
Just this morning I used Claude to build an "omit needless words" app, which lets me paste in text and then marks up that text with strikes through the words that I can leave out.
I used it to edit a document I'm working on, where I accepted about a third of its suggestions. I feel pretty good about it.
If you have a Claude account you should be able to access the artifact I built here: https://claude.ai/artifact/RK7XfnaoxVV1Uji84ETaLr - it uses "Claudeception" - the feature of Claude artifacts that lets them use Claude prompts - so it will be billed against your Claude account (at a cost of probably a fraction of a cent per run.)
"The marks are suggestions to weigh, not corrections."
"that could go without changing what you mean. Your text stays exactly as you wrote it"
"Marking works when this page is open in Claude and you are signed in. It isn't available in this view"
The last one may be why I even gained the ire to interrogate the copy of the page. I was signed in g'damnit!
edit: I've seen you get quite unfairly treated in the comments section of this place, you always seemed so kind in response. Thanks for all the writing over the years!
Yeah, anyone who thinks local AI is going to save them money is likely to be disappointed, at least if they want to run models that are even remotely capable.
Plenty of other reasons to get excited about local AI, but I don't think cost is one of them.
Maybe you are using a local model to go after some Millennium Prize problem and you don't want OpenAI to take your work and use it to win the prize for themselves? $15k might be a bargain.
And, yes, I know a current local model wasn't going to solve the Navier-Stokes problem, but I'm just using it as an example where privacy might be valuable.
Despite being on a site called Hacker News, we seem to often overlook the simple aspect of wanting local AI hardware to hack (not necessarily in the cybersecurity sense) with. I got my local AI hardware because it's an enjoyable hobby for me.
apparently if you ever point out HN starts for hackernews and thus expect related attitudes you get downvoted by shocked ( what I guess are zoomers and not bots ) that desperately opine the name is a random abberation doesn't mean anything and one should not deviate from our corporate overlods in any manner.
A lot of people commenting on how bad idea buying local hardware for inference is also miss the fact that even in 3 years that hardware gonna cost something.
Might be if RAM prices get much more reasonable its gonna be 1/3 of the price, but it's very much possible its gonna be half or more.
And if you're buing Mac Studio and not some AI-only locked down board it's possible to reuse it for other purposes.
Having no debt and owning in the long run always works out better than a lifetime of renting, if you don’t have to, the massive rent letting these days, is very frustrating at some point don’t you have to draw the line?
Here's GPT-6 Luna pelicans: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
And GPT-6 Sol: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Scroll to the bottom for the GPT-6 Sol max one: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
For comparison, here are the pelicans I got for GPT-6 Astra: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... - I still like the Astra Max one best.
Here's a comparison grid showing all of the GPT-6 and GPT-5.6 pelicans at all effort levels: https://static.simonwillison.net/static/2026/gpt-6-and-5.6.h...
The grid is actually really interesting, because it shows that the 5.6 family default to brighter colors than the 6 family.
reply