One thing to consider is that even when some model has big gap between those two benchmark it doesn't mean it is much worse. It could be worse becasue he has assuptions based on training but on real scenario it wouldn't have it so it still could achive great results

This test still says something about how well the model adapts to subverted expectations. Some models get more stuck in local regions of their latent spaces than others.

But I agree witih you an important piece of the generalisation puzzle is missing.

I see I am the only one that is actually surprised that LLM are so good on this versus tool that is specialized to do one thing

I don't think new benchmarks are saturated. They still give you a clue, they arn't perect but they have value. If model can't even do some easy tasks from benchmark then why would u even consider using it?

Cost are bigger than Opus 5.5 for that effort

bro agent scripts doesn't matter at all imo

It is a 9B agentic model developed by Xiaomi MiMo through supervised fine-tuning of Qwen3.5-9B on MiMo-generated data

Seems like benchmaxing? For example for Terminal-Bench 4 it doesn't have great results. And why not show other benchmarks?

Probably, FrontierCode is made by Cognition itself. The model also seems worse in every way than DeepSeek v4.1 Flash, launched today.

Also the submitter's account is very new which makes me suspicious of self-promotion.

I don't get it. For me it seems Opus was more accurate in terms of for example this small building in the right down corner

I would say opus was in some ways more accurate, but missed the higher level curvature feel of the site that astra picked up on.

It sounds completely trivial and likely I'm wrong here, but could it be that opus saw the reference image squished? That might explain the sharper horizontal curvature

Opus 5 is quite good, and has been the SOTA for this test (by my own subjective comparison) for the last few months. One thing it's always struggled with though is recreating smooth SVG curves/cutouts.

Another way to think about it is you could prompt Astra to put the building back in. You couldn't prompt Opus 5 to get the correct curvature of the line / cutout. That part has always been a huge struggle for models.

IMO it depends on the company. If everyone in the company has to provide a lot of PRS etc. you have to stop care about the quality and just get the paycheck. The CEO decided not you. (Of course not when the product may charm ppl etc.)