I ran it through my own coding eval today using pi/OpenRouter (11 small bug fixes from my own projects). It solved 10 of 11. Compared to Sonnet 5.5 and GPT-6.1 Sol, a median run cost me $0.52 vs $0.18 for Sonnet and $0.09 for Sol, and took ~4 minutes vs 32 seconds and ~1.5 minutes.
It thinks a ton, ~24k output tokens a run vs ~3k for Sonnet. It missed a solution by thinking its way into a 10 min timeoout.
Something to look out for - if you're using pi, it doesn't have Argon in its catalog yet and reports cost at Kimi's rates, about a third of what OpenRouter actually bills.
I've noticed that Gemini Argon/Flash 3.8 tends to read all the related and unrelated files into the context compared to Claude Opus 5.5 which starts the editing as soon as the information is adequate. I'm not sure if this could be fixed by the system prompt or AGENTS.md
There is no model passing all the tests. Gemini is on top because questions also include general knowledge, trivia or domain-specific questions, which Gemini models are way above any other model.
Even if models have similar scores at the top, it's useful to compare costs/response times.
Everything in the top 25 is above a 9.4 out of 10, so it seems a bit saturated?
Also seeing that Claude Opus 5.5 is at #23 makes me think this is not the benchmark for me
It's actually quite hard to make a single task/question with few steps max that SOTA models fail on. Gemini are on top, because asking domain specific questions still breaks most top models, but not Gemini ones.
Claude is lower, and also most Antrophic models, because (historically) they have one of the highest refusal rates and also struggled more with precisely following instructions (i.e. outputting in a specific format/way).
You can assume that all models above a score of 9 are "good", then it's more a matter of preference for their response times, costs, personality and strength in specific areas.
Opus 5.5 are best at coding, and even art generation, but struggle a lot with built-in knowledge, when web search is not available.
Damn way to undermine yourself in your own blog post Google:
"The end result is a memory-safe video decoder that runs 2.7x faster than the Rust port, with identical video output, bringing it closer to the optimized C++."
there was already an optimised c++ library and a rust port that was safer but less performant; they managed to get a new rust version that recovered a lot of the performance gap. sounds pretty damn good to me!
I'm just surprised that the marketing blog post about the omnipotent new AI model (that no one outside Google can currently access - contrast with the Opus 5.5 / Astra launches) - doesn't pick examples where every metric is better than before.
tbh while the astra announcement in that link had some good stuff it was scattered through so much boilerplate marketing speak that I had to force myself to read it and look for the content. I found the gemini blog post in the OP a lot more readable and engaging.
but that's a side issue; my main point is that you are underrating the impressiveness of getting a safe rust port of a highly optimised c++ library even nearly up to par with the original. the tradeoffs rust makes for memory safety cost it some of the raw speed of c++ even with all the zero cost abstractions and purely compile time guarantees they have. (tangentially i wonder if ats (https://www.cs.bu.edu/~hwxi/atslangweb/) would be a good candidate for LLM assisted ports; it seems way more advanced than rust and might actually get c-level performance with safety, but it's really hard to write.)
bounds checks are one, yeah, but I was also thinking about the tricks c++ could play with optimising undefined behaviour, as well as unsafe patterns involving shared access and pointer aliasing that a human could determine was safe in that specific case but that the rust compiler would balk at. and maybe it wasn't actually safe in which case the rust code would have the last laugh.
AFAIK, Rust benefits from the same optimizations that Clang can. In some areas Rust can actually benefit more from better aliasing information. But you do have to use the keyword unsafe for some things that you can Just Do in C++. You're right it can be difficult to achieve hand-optimized C++ levels of performance with Safe Rust.
not interesting as the writeup as very AI sloppish :(. i'm sad about that as I would really like to have an intuitive sense for how good the current frontier models are at coming up with interesting 3d printable parts.
I've been just getting started on this with Astra. Astra is one-shotting good 3D models from photographs for me. I was unable to get Sol to do this, even after multiple rounds.
It's not a small change, it's from "not useful" to "pretty good"
Right but there is no meaningful specification failure mode in creative CAD, is there? I mean, yes, image to CAD for a little miniature, I can buy that it is OK. Fun, even, especially when your use is sort of transitory.
But this article is talking about functional parts. A functional part exists because of its spec, usually fails because of some combination of its material, manufacture, and spec. And the spec is chosen with that in mind.
Even working from a text description of a part is error prone because language is imprecise and interpretable. I just don't understand why AI people want to race past actual proper sensible solutions to a problem just to automate generating from text.
People don't have time (or even mental capability / even more time) to learn CAD but they have a task at hand which they want done. If AI can generate a part which is "good enough" for the user at a cost that the user considers reasonable (I.e. zero) then AI gets used. If a part fails thats rarely a concern because these parts are just 3d printed and likely cost single digit cents in filament. User can always print a new one.
Right, but the thing that I think sort of militates against this scenario is this:
If a non-CAD person imagines a part that they need, the chances are very high indeed that either it isn't what they need, can't be made robustly or sensibly, or if it is, it already exists in some meaningful way that can be adapted, because one of tens of millions of other people already wanted it.
This scenario — needing something very specific that can be described only in words and nobody else has ever needed before — is almost imaginary. Most stuff just isn't like that.
Now… if someone were to design an AI CAD package that could remix objects that exist and combine them, perhaps with a GUI tool that allowed a user to identify attachment points, maybe that would be interesting. But a lot of that has been solved in non-AI GUIs already.
Text-to-CAD just isn't as good an idea as it seems. You are not going to be able to magic up some novel invention from a text description without a significant amount of domain knowledge of the words needed to describe a part, and that domain knowledge usually comes from learning CAD!
(Bit like getting a coding agent to write really good code)
And simple parts, sorry, just don't need much CAD skill. Get a CAD package on a free plan, learn enough to make your thing. You will get so much more out of that and it will be easier to make changes. It'll probably have standard tools for adding threads, even.
Even image-to-CAD for a functional part requires a significant amount of understanding of what to photograph and from which angles, etc.
Dude, coat hanger joint is broken. The free chatgpt gives openscat file I can just paste and convert to stl. I didn't even measure the diameter of the pipe, apparently those are standard. In 10 minutes I have a new 3d printed part to fix my coat hanger. No CAD UI app subscriptions, not reading CAD books or whatever. That's real life. Rest is in your mind.
For OpenSCAD, AI models from about GPT4.5 onwards have been able to build and combine shape primitives pretty well, but that was never immensely useful.
Just last week I had Gemini 3.8 Flash and Antigravity take an existing STL of a part and swap out a piece of it with another design (fan shrouds on a computer case front plate). In the process it created exploded parts diagrams from the STL, created 2D renders from each axis (for its own visual re-ingestion, presumably, since I neither asked for or needed them), projected measurements, and then it implemented my design swap request. Which it didn't do with SCAD, it wrote a Python program and directly manipulated the STL geometries. It went perfectly. A week before that I had GLM 5.3 and Hermes take one model that was intended to fit into a part from another designer's part and change it; I didn't really like the interior part's design so I gave both STL files to Hermes and asked for a new interior part design that could be printed with vase mode to speed it up. It did and the new part is better than the human-designed part I grabbed off the internet.
I had another task mid week where I needed a case for something but didn't have the dimensions, so I just had Gemini do deep research lookup for dimensions and to then spit out a case STL for me. It was boring but functional and a good enough starting point for me to work with.
Right now my wife is enamored with a 3D printed purse design for halloween but the separately printed hinges are a weak design (a circular mortise and tenon rotating hinge). Working with Antigravity again, it's been able to take measurements, rotate parts in space on its own and create me a new object design that's better suited to be printed in TPU. It's held dimensional accuracy and even inferred some dimensions and spacing I didn't have out of the primary purse's shape. It's gone great. Next up will be having it combine all the parts into a single STL so I can do a multimaterial print in one shot, printing the purse in PETG but the living hinge in TPU without assembly needed.
I think "it's here, now" for home and hobbyist CAD needs for functional prints. It's not displacing a mechanical engineer's job yet, but this recent iteration is markedly better than the previous generation of models at spatial reasoning and 3D design because it's suddenly capable of it at all when even 4 months ago it was not.
In your other comment you mentioned "just learn CAD", and I have already, I could have cobbled together all of this myself in Fusion 360, but that would have taken far more time than a few minutes of prompting and walking away. I know enough domain lingo to describe the task well, and if it falls flat I will do it myself, but it's coming along faster than you seem willing to accept.
> I know enough domain lingo to describe the task well, and if it falls flat I will do it myself, but it's coming along faster than you seem willing to accept.
Could you do meaningful things without domain language you learned the long way?
I don't doubt it is coming along. But as with all of these things, this one is painted into a corner from the very start by using OpenSCAD.
And a mechanical engineer's job isn't just fiddling away in CAD, is it? If it was I would be on the way to being one by now, and I am not.
What it means is that a good CAD model is not just sticking stuff together in 3D space. Parametric flow means that a small number of measurements ripple through the design, and sketches and solids adapt. To do this right, it's easier to just understand the purpose and intent of the model, how it was manufactured, and draw from there.
One of the things you get from studying CAD diagrams — like the Solidworks exercises everyone works through — is that real world parts often have surprisingly few dimensioned constraints. A lot more emerges from geometry.
So you can look at a diagram and initially think "where the heck does it define this radius!?" and find that it never needs to, because that radius emerges from tangency, parallelism, perpendicularity etc., combined with fewer measurement constraints.
Specifying it as a measurement would therefore be overconstraint; it might lead to solids that are difficult/impossible to manufacture or erroneous when another measurement is changed.
The diagram will specify the actual driving measurements that come from the part's purpose, with the intent to adapt to different real-world requirements.
Any CAD tool that works from images is going to have to approximate, which will create over-complex models with to many measurements that are more fragile with respect to changes, whereas a design that has geometrical constraints in place that are logical will be simpler and more adaptable.
With functional parts, roughly, form follows function first, and manufacturability second. But you have to understand why, to be able to make a good part. Even (perhaps especially) with 3D printing. But not uniquely. For example, would this tool be able to understand why the draft angles on a plastic part were important, what determines the bend radius of a part that was made from pressed metal, etc.?
I must stress that I am an amateur. But my amateur skills have come from reading and learning about CAD and learning what makes a viable design.
(Sorry… this was a stupidly long answer but I am hungry and I figured I should try to blurt it out before I have to get up from my desk and cook. I've edited it a lot and cut out a bit that I will probably put in another comment later)
Astra does that. I've fed it STLs, told it to inspect, make a model for it, derive 'conceptual' measurements (e.g. - there were USB type A sized slots in one design, it recognized those, and knew not to change that, but only let it be swapped out for type C slots), and make parameters for everything that is relevant (that's pretty much the level of detail I gave it). It distinguishes 'implementation detail' from 'conceptual measurement' and implements the model in terms of those 'conceptual measurements'. As always, with OpenSCAD (well CSG in general) the problem is bevels/chamfers/fillets, so it needed some instructions to look for those and not mistake geometry stemming from them for 'real' measurements. It does that too, now.
And yes, telling it 'add draft angles to make this castable' works.
Then you are not actually getting a model with a constraint system at all. Which is the point of the thing I'm replying to.
CAD constraints are more powerful and flexible than OpenSCAD, and the power comes from analysis of what the part is actually for, not merely what it looks like.
You can't even compel OpenSCAD to keep a line and an arc tangent. If hull() can't do it, you're making it up yourself with maths.
Wait.. Al this started with you saying Ai can't design cad that is useful. Does that mean you consider openscad not capable of doing anything that is "useful"?
It's a slight overstatement, maybe. But in the general case the answer to your question is yes. I believe OpenSCAD is absolutely not the right tool for functional parts. It has fundamental weaknesses.
You can use it for basic shapes and very simple geometry. But you can use Tinkercad for that too.
Should you use it for CAD generally: I think not. Because complexity balloons out of control. An AI is going to struggle as much as a human does with a complex part in OpenSCAD.
All the text-to-CAD packages are relying on it because there's lots of simple examples in the training set, and because it is declarative — there's a belief that there is a mapping from text to declarative spec and from declarative spec to an object.
But it is the wrong choice for flexible designs that adapt correctly when measurements are changed.
It's clear to me that "AI CAD" developers who think OpenSCAD is adequate to the task have a fundamental misapprehension of the capabilities of the technologies they are actually seeking to replace. OpenSCAD is not representative of the nature or capabilities of modern CAD.
I think that if you are trying to make an adaptable functional part you would be better to start from CAD drawings, with proper constraints on them, and that an AI CAD package should be interacting with those (there are some attempts to do just that).
It's just not that difficult in a modern CAD package, it's not arcane or obscure.
But if you're going to be using code-CAD, you want a programming package that can represent fundamental aspects properly. OpenSCAD cannot.
CadQuery, to be fair to the authors of this post, does now have an experimental 2D constraint solver:
And it has a proper bRep kernel, so it can represent vertexes, edges, arcs and faces in mathematically abstracted (not mesh approximated) forms, and do operations on those (including fillets, though it has its weaknesses there inherited from OpenCascade).
I think an AI that could operate properly on CadQuery models and implement appropriate sketch constraints with an awareness of their implications, choose the right operations, make use of libraries, etc., would be a lot more capable. But it's so much further down the line than people think based on these OpenSCAD toys that produce trivial parts.
There was a blog post from a couple of days ago that had been initially flagged, it seemed to me to have a good overview of AI for CAD, will try to find the link.
What word list are you using as the valid words list? The old solutions list from nytimes wordle is no longer very relevant, e.g. the last month or so has produced several Wordle solutions that were not on it.
It thinks a ton, ~24k output tokens a run vs ~3k for Sonnet. It missed a solution by thinking its way into a 10 min timeoout.
Something to look out for - if you're using pi, it doesn't have Argon in its catalog yet and reports cost at Kimi's rates, about a third of what OpenRouter actually bills.
reply