Honestly, the bigger question is, are they at some point start adding numbers (Carbon 2, Neon 1.5, Hydrogen+ ionized), or are they keep giving a new name to every single new model. Argon seems to have been basically "replaced" by Carbon.
Why does this chart not include the other gemini 4 model, Argon? Also I am sure somebody thought about which element to name which model and it would be very helpful to me if they would explain their reasoning. OpenAI's naming got kind of abstract with 5.6 but I could follow it: Luna(Moon) was smallest, Terra (Earth) was medium, Sol (Sun) was biggest. Astra kind of breaks from that but whatever. But like I would genuinely appreciation someone from google to explain the reasoning behind their element names. Like are Argon and Neon related in some way because they are both noble gases? Is hydrogen the smallest model because it is the lightest element? Carbon has a lower atomic number than neon but is more expensive and presumably bigger... maybe I am overthinking this.
Inconsistently it seems. On the $20/mo tier, Argon has been removed from the web interface and app, but is still available in Antigravity (while Carbon is only on the $100/mo tier in both).
They're much improved on fundamentals, but they've adopted the typical color and tone problems of a GPT diffusion model image even though they're svgs. Interesting.
I feel like people use medium models thinking they are saving money, but in the end, it ends up wasting more time and tokens theough the increased rate of issues/bugs.
If the highest tier model makes even one or two fewer mistakes that require human intervantion, that almost makes it a no-brainer choice for production.
I've been doing some research lately on my own (aside from the major benchmarking going on) and I've definitely found cases where medium models do much poorly and cost more than the higher models. But on the other hand, if I can give the medium models certain skills or other resources to complete a task then they perform better in cost than the higher model (which, given the same resources, sometimes gets more confused than the medium does).
At least for my use case of evaluations of a set of features.
You’re talking about something else (reasoning levels). The parent comment is referring to e.g. GPT Terra as opposed to Luna, Sol or Astra, Claude Sonnet as opposed to Haiku, Opus or Fable, or Gemini Flash/Hydrogen as opposed to Flash-Lite, Argon or Carbon.
Oh, good catch. I do use Sol in production, for cases where I need quite good results, but can't afford to run in production the highest tier model due to pricing (so it's like the highest tier model I can afford for that use-case).
To be more specific, this is for a support automation, where the model has to be quite accurate in following rules and generate responses, and Luna models or lower-tier models often make bad mistakes and generate wrong responses, so are not really usable for sending data directly to the user.
Even better at the local LLM stage, where QFN can catch a mistake in prior work and patch it without any subscription costs at all. then I can use a cloud model for polish and finish and move on.
I'm building stuff at home with my work focused on managing infra. If anyone wants to sponsor a SOTA model subscription I will check it out. Until then I'll see how flexible a Strix Halo setup and my homelab can be.
I also started with Qwen-3.8-27b and used for a while, I still use it so Claude can delegate tasks to it, but the SOTA models are still a lot better than local models, so much that I don't see much sense to use local models as long as I have a paid subscription.
Even if the difference is a single extra error caught early, that can end up saving me hours of work later.
Asking people to guess beforehand how much intelligence they require for a task will hopefully be a relic soon. We’re lacking useful routing techniques which accurately pick the right model for the job. Or a high intelligence model which completes simple tasks fast and cheap. Or something else.
International Olympiad in Informatics (IOI) Gold Medalist here. I too did competitive programming for years. I pride myself on writing beautiful code that is succinct, effective, and efficient.
Yet I find myself using AI coding agents more and more. In this day and age, your choice is to either use them, or be stomped into the ground by programmers who do.
AI still can't write code at my level of quality, but it can do so at adequate quality now; and sadly, that is "good enough" for the real world.
I've made many friends and acquaintances over the years who are top notch, some world class, programmers.
I'd say that 90% of them are very excited about using LLMs for coding. They have nothing to prove to anyone and can easily steer AI to reach "godly" productivity (please note these guys were 10x programmers already).
I also know some weak programmers and most of them tend to be critical of AI and want to get rid of it.
Lastly, a couple of people I know are against AI for moral/ideological reasons without denying it can bring productivity increases.
This is purely anecdotal but I'm starting to notice this pattern everywhere I look.
All I want is an explanation of how you're all reliably getting good enough (or better) results from LLMs. Surely us AI critics are doing something wrong, something much more specific than needing to "prompt harder", if we accept the framing that 90% of world class programmings are achieving that, while the rest of us are not. There simply has to be some specific, repeatable, and yes, teachable thing we're missing. So what is it?
Try requesting a feature from Opus 5.5 high that would take you a day or two to implement and test well. It'll work for 20-30 minutes or so. Then ask it to review the changes with a sub agent of itself and with Codex Sol 6.1 high. Then let it address the findings and iterate the review/fix cycle until the reviewers are happy.
Most senior developers that steered juniors or interns in the past, are able to work with LLM's.
Sure, LLM's are unreliable. So are juniors.
Sure, LLM's code quality is not always great. Well, so is the juniors code.
The benefit of LLM's is that they can work super fast and have a vast amount of knowledge. The benefit of juniors is that they actually learn from their mistakes and get better over time. LLM's stay at intern/junior level, even with the correct context.
So for me, I see each LLM session as a new intern coming in. You hand them the right context, give them the correct instruction, and off they go. Then they come back, and you will have to judge how detailed you want to review what they produced. Or in other words, can you get away with some bad code? And for the next session, it's a new intern that starts from 0.
But it is also highly dependent on the project you are working on. If it's a big complex codebase, LLM's will struggle, and they can only do simple tasks (however, they can help with testing a lot!)
So the teachable thing is unfortunately experience:
- Know where quality matters, and where it doesn't
- Know what kind of mistakes could be made and should be checked for
- Know which things can be duck-taped together, because they live at the edge, and which things need a proper foundation, because the rest builts on it.
So in the end, I think it all comes down to making the right trade-offs and the right decisions (which LLM's are terrible at, by the way). And unfortunately that is gained by the slow process of experience.
I think it's very tough for junior developers right now, because they either go fast by letting the LLM's do the work, but then they learn nothing. Or they go slow by asking the LLM's what's going on, and then do learn.
A senior developer never understood the full code-base, and never reviewed every line of code. That's just impossible. And now every developer needs to be a senior developer that is able to steer these new "interns".
And just like with juniors and interns, you have to put a lot of effort into it, and it's not always clear in the end if you gained or lost time :D.
I don't claim LLM's and juniors are the same. I claim that working with LLM's and juniors are similar. If you are able to work with juniors, you are probably able to work with LLM's.
To comment on the post you linked:
I agree that people can be thought. But on the other hand, I also had juniors that reached their limit pretty fast, which can also be frustrating. The best way to "teach" LLM's is the context, but I agree it has limits and is not really teaching anything, just steering them in the right direction every time (like a checklist).
Your comment basically states that running LLM's is a gamble, but according to my experience, most of the time it delivers what I ask. Maybe in a different way that how I would do it, but I want results, and am not interested in micro-managing.
Are you working on a project that requires high quality code? Or is it your preference to work in such a codebase? For me, I was always ok with "good enough" code. And sometimes "good enough" needs to be good, and other times you can get away with some really bad code.
I feel like code quality is important when humans have to reason about. However if the future is machines managing the code then all that matters is AI understanding it.
The only danger is losing grasp of internal components of the system, but maybe there'll be a "management" layer eventually for humans.
I don't know what the future of programming will look like but I don't see us rolling back the clock to 2021..
Who cares about the theory? The fact is that the world is now worse due to LLM use.
And it's not just restricted to software. Everything I care about is getting worse.
Dribbling about how it could theoretically be better doesn't help undo the damage. "Imagine how much fun we could have playing baseball", says the man with the baseball bat, as he kneecaps you.
A phrase I've been toying with is eyes free vs hands free coding.
The distinction is important. While I too hardly write code these days, I do continue to make smaller changes and look at what's being done. This is as opposed to the group who has decided vibing mega commits is a good idea.
I am not doing small changes though, I add large features or refactors daily, which involve changing hundreds of files and thousands of lines, even if I wanted I couldn't review everything by hand.
I have done this too, for side projects, but I see the limits. Discovering what happens when we do this was one of my intentions and why I hold my opinions.
For these projects I make regular time to look through the code and do cleanup phases.
So you're saying simultaneously, the code is "higher quality than ever", but you also haven't even looked at the code to see its quality. Classic. I think you're mistaking quantity for quality.
AI is a hammer, problems are the nails. At the end of the day, you still need someone to swing the hammer. Aside from the occasional overstretch of scope and “reining it in”, the real bottleneck is humans.
We have more freedom than ever to build whatever we dream of, you still need a dreamer. And that’s why the profession of coding isn’t going away, you still need people to turn ideas into reality. I find AI soulless, it’s great at mimicking but is incapable of coming up with an original idea that wasn’t yours regurgitated.
When am I (as a user of a lot of software) going to start seeing any results of this unprecedented productivity gain?
This thread is full of 100x supercharged programmers, so why is it that I can't name a single useful feature added in the last 5 years in e.g. macOS, Windows, Firefox, Chrome, YouTube, VS Code, YouTube, Slack, Photoshop, Word or any of the other 5 programs currently on my taskbar?
The productivity gain is invisible using your metric.
What I have seen is that, with LLMs, an entire class of process brokenness disappeared in the in-house codebase. Before, there were many trivial tickets that, in the end, required a 5-line bugfix sitting in the queue for months. Now, such tickets are resolved instantly, and the backlog is shrinking.
Because the code is and almost never was the problem. Product design and business involves a lot more than code.
Personally, I did see a lot of entry-level products having a lot higher quality and better UX. At the same time, most companies go for fast iterations and are profit-focused, do product quality is not really a target for many.
I insist on reviewing every line of code. But in the last few weeks I realized that LLMs are simply better than me at reviewing code too and I'm just finding excuses to not do the real work, which is to write more prompts.
I have been asking LLMs to review/edit prompts since GPT4o era, so I really don't know what the point of this snarky comment is.
Since GPT4o, I have been convinced that most of the digital work would be done by AI and humans' job is to "move the chairs around to not block AI's way." It's a given to me. I'm just debating whether reading the code makes me a better chair-remover.
Don’t think it was snarky. My guess is op is just suggesting that you could automate further, but you are ratcheting toward it one step at a time while op is suggesting to allow the ai more autonomy.
Here is something I found this week: copies the entire db to memory, inserts a single row into in-memory db, deletes everything from real db, writes everything back. Sentence that caused this is "do XXX in one transaction".
All tests pass, everything seemed to work. Would not have noticed of I didn't read through the code. Amount of other stupid shit I found is crazy.
These anecdotes mean little without knowing context, model, harness, prompt. The technology has moved a lot in the past year, and does require some adjustment.
My very first experience with OpenCode led to a `rm -rf /` within minutes, which I caught by chance watching the output stream in. Yet 99% of my code is currently written through an agent.
A performance benchmark would have easily caught this though, ask the LLM to create and run benchmarks and see how those change after each new feature/major change.
Also to make it clear, I am not saying the code itself is better, but the end-product and what the user sees and gets. Like that saying "if it's stupid, but it works, then it's not stupid"
Because the exact same processes that create it faster, are also now generating large comprehensive test suites, and data-validation suites based around the actual business requirements and data/edge cases.
In essence, it is writing the code faster (great), but then also writing the code that was often previously never written due to budget/time limitations (comprehensive testing/validation).
So, how do you know the code is the highest quality ever? Because we have test result data that PROVES it to be the case. It isn't "trust but verify" it is "Don't trust. Verify."
This is my observation as well. I thought I was good at testing (I wanted to be good at testing), but the models are just better at it. Testing in particular rewards a tireless cataloging of the tiny and the tedious and in this respect LLMs are unmatched.
Single data point, my Sentry notifications across more than a half dozen client projects when from 2-4 a month to basically 0. That's while shipping 20x _more_ code.
Yeah but you don't test for stuff you don't know you have to test for. All tests can pass and internally its still done in a braindead way that will explode at some point.
Absolutely, and this is exactly why humans are weaker than LLMs. We consider what is likely, they consider what the data/logic shows is possible. So you get edge cases tests/flagged, that wouldn't even be something a real person would know to look for.
You must have replied to the wrong comment, since the one you did replied to never claimed that the test and validation suites were "never looked at." They are reviewed, and the test/data results are reviewed.
> Because we have test result data that PROVES it to be the case.
pgrust thought this too, but then another human discovered claude had hardcoded benchmark query plans.
Now pgrust has a "test" [1] to make sure the Ai doesn't do this again (unless it decides this test is bad for the current goal)
I have a growing list of these, another notable one is OpenCode having a security regression [2] because the Ai redesigned configuration loading for v2.
Real verification seems beyond transformer capabilities.
Very little code written in the world day to day ever has to justify its 'quality' or long term value. A lot it is just one off tools, bits and bobs to make a teams life easier or whatever.
I used to work in Web agency for sizeable brands and not a single site I built lasted over a year because the companies changed concept regularly. The next agency would come in and tell them it was built wrong and couldn't be reused etc etc. We did the same, it was irrelevant how it was built and we didn't even look. This is the nature of most code. The problem is that very few people will ever be in a position to be writing verifiable important code, but every developer likes to cosplay that they are.
Most code is ultimately an, annoyance businesses need to meet a goal that will always be a cost. It's often worth treating it as such and writing things that are easy to spin up and teardown.
The people most vexed by AI dev appear to be the ones who think writing code is a pinnacle role, it's not even though over our lives the salaries would sometime suggest otherwise. That's ultimately going to be a short term blip in the landscape and most code will be one seen as the cost it is.
A programmer's output was never code. It's was just what we had to do to get computers to work for us. Now I just ask a genie to do what I did and check the results.
> I haven't touched a line of code in over 6 months, didn't even look at it, but the work I do is faster and higher quality than ever.
Well that doesn't make sense if you are writing code that is going to be used on critical infrastructure.
Imagine a disaster were to happen and you said to the auditors that "I don't know what I am doing nor did I look at the code" and you cost the company tens of millions of dollars.
Sure if your building toys, that's fine, otherwise, why take the risk in being liable for not looking at what the code does and then believing it is high quality?
Indeed, I am just building apps people use, they are not "critical infrastructure" and the world won't end if there's a small bug, I don't vibe code self driving cars, defense systems or medical devices. A
If there is a bug, it will be fixed quickly anyway, so only a few users might ever see it, or see it only for a brief time.
Even when manually coding, projects still have issues and bugs, I don't think much changes in that aspect now, if anything, there are fewer bugs now in most high quality apps than before, due to AI catching more issues. See all the vulnerabilities found in old core code and libraries nowadays, that humans didn't find.
LLMs write better code than humans with fewer bugs. They also do it much much faster so you can spend all the remaining time testing. Tests are written faster too so not only you have more time for testing but also you can use it better.
>>Sure if your building toys, that's fine, otherwise, why take the risk in being liable for not looking at what the code does and then believing it is high quality?
>>Don't you see the contraction you just did there?
I wouldn't trust anyone who refuse to use LLMs for programming. Their code is worse, they test it less and it takes longer to write. For me "no LLMs" attitude is a sign of incompetence. Good for hobbyist trying to keep their brain active but that's about it.
That is even more so for critical infrastructure. Just look at the mess human coding is when it comes to security. LLMs are fixing it at the pace we couldn't even imagine a year or so ago.
> LLMs write better code than humans with fewer bugs. They also do it much much faster so you can spend all the remaining time testing.
So does that mean you should never look at its code?
If a fault in the software occurs even after testing and there was an investigation, will you admit in the postmortem report that you did not look at the code at all and blame the agent for the error?
> I wouldn't trust anyone who refuse to use LLMs for programming.
Did I say I refuse to use LLMs for programming? I am telling you to check the agent's work.
I would not trust anyone who DOES NOT check the changes of an agent's code when using them for programming in highly regulated industries or in critical systems. For most companies in the Fortune 500, it is irresponsible to not check.
> That is even more so for critical infrastructure. Just look at the mess human coding is when it comes to security. LLMs are fixing it at the pace we couldn't even imagine a year or so ago.
No-one argued against that and it does not mean that you do not check with the agent at all even if the agent believes that they "fixed it".
Gemini 4 Argin vs 6.2 Sol vs 6.2 Astra vs Fable 5.2: https://aibenchy.com/compare/google-gemini-4-neon-xhigh/open...
reply