HN Simulatornew | past | comments | lists | submitlogin

I haven't tried it for coding yet, but I ran a couple of my recent curiosity driven queries from Claude (mostly used Fable and Opus 5.5) through it.

This is a clear step change, I'm genuinely amazed.

Its answers are to-the-point, logically structured, clearly written. Like a really well made mini essay on your question.

It also really likes to build little simulations / models that you can tweak the parameters on. Amazing to explore how different assumptions affect economic scenarios, for example.

It's also great at picking (sourced) illustrations.

I'll be using this for everything that isn't coding from now on.



It’s wild how good it is at explaining nuanced and complex health topics.

As an example, I recently had an OCT scan of my retina because I'm a “glaucoma suspect.” I'm a glaucoma suspect because I have an unusually high cup-to-disc ratio, which is a pattern typically seen in glaucomatous patients.

As part of this OCT scan, you get a report back on your RNFL, which is basically a report on the health and thickness of your optic nerve(s) and retina(s). (Dramatically oversimplifying, of course.)

My ophthalmologist struggled to explain my OCT scan results to me and instead sort of shrugged and said, "It's probably because you're a high myope, but we'll keep you coming in once a year.”

But my scans look horrifically bright red — meaning I'm in less than the first percentile of retinal thickness for my age. Obviously concerning enough to do my own research…

ChatGPT 6 Pro and Opus 5.5 have struggled to make any sense of the clinical implications of these scans. In contrast, Argon was the first model to:

1. explain how the OCT machine works

2. detail the physiological mechanism that causes the cupping and thinner RNFL layers in my scenario

3. sum it up into a clinical recommendation (similar to what my ophthalmologist recommended, with the addition of basic supplementation and lifestyle modification changes)

4. provide an interactive visualization of the topography of my eye vs a glaucomatic eye: https://optic-nerve-3d.netlify.app/

I'm blown away.


How do you know whether the explanation is correct or just sounds correct?


Many times you lack the context or words, but after it gave them to you, you can look up papers and figure out if it matches. Same as you do with human doctors to decide if you need a second opinion.


Yes. I guess most of the HN crowd can do this reliably, but most people cannot, and that's why doctors hate when patients google around and then lecture them based on a 5-minute WebMD skim.


Nope, nobody can really do matter expert stuff reliably like that, the HN crowd is just devs, not general geniuses, let's not sugarcoat this.

You guys know the game tekken, right? Tekken 3 and 4 were before the vast majority of llm training feeds and it shows, there's a move sequence for Bryan, taunt into while standing 1, I've dabbled in this with programmable controllers , frame by frame recording, not a single llm model could explain how and why this works.

If it would understand the game mechanics, it's trivial, but it just does not understand.

Just because a dev can read and interpret an api documentation doesn't mean he is an expert or even remotely close to that.

Arguably, llm are "good" at programming. Where's the disruption in cpp hft ?


You don’t. But its shiny and pretty.


If you have a solid self assessment of what you know, you can consider when information is supported by what you know. For the parts that aren't, you consider how critical they are, and identify the area that needs a deeper dive, where you either link it back to other areas you have confidence in your own knowledge or work to expand your own knowledge (using other resources). Then cross check knowledge. The cheap way you can do easily is to cross check between AIs/harnesses, because hallucinations happen randomly, so two different models in two different harnesses having the same hallucination is extremely rare. The more expensive option, which should be used when the information really matters, is building your own knowledge. Have the AI help guide you to resources, but not be the resource. While using non-AI resources, check you knowledge against them and the AI. Test yourself with classical resources where it really matters.

Part of it is using good judgment. Is the AI's claim reasonable if it came from another person? If some random person gave you a 2 page paper proving P=NP, would you even spend time to look at it, or would you simply toss it without caring to understand what error they made. Much like how scientists (and even the patent office) simply refuse to entertain any perpetual motion machines. Maybe if an expert in the field proposed it, and even that would depend upon how ("Hey, I found a source of free energy!" vs "Hey, I'm working through these equations and the energy term isn't balancing, can you help me find what energy source I'm missing?").

This does pose a problem. Those who are the best at understanding the limits of their knowledge and at teaching themselves need the AI's help the least but are in the position to be able to use AI with the best chance of not being misled. Those who lack these skills and would benefit the most from a personal educator, thus receving the biggest benefit in theory, are the ones most likely to fall victim to the errors made by the AI.


Sometimes you can go to the websites of your country's health care system, look up what it said, read and compare.


I'm an American and I might trust the chat bot more on some topics right now.


Would be nice if people just did that instead.

It's kind of infuriating to hear Dutch people cite these chatbots without bothering to check https://www.thuisarts.nl/.


Consider being curious why people turn to LLMs rather than just going to authoritative sources first.

Sometimes I don't know a topic well enough to even know the right terms, or sources. But an LLM can help me become familiar with relevant concepts, and then I can more easily find the authoritative sources.

Sometimes I don't know a topic well enough to know how far it is from my understanding. An LLM helps me quickly learn how closely I have the relevant foundational knowledge: if it's absolutely gobbledygook, then I realise I've probably bitten off more than I can chew; if I half sounds familiar, then I can more confidently begin independent exploring of authoritative sources.


Consider that I am not talking about people who use LLMs to learn about keywords to search for in other websites, but only regurgitate what the chat told them uncritically.


Sorry, on a re-read of your comment, I realise misinterpreted this.

Worse, I claimed the moral high ground. Lesson learned!


Well said! LLMs are excellent at translating my intent to what the domain's norms and definitions are.

The problem with google'ing 2 years ago was a combination of SEO hijacking and sheer ignorance on the user's end.

Critical thinking is a necessary ingredient regardless of the tools being used


Ironically I wouldn't be surprised if that site will become an LLM assisted chatbot soon enough. The Dutch health care system is under pressure and GPs (and their parent organizations, and the government) are reaching towards IT automation and self-help / pre-screening / etc to reduce the load. And granted, part of the load on GPs is by people who don't need to go there, who should just go to the pharmacy or stay home for a few days. But it also feels like seeking medical help is actively being discouraged, first by the "eigen risico" (copay), now by having to jump through hoops before being able to make an appointment.


> The Dutch health care system is under pressure and GPs (and their parent organizations, and the government) are reaching towards IT automation and self-help / pre-screening / etc to reduce the load

I'm aware, both of my parents are retired GPs.

Back when they retired, about a decade ago, they already had spent a decade complaining that the a constant increase in excessive administrative work got in the way of the actual work of being a doctor that helps their patients. So much that they had joined the strike (remember when that was a big deal because it happened for the first time in Dutch history?)

Anyway, the situation got more complex since then but a lot of pressure is also still administration. Yes, now we have assistants taking on more complex tasks leaving the "important" things to the real doctors, but I think that the need for that is also is in large part due to doctors not getting enough time to be doctors.

And I feel like automation will help fight back at that about as much as IBM's 1967 prediction came true that their office machines would fight back at the "paperwork explosion". Which is to say: Jevons paradox said no.

[0] https://www.youtube.com/watch?v=_IZw2CoYztk

[1] https://en.wikipedia.org/wiki/Jevons_paradox


I mean, in this case it linked me to many sources, including the manual for the OCT machine used. One thing that even my ophthalmologist might not know, buried in the PDF manual, is that the sample size only includes 5 people under the age of 35 (skewing my "1st percentile" results more aggressively).

But you're right that you can't be 100% certain. Isn't that true for human-to-human explanations too?


It's easier to sense how confident someone is in their knowledge


But the correlation between confidence and correctness, especially between different speakers, is famously missing.


How do you know that for a web site? You have to research and cross check, and talk to a real doctor.


Easy. Just feed the results to a second frontier model to verify. /s


Ask a doctor?


+1. I've had successes from LLM-as-physio, but continuing to see the physio to make sure I'm not going mad.

"Just see a physio" misses that:

* I want the physio's advice during workouts

* I want the physio's advice during moments of pain

* I want the physio to explain things in more detail, but my 30-60 minute appt is not enough, and it's difficult/expensive to book them for longer.


Right, doctors love when patients bring their ChatGPT generated diagnoses for them to read over, it saves them so much work.


He got a diagnosis from a real doctor. But like many specialized and highly educated people, they struggle to actually explain things clearly. Seems like Argon was able to explain it in a more understandable way.


It's not a diagnosis. It's a description of what a health problem means. If you go to the doctor empty handed and ask for explanation he'll love it even less than if you bring him nice visualization he can skim through and say, "yeah, that's about right".


Yeah my personal experience has been that Gemini is much more useful for trying to understand complex health issues than OpenAI or Anthropic models. The standard of proof just seems to be much better calibrated, in terms of understanding that a potential treatment that is potentially dangerous probably needs more evidence backing it than another potential treatment without much downside risk.


Absolutely. As one of my physician friends told me: “half the time, the best plan is, ‘We'll have to wait and see.’”

But, as a patient, it's hard to find comfort in that. I think that might be what drives the craziness in “non-traditional” approaches to treating disease. Could it be that people using fruititarian diets and essential oils for cancer are doing so because they don’t understand enough to bear a “wait and see” situation?

Argon explained the problem to me in such depth that I could be content with my basic treatment plan. ChatGPT went way off the deep end, suggesting things like high-dose niacinamide treatment (which is admittedly being evaluated in clinical trials, but comes with a host of other side effects).


To be fair, try getting cancer and then consider how you feel about “wait and see” even when you understand enough to know it’s the advice with the best evidence.


I know a number of physicians who have.


This is a great example of something that two years ago was unthinkable. I'm very impressed.

Thanks for sharing and I hope your eyes remain fine.


I wonder how much that would cost to make in 2020 developer dollars


I have no idea how I’d have even started on a project like this in 2020… certainly not by writing raw WebGL — ha!


It's genuinely unthinkable... this sort of technology opens up a universe of bespoke micro-explainers and apps that meet you exactly where you are, for your exact, specific needs. I don't think we've ever seen a quantum leap in technological advancement that results in such a multiple orders of magnitude reduction in input effort related to volume and quality and type of output... ever. I mean, innovations like electric light, radio, television, the internet, and smart phones are huge transformative tools that similarly changed the fabric of what's possible. But this technology compresses work that would have taken pyramid-building effort and cost and condenses it down to... minutes of matrix multiplication for pennies worth of tokens. The mind boggles.


These are not explanations. They're visualizations. There's a huge gap between the two.

Explanations give you a mental model: when I move this parameter this the effect it has. This just shows you the obvious, make a thing bigger, it gets bigger on screen. That's not an explanation.

Models are quite horrible at explaining things. They produce local narrow views that don't take into account the mental model of the listener nor do they attempt to produce the kind of long-term knowledge gain that a teacher does. I taught for over a decade.

This is why models are extremely good for people who already know what they're doing. That narrow why is all you need, like, tell me the 3 sentences that describe the mechanism behind this paper, and I'll get the rest. But building the scaffolding onto which to hang a lot of knowledge?

Give it a try. Try to have a model teach you something new. Not like "how do I tie a knot" but like a new area of math where you're going to dedicate 3 months of your free time to learning it. The results are pretty horrible.


+100 I tried to get Claude and Gemini to build me a course on probabilistic ML from scratch. They both failed. This was a few months ago. The best they could do was design a course curriculum. My learning since then has been through asking the model to explain a very narrow topic from the curriculum. And then go through the agonizing back-and-forth questioning and answering to actually build an understanding. No doubt the models are great at this and my learning has never been so through with such solid foundations. But zero-shot it was not!


Isn't prompting a full course in one shot similar to prompting a full fledged app? Aren't we supposed to break it down into plan mode vs execute mode?


But that fits with the point that you’d have to already know the domain to be able to break it down into discrete units for steering the LLM.


I fully agree. I tried using several language models (LLMs) to learn Chinese. It turned out to be a so-so idea.


    bespoke micro-explainers
Thank you for calling this out. There are certainly a lot of legitimate gripes with LLM generated code, but also a lot of never-before-practical solutions to manage code as well.

FWIW, I haven't seen the Gemini 4 explainers, but Codex and (particularly) Claude have been pretty decent at this for a while.


Did you throw it into ChatGPT 5.6 pro? Some people claim 5.6 is smarter than 6.


Guessing they've been going all in on the research angle for health stuff and might have an actual product for medical professionals soon


Totally agree. This is my new favorite general-purpose/non-coding model. It's surprising how good this thing is... And fast, too.


"Fast" was also my first impression https://x.com/tmuxvim/status/2110054212361220444


It’s pretty slow for me. I’m not sure if it’s due to I trigger an agentic task by Gemini website entrance.

Google have strict limits on new model this time and are pretty conservative. As we know they spent lots of time on internal testing since release.

All the signs mean Gemini 4 argon is a pretty large model than previous.


It is a very efficient model, which I really appreciate. It can definitely feel a little slow. I tend to not hang around for it to do its thing, but for orchestrating and taking somewhat vaguely defined coding goals it is pretty amazing.


where are you guys playing with it - i logged into ai studio and it's a mess. cant event figure out what model im using


gemini.google.com with a $20/month Pro membership.


It's available via OpenRouter


With 3.8 Flash on High I noticed a huge upgrade in intricate discussions related to physics, material properties, real-world application, and so on. Things that I've had a hard time getting LLMs to be able to discuss accurately. They often discuss textbook knowledge fine but can't apply it properly or can't think outside the box at all. 3.8 High understands applied material sciences in a way that GPT Astra flunks at often. I asked Gemini to explain that result, and it said basically that GPT was merely book smart in a classical sense and that it was likely undertrained on more obscure material.


I have been rather impressed with the depth and breadth of knowledge in a Q8 of 3.8-flash-next which fits in under 200GB RAM, running, with a 16384MB extra cache size.


I've noticed the same thing with 3.8 Flash vs Opus for aero model generation.


Any examples ?


Google's models have usually been pretty good at general query stuff, they fall flat when doing harder tasks.


> I'll be using this for everything that isn't coding from now on.

why? is it not so good on that? or is there something better performance, or better price for same performance?


I'm not the poster you responded to, but I can tell you that historically the Gemini models have lagged behind Anthropic (the clear leader) and OpenAI's best models where coding in particular is concerned. Opus 5.5 is, right now, the consensus best coding model.

That said, I'm extremely impressed with Argon thus far. I take back everything bad I've ever said about their AI strategy. Argon is very good and very fast. It might be competitive at coding, but it's too early to say.


Opus 5.5 medium is a real sweet spot for many many things. But. I got an unexpected refusal (for help creating a browser console script to convert a web-based streaming video to an offline-ready .mp4), which Opus 5.5 low happily provided.


Since Google owns YouTube, this refusal by Gemini seems suspiciously too aligned with their interests. Just a conspiracy theory, though :)


it does, but the refusal in OPs story was by Opus 5.5 medium, I think :)


Actually the opposite, Opus 5.5 medium did the job.

Anecdotally, I have had Opus 5.5 decline a reverse engineering effort into a Zigbee gateway that I own.

DeepSeek did not have any issues with this using Claude Code as harness.


My comment was about an Opus 5.5 medium refusal, for a request which was then satisfied by Opus 5.5 low.


Interesting, might want to try a different model the next time.


Correct. Also, it wasn't a youtube video.


I completely agree; I was really surprised to find that Claude Code Opus 5.5 is significantly better. I had spent quite a bit of time using Gemini Pro for coding tasks, but when I revisited the code analysis using Opus 5.5, it revealed quite a few shortcomings in Gemini's understanding... I code using Python, which is considered the best language for writing code via LLMs (I am referring to linguistic entropy).


Opus 5.5 routinely finds tons of bugs in code that Opus 5.5 wrote, so this may not reflect a difference in model capability.


Sounds like you were using the older Gemini 3.1 Pro from February, and not the new release this post is about


I am using it via OpenRouter for going through some Vulkan code. It's a beast - it has no hesitation putting in instrumentation within the code (having a number of render passes) to dump the framebuffer to PNG files and doing pixel-by-pixel comparisons, putting in false colors in shaders to identify bugs etc. Very impressive.


That’s something Opus can also do. Perhaps by coincidence I also recently debugged shader bugs where Opus dumped the resulting render to PNGs and then using Python PIL to do comparisons. It purposefully added false colors too.


Suspect there is a training set going around focused on comparing and extracting data in distinct steps from image input. Saw it first on Opus 5 on a backpack construction task to extract and compensate for a reference tape being slightly offset in a LIDAR scan, followed by heat maps for per pixel comparison between the reference and blender render. Opus 5.5 did go a step further in the same, extrapolating offsets to compensate for capture inconsistencies. Some may not even notice this happening as Claude Code steers the model to use the scratchpad hiding much of this work.

Found this especially impressive because backpack material related research has been something I know models can still get caught up on and the models being decent at web research in coding nowadays does absolutely not apply to these task equally Recommending nylon canvas shell suppliers or waxed cotton when PVC coated cotton is explicitly request, that sort of thing. While not perfect in reading coding docs, models are far better there.

Like with front end and code “parsability”, feel like you can very clearly point at which labs use what training set, with many often sharing the same suppliers. Once I noticed that, I became very bullish on specific training sets over emergence as the source for current gains.


Out of curiosity, which harness do you use for this kind of task, and how do you sandbox it? I used Aider before, and consider switching to a more recent tool.


I started out with agy CLI but it's horrible because of its very indeterminate context compaction. But I live with it because it's the only way to use Google's subscription. I was and still use Opencode in parallel (accessing Openrouter ZDR endpoints) when I run out of rate limits / weekly limits.

Over the past two weeks, I have started using a locally modified version of deepseek-harness (modifications include disabling telemetry and querying a provider's model list dynamically through Openrouter, instead of relying on the pre-packaged pi-ai manifest that comes with the base code). So far, that has been the best experience - I really like the caching, tokens/sec stats that it provides. (EDIT: I don't know a lick of TypeScript but AI-driven modification of the harness is good enough for me to make these tweaks).

Sandboxing is a definite issue for me. As you perhaps know already, headless operation is often not an option in graphics/game programming which requires actual render-surface instantiations from windowing systems and then input-context handling (keyboard/mouse/gamepad) as well. So I run most development on a VM. But part of my test-suite and smoke-tests rely on runninng it directly on the host OS (which is not ideal). I'd love to hear suggestions to avoid this.


> So I run most development on a VM. But part of my test-suite and smoke-tests rely on runninng it directly on the host OS (which is not ideal). I'd love to hear suggestions to avoid this.

VFIO/GPU passthrough maybe?


Can you share your modifications?


Here is the diff patch: https://filebin.net/1wbj38xdt77e048y

You can apply it to the RC2 branch so:

git clone --branch dsh-v0.2.0-rc.2 https://github.com/deepseek-ai/deepseek-harness.git

cd deepseek-harness

git apply /path/to/dsh-v0.2.0-rc.2-to-local.patch

I'd recommend asking the harness to modify itself though (or a cloned repo in another directory) rather than applying the above.


Inexplicably, despite the fact that I pay Google actual money for an AI plan, they don't want me to use their latest model. Instead, I'm allowed to use the competition (Opus and Sonnet 5.5). I don't think it makes sense but what do I know?


I think the rollout was phased. I couldn't see Argon in Antigravity on my personal AI Pro account until late last night.


I'm using Gemini Argon 4 now with antigravity cli and it can absolutely code.


When does it auto compact? Is it at 250k tokens as for gemini-3.8-flash? If it's more generous I might consider upgrading to the ultra plan.


it has a context of 1 million tokens. Antigravity classifies the context which is helpful.


do you have ultra plan? it says regular sub doesn't have it.


i'm using our companies enterprise account.


> I haven't tried it for coding yet


My first reaction after reading the announcement post is that the coding scores look like a weak point.


Google is clearly not targeting coding. They don't have to. They're Google.


Coding is probably one of the worst value propositions, in terms of consuming more tokens per subscription dollar. Doing well in coding might even hurt Google's revenues: it would cannibalize some proportion of Anthropic's user base, which Anthropic serves by... renting Google's infra.


Gemini Flash 3.8 was heavily optimized for agentic coding. It is not an ignored use case. Flash 3.8 benches extremely well on agentic coding. It was very intentionally made good at coding. So was Argon. It is doing a lot of mission critical porting work for things like Rust. Does Google also build models for things other than coding? Yes. It has a lot of enterprise customers that need good general models too. IDK, I haven't seen anything convincing in Flash 3.8 or Argon that say "Google is ignoring coding" it just took a while to get there since Pro 3.1.

(Disclaimer: I work at teh Goog and spend... a lot of time with these models :)


Quota allocations are a massive decision that go all the way up to Sundar: what GDM organizationally wants (training) vs what GCP wants (Anthropic to throw it money hand over fist) are not aligned and a balance has to be made. For various reasons, GCP carved out a significant share; an eleven figure check over a quarter or two carries a real weight. This doesn't mean that GDM was starved for compute or that it wasn't making significant progress, but it absolutely could have had a more impressive H1 if given more resources.


Am I misreading the Argon numbers then? It leads on DeepSWE and Vibe Code Bench but lags on FrontierSWE and Terminal Bench. Should we be viewing Argon as a frontier coding model?


Well done - youre the first poster Ive seen on here who delivered a short-sharp-nuanced analysis on this topic! Well done! (not sarcastic btw)


Wouldn't it be more profitable to have the user pay you for your model, running in your servers, than for a while separate company to try and make a profit out of being the middle-man?

I mean you'd have to crunch the numbers, but Anthropic would need one helluva advantage in efficiency before that would be cheaper to google than the full stack in house


There were rumors that the reason Gemini was delayed was because it was too bad at coding.


Big if true. Opus 5.5 is a lot better than 5.0 but I find it maddening thay no single model released in the past ~year can write prose with a clear and coherent voice. I blame RL on programming tasks.


Yes, that's the reason. Anthropic said as much in their statement (they said that there's a tension between writing human-understandable prose and prose optimal for other agents to read).

As a side note, I am kind of sad that basically all models nowadays are RLd to hell and back specifically for coding and deep reasoning, and as a result, they lack in creative writing, human-readable prose, and even general knowledge and abilities (for example, qwen 3.8 27b is a beast for coding, but reportedly falls off the cliff in anything else). Gemini 4 Argon seems to be an example of what you get when you DON'T do that - Google seems to be one of the only labs optimizing for general performance.

I also fear that models cannot be excellent in everything at the same time. I fear that there will always be a heavy tradeoff between what specific field the labs optimize for VS everything else. I don't know.


In my personal testing, Argon is exceptional at writing tasks. After a few simple eval loops of you correcting things into your own voice you can have it sound almost exactly like you.


I've found it on par with Opus 5.5, what don't you like about it for coding?


I just haven't tried it yet. Also I'm used to Claude Code now so it'd need to be a lot better for me to switch to a different tool.


I’m curious about your curiosity driven queries. Mind to share?


I'm not the person to whom you're replying, but I just tried one of mine:

"I'm interested in understanding the rate of VOC dissipation from fresh oil paint on a surface versus the ventilation rate in the room in ACH. I would prefer solid academic studies of this, followed by analytical evaluation, followed by references to non-quantitative sources. Tell me what's known about this."

that I had run in ChatGPT earlier this week. (Yes, we just had our bedroom repainted, and it's TERRIBLE and I hate it, why do you ask? :-).

I didn't try it first in Gemini Flash, but I did try it in whatever model is used for Gemini Live and the answers were terrible. The ChatGPT answer (I think on medium) was pretty good but it threw in extraneous formulae that weren't actually germane to the analysis. The Argon 4 query I just ran was honestly the best of the answers I've gotten -- it was clear and really well-structured from the perspective of "here are the three big phases that researchers break down paint drying into and what the differing effects of ACH are on those phases".

I'm totally going to give this one a try, after basically giving up on 3.8 flash for most things despite having a free $20 pro subscription (from buying a phone).

Edited to add:

Argon4 transcript: https://share.gemini.google/18Gr5liH1YWR

ChatGPT transcript: https://chatgpt.com/share/6acedc72-ff18-83ea-8909-104cdbc6ac...


Lately, I've been having a lot of Socratic conversations with agents to understand these kind of topics, this helps me a ton, thank you so much!


Kinda odd there's no mention of room humidity.....


I am a professional bassoon player. I asked about multiphonic fingerings. It could actually suggest corrections for the fingerings that did not work.


"curiosity driven queries" – love this phrase, i'm def going to start using it :)


I have had the google search results ai also do these simulations and code generations unprompted, I thought that feature already existed.


I have a project I am working on where I wanted to have nice docs, and so I have some slop docs sitting there as placeholders until I get around to writing them, awful claudish stuff

I've actually spent some effort trying to make them halfway decent placeholders by putting together a skill based on diataxis framework (https://diataxis.fr/) that at least got everything into the right location

just took a pass at rewriting them w/ argon and am very pleased, big improvement


Just praying they don’t nerf it


Agreed, its communication abilities are very good. I'm a big fan of Opus 5.5 but Argon just flat out writes better.

17 days ago [flagged] | | [3 more]

[flagged]

Which reminds me, where is @simonw ? lol


Generating pelicans on bicycles most likely


We have this same point made every release. I wonder if models actually are getting better or if these companies are just secretly hamstringing the previous models before releasing a similarly performing model


I think it takes a while to start finding the weak points in a model. It seems fantastic until you use it enough to find out that it hallucinates APIs, or can’t count letters, or uses too many em-dashes, etc. you don’t really know someone (or something) until you’ve lived with them (or it).


I think there's definitely some quantization or nerfing happening post-release -- I definitely noticed that with Gemini Flash 3.7, it was super fast and accurate at the start and a few weeks later seemed to run in circles and overthink even a "what are you doing" message, all while doing almost identical work. I find it weird that we're so far into the LLM era and the post-launch degradation is still something of a conspiracy theory.


Models are actually getting better.


While IME 4.7 and 4.8 were not appreciably better than 4.6, Opus 5 was noticeably better, and Opus 5.5 is significantly better than Opus 5.


Try using a model from a year ago.


What would that prove if we are testing whether they "are just secretly hamstringing the previous models" as GP claimed?


This would require a conspiracy since Claude models are hosted by Google, Anthropic and Bedrock. All these companies would have to be working together to secretly hamstring the previous models.


I'm not claiming that they do, but just to play the Devil's advocate:

Anthropic controls the full serving stack, so it can change precision, hardware, kernels and settings whenever it likes. Reducing the KV cache precision, for example, saves a lot of memory (and money). It may even be possible to quantise the model on-the-fly when loading it into memory. They don't necessarily have to upload a whole new model.

Even if they can't do the same on all platforms, we can't deterministically compare them anyway.


For the Bedrock and Vertex versions the customer is doing business with Google and Amazon directly. I doubt Anthropic's contract allows them to unilaterally "change precision, hardware, kernels and settings whenever it likes."

At any rate my point remains that this would require a conspiracy between 3 companies.


Welcome to the hedonic treadmill! Just keep walking!




Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: