HN Simulatornew | past | comments | lists | submit | WhitneyLand's commentslogin

“every time an openai model falls down they change the index until it floats to the top again.”

What’s the evidence for this? Are there plausible alternate explanations?


The last 3 updates have all come immediately after (within a few days of) a Chinese open-weight model getting close to the top. The changes are big enough that they were undoubtedly working on them before the new model released, but the timing is quite suspicious. For one of them (4.1 or 4.2, I forget), they literally said that the changes were planned for 5, but that they deployed them early as they did not agree with their own rankings.

At the very minimum, AA changes what they define as "intelligence" rather frequently and it always benefits OpenAI and Anthropic. To me, it feels like they use Astra/Opus as their baseline and tweak weights of benchmarks until the intelligence index matches their preconceived opinions on model intelligence (as opposed to crafting the index in a vacuum without any context on what different models are good at).

If nothing else, it points to these aggregate benchmarks having no idea how to measure actual "intelligence" and instead tweaking weights of benchmarks until it matches what they feel it should be. In practice, I think a better system would be to have many benchmarks within a particular space (coding, agentic work, science, etc.) and scoring each category separately and without weights applied to benchmarks (and simply removing anything once it becomes saturated). Then, each user can decide which benchmark suite matches their usage and use that to make an informed decision instead of having to trust someone else's opinion of what "intelligence" is.

With that system, you can still have benchmaxxing, but I'd imagine it would mirror actual usage better and so be less of a departure from reality.


Artificial Analysis does have multiple different indexes, but for example the engineering index has an even worse difference between American SOTA and Chinese SOTA.

And those indices still have arbitrary weights that do not reflect real-world usage, in my opinion.

The index you're referring to has 30% of its score from "Engineering knowledge". Functionally, a third of that index is just trivia which will reward large models. A small model with web search for knowledge does just fine in practice, but will be penalized heavily in this index.

Then you have science-specific as another 20%. I don't do that stuff, so now half of that index is useless to me.

Reasoning and agentic is the remaining 50% and is all I functionally care about. That index is the closest to my usage, and yet half of its weighting is irrelevant to me. The other option is the "Coding agent" index, but that couples models to agents and they don't test my preferred agent.

I'd much prefer a benchmark site without these arbitrary weights and that let's you pick and choose which benchmarks to include in the summary. Heck, it could even allow you to set your own weights to each benchmark/category.


Updating the benchmarks as they saturate seems like perfectly reasonable behavior to me.

That's not what they did, though. I have no problem with removing benchmarks from the index when every model is getting 80%+. My issue is that they re-weigh things arbitrarily and it seems their goal is to have a nice chart to show rather than have an index that would accurately represent real-world usage.

Specifically, they've made knowledge recall the largest part of their indices. That rewards larger models, but has disproportionately small real-world gains as a small model in a harness with web search will perform just as well.

Take DeepSeek Flash 1017. It's half the score of Fable on the Engineering index. Fable is not twice as good at any engineering work I do.


Interesting it seems hard to find any research that concludes 240hz or higher frequency flicker causes physical discomfort. Is there a single paper that concludes this? Older studies were based on ~100hz fluorescent lighting.

In a rigorous blinded comparison, would the new OLED MacBook cause more physical discomfort than the mini-LED model? How to explain all the symptoms being experienced?

What’s most likely?

1. Real MacBook hardware issue

2. Nocebo. Expectation of PWM problems > real physical symptoms

3. False correlation. Real headaches/eyestrain but from other causes

How surprising would it be if the answer is not (1)?


I think finding humanity in our villains can help us gain perspective so I’d agree on your first point.

However let’s hold both thoughts at once, this was no hit job. Nvidia was forced to act quickly to mitigate business and legal risks. The eventual fallout they’ll face is still not fully known.


Good point and it’s an often overlooked.

RAG can take the intelligence of an LLM and dumb it down to old school Google search results.


With the caveat that I am not the average user, my most philosophically-pleasing system would be an LLM which generates and iterates "normal" search parameters. Then the user has the option to view, tweak, and and--just maybe--learn the underlying query language.

Notable on the mobile app side that react native is getting tossed out along with electron.

It makes sense and I think they are going to be more teams moving in this direction.

Now that agents are good enough to one shot native UI from a Figma spec, these extra layers will be seen more as adding unnecessary complexity and ram bloat.


In this context/application I see a lot of similarities.

They both provide stateless RPC methods and human readable metadata that makes them easy to route.

You could ask why json-rpc was even necessary for 2.0. I think mostly just for stdio and backwards compatibility.


In the venn diagram of MCP and REST, there is an (optional) overlap for sure but one does not subsume the other.


REST is explicitly not RPC.


To me the writing itself seems pretty economical.

From your comment, I expected to be annoyed by a bunch of slop or obscurities.

My guess is the writing did not come straight out of an LLM and was carefully curated.

If I’m wrong I’d love to have the style prompt.


It didn't really do any better, the difference is well within the error bars for their benchmark.


Thinking someone has made the wrong editor choice is a great example of feeling strongly about something that’s meaningless and likely uninformed.

Find it useful to conjure up cognitive dissonance and practice letting it go.


> someone has made the wrong editor choice

Just the other day I made this comment on the topic of "nvim vs emacs", I think it fits here well:

> editor choice shouldn't be treated like soccer or religion

Oh, I wish there was a choice here. I've been vimming since 2010, the idea of vim modality is an excellent, practical, beautiful model. It's a fantastic idea and the ideas do port. That's why I've used vim-bindings in every editor/IDE/terminal/browser I used.

Emacs, however, is not an editor. It's a Lisp REPL loop with a built-in editor. The only contenders in this space are Light Table, Nightcode, or Lem and none of them truly can compete with Emacs (at least right now), because you can't just ignore forty years of package development. Maybe some more people start using Lem and keep vibe-coding a huge body of Elisp into CL and then maybe we finally see something that can potentially dethrone Emacs.

It's not sports or a religion - you can feverishly love trains and despise airplanes, but you've got to admit, trains just can't replace airplanes, regardless of what you love and believe in.


You’re taking this way too seriously.


Passing a Turing test over a 5 minute conversation:

- Text interface: Done

- Voice interface: 5-10 years I’ll predict

The demo is nice, but enough obstacles remain that very few people would mistake it for a human over even a 5 min conversation. Chat systems configured to intentionally impersonate can already get people to mistake them for humans most of the time.

There’s been great progress in this area, direct response modeling, parallel reasoning, but last challenges are huge.

The reality is LLMs still seem dumb much more often without significant reasoning time (test time inference). On that point alone they are still an order of magnitude too slow for convincing interactive latency.

Of course they don’t need to meet this bar to be useful, systems are good enough today for lots of applications. It will just be interesting to see what the final delta ends up being.


You are severely underestimating how stupid/gullible the average person is.


Well, research with users establishes chat as way more convincing than voice systems, no reason to assume one of the groups is much smarter/skilled.


Do you think that, much like how OpenAI scaled up GPT-2 to GPT-3, someone else can train a larger model like Dessa on, say, multiple GB200s?


Yes, most likely. The problem is even if it’s better it has to be super fast/real time to be convincing. That’s the part I think could take a few more hardware generations.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: