The 10-day window is mainly a tradeoff between sensitivity and detection speed. Shorter windows give faster results but are much noisier; longer windows give more statistical power but could take weeks to flag a change.
Also, it isn’t comparing one 10-day period once and calling it done. The window rolls forward daily, and a change has to clear the pre-registered 99% threshold in two consecutive windows before it’s flagged.
Ten days isn’t sacred, though. Once there’s enough longitudinal data, one of the things I want to evaluate is whether that window length is actually well calibrated or should be changed in a future version.
When you simply large amounts of information, the knowledge probably shifts to upper level reasoning, then complex reasoning develops on top of it again. But who knows.
The problem is the precedent this creates. For non-famous people using public APIs like this it could mean AI companies sucking up the information and throwing millions in compute at it.
The sequence for Navier-Stokes was that these researcher spent a year working on it, then they published a possible breakthrough, OpenAI then spends $15M within a couple days to finish it.
What Google means to say is that MV3 will allow them to more closely control what you’re able to do with your time, since this was done primarily to target ad blocking software.
Not that this matters much anymore as Google is now more an AI than it is a search engine, and an unreliable one at that.
What’s even worse is intentionally degrading performance for sign-in’s. This is still circumstantial but I’ve been looking into that over the past few weeks.
When switching between google accounts on a random app or game, no issue. Quick code, you’re in. When I switch between accounts on Claude on mobile, I have to sign in, enter the code, screen freezes, I sign in again, can’t verify it’s me, I sign again, wait 30 seconds, I’m in.
I have to sign in three separate times when signing into a single google account in Claude. For every single other app it’s very quick.
I have a sneaking suspicion that Anthropic and google are working together to degrade sign-in performance for multi-account users. I can’t fully prove it yet but I plan to be able to in the week or two.
It is possible that Anthropic (and OpenAI for that matter) actually put out some pretty low quality software.
If you’ve used Claude Code for any length of time you’re familiar with all of the strange rendering bugs, freezes, etc. OpenAI is even worse, their horrific software makes it difficult to _pay_ them, which should be top priority for a company.
It's weird to me how so many people just put up with crappy, low quality software. If you buy a physical good and its defective, you return it, stop buying that brand, maybe even leave a negative review or contact consumer reports, etc. If it does not live up to what was advertised, you go to the FTC.
But when it comes to software, we all just kind of accept that shitty software is the norm and totally fine? Let's start calling it what it is, its defective.
I don’t think we should allow posting links here that require you the purchase a membership to continue reading. Or at least redirect with an ad block or something through a custom site. That would be rather hacker news of us.
I might’ve missed it, but why was Fable 5 tested on high while Opus 5 was tested on max? Seems like quite a few of them aren’t on the same effort setting as well. Although effort doesn’t really matter anymore since they can change it dynamically, seems like that might be viewed as an experimental error to some.
The header mentions something about the best run so I assume they picked it. But this really reads like they write this section by section with AI (admittedly with a prompt that stops the most obvious tells - though there are a bunch of semicolons which is what I tend to see also when I say no em dashes). The style is different each section - the results section has the random irrelevant description ("this section does x) that the slightly dumber models do a lot, and lots of invented terms (in the form "the x" where it's a name some model came up with at some point where it just assumes we know what it means for some reason) and assumptions about us knowing stuff we'd have no reason to know ("re-ablate the stack - wtf does that mean).
And like this section screams opus 5 gobbledygook to me
>Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind. They preserve weak signals long enough to validate them, but they also have a better understanding of the results. These are not separate capabilities, they combine both research taste and good noise modeling to climb the speedrun.
WTF doe any of that mean. What winning ideas. What experiment leaving what behind. What's a weak signal what are they preserving how do you know that they aren't. Also if youve ever looked at a Claude code transcript the harness is constantly re injecting random reminders to keep models on track, the models are writing (imo trash) memories to reference - did they test that the _model_ has those capabilities or model + harness?
> We see similar patterns across the traces: models develop their own experiment drivers, simulators, and analysis tools as they go.
What traces? Kimi traces? Other models in prime? Other models not in prime? Building their own research tools is like a normal thing models do now. Is this just like "they built their own test suite" or why the focus on prime? Does ipython somehow magically work better than bash for this?
> "We were again surprised by the lack of novelty. The models clearly understand the objects they manipulate at a deep level, and yet very few genuinely new ideas emerge, which makes it hard to tell if this is an artifact of the speedrun setup or a real capability limit."
I know it sonly one word but God is that "Real" such a Claude real lmao. I use it too - after I've been using Claude code too much. I guess the genuinely too. Also wtf does any of that mean and how does that square with
> "A good research decision is sometimes not to spend another GPU run. Several models built small simulations or tests to isolate a mechanism before going back to training with a sharper hypothesis. This wasn't systematic, but when it happened it often led to a better understanding of the object they were manipulating"
Did they all have strong understandings of "the objects" they were manipulating or were the strength of their understanding of "the objects" (different objects?) the distinguish factor here?
Also how does any of this square with
> Models also have different knowledge cutoffs which limits access to certain papers. This was a deliberate choice. We tried a few runs with a CLI tool for searching papers but found that restricting internet access including arxiv made models slightly more creative.
So this is a pre existing thing? Why TF would you expect novelty when their nanogot is benching below the state of the art still? They're gonna start with replicating existing work before they get to anywhere you'd expect something novel
Anyway - all that to say - if fable orchestrated this, its genuinely believable that some real insights were obtained (is a good model) but the honest caveat is that it's not the research quality, it's the communication. Your pushback is valid and these models have a way of writing tons of words that you can read and still not understand wtf they actually did or what anything means. Maybe it means something to them in latent space
I misread the title and thought it would be about the (for lack of a better term) NanoGPT speedrun[1]. Which, previous to the article, was meant to be the world speed records for Andrej Karpathy's GPT-2 (small) reproduction.
I think that large companies having next to no consequences is in large part due to capitalism doing what it does over a long period of time. There’s a deeper and deeper consolidation of money and power the longer time goes on it seems like.
If we think back to the various lawsuits Facebook has gone through, they paid out about $10 or so per individual affected, totaling a few hundred million dollars, which they would make in a couple months for selling user data and whatnot.
This is something that every company gets away with mainly I think because of just how large their wealth actually is. It’s difficult to actually punish a machine that acts almost like infrastructure. Punishing an individual is easy.
I don’t know if there’s really a solution at this point, maybe we could’ve prevented this reality at some point in the past but I don’t think that without actual global collapse it would be something that can be retroactively changed, and I don’t know if global collapse would necessarily lead to a better future.
I think that for one, Zuckerberg should be in prison, if someone oversees a massive theft like this, I think they should be held criminally liable. Same the CEOs of Anthropic and OpenAI for their parts in the massive theft that took place. They should all be doing prison time.
The reason I don’t think they will is that their investors probably have a good amount of leverage over anyone who would prosecute them, so it would never make it that far.
> It’s difficult to actually punish a machine that acts almost like infrastructure.
It's not at all. There is zero reason this couldn't be applied to Zuck. [0] There's also no reason why fines couldn't be 10% of global revenue, or more.
I agree that they should, my point is that it’s difficult for several real logistical reasons. These companies for one do a large amount of lobbying and fundraising for political campaigns, they do control most digital infrastructure, with AI expanding are securing multi trillion dollar datacenter funds, etc.
There’s a ton of money at stake for the weathly aristocrats. They push politicians to delay or do things in favor of the companies above the people, and like I already said punishment is hard specifically because how do you fine someone with infinite money? They will just get more.
That’s not a punishment. I think that personal liability to the actual CEOs is the only way to get around this, leadership is generally never held responsible which means they can do whatever they want and the company bails out their greedy decisions.
It’s late stage capitalism and there’s no clear answer at least from my vantage point. Maybe if plug it into Claude it will give us a more coherent answer.
> and like I already said punishment is hard specifically because how do you fine someone with infinite money?
This isn't true though, if they had infinite money they wouldn't be spending every single second of their waking existence trying to get more of it and move higher on the leaderboard. If your opinion is that Meta/Zuck wouldn't care about being fined a year's worth of global revenue, or in his personal case, 90% of his net worth, then I'd really like to hear why. Because it contradicts all of their actual actions showing that all they care about is hoarding as much money as possible, meaning they'd very much care about it being taken away. Maybe I'm missing something.
I mean, total societal collapse would absolutely suck, and there's indeed no guarantee that whatever will replace it won't fall into similar traps eventually, but if things actually are as you describe them, it only can get worse and worse indefinitely, until we get to global collapse anyway.
The thing about treasure is that you often have to find it for yourself, it can be hard to find, butt when you do find it you get some legendary booty.
The thickness of some caves in terms of genuine girth is truly something to marvel at.
Also, it isn’t comparing one 10-day period once and calling it done. The window rolls forward daily, and a change has to clear the pre-registered 99% threshold in two consecutive windows before it’s flagged.
Ten days isn’t sacred, though. Once there’s enough longitudinal data, one of the things I want to evaluate is whether that window length is actually well calibrated or should be changed in a future version.