It would be nice to know how they select. I max out my $200 subscription every week and I was selected. On Saturday, the Opus on my account that happened to be supervising a running live process, became unusable.
It would be trivial for them to opt-out accounts that are doing benchmarking from a planned regression.
I really really really do not trust companies on the verge of IPOing from goosing their numbers. Even if the lawyers are up their ass, the social pressures are more than enough to compel the behavior. For every good person willing to do the right thing even at the cost of a delayed career progression, there are 20 people willing to do anything not even for a dollar but for an "atta boy!" from the boss.
That's a major reason to use local models. Or open weights models on a trusted cloud provider. Even if they are way less capable, they're more predictable
In the context of public relations they would not need to ID every benchmark only the most credible or most public benchmarks many of which are openly available on github and whose problem-set could be fingerprinted.
You're absolutely right that it is a game of whack-a-mole and eventually we may have leading private + credible + popular benchmarks that ultimately dissuade the behavior, if it can be dissuaded.
Look up eval awareness. There are all sorts of subtle signals an LLM could use to know if something is shaped like a benchmark or not. But that's not a bad thing, every animal behaves differently in play vs real-life.
Yes the model can do that, but it would not be "trivial" (as GGP had said) for Anthropic to opt out accounts performing private benchmarks from this theoretical model downgrade people think is happening.
The most plausible approach would be for Anthropic to train some separate classifiers to detect benchmarks. I'm not saying it's completely impossible but IMO this is conspiracy theory levels of effort.
One can also monitor for it in nftables. This is from my raw table:
grep CVE /etc/nftables.nft
# Options CVE
iifname "eth0" ip daddr $ADDR4 meta l4proto tcp ip hdrlength gt 5 counter log prefix "CVE-2026-98432: " level debug
Once confident nothing legit would be blocked by dropping the packets one can change "log prefix "CVE-2026-98432: " level debug" to "drop" if on an unpatched kernel.
$ADDR4 in my case in my internet facing interface.
There have been a lot of write-ups about the future of AI. I see two distinct futures:
1. You get ID'd and biometrically scanned to be able to use powerful AI, and after you get de-platformed by an opaque censorship system, you are a digital bum. Same life as homeless now: they cannot even open a bank account because they don't have an address.
2. Local models win, perhaps with the assistance of Apple, and people run suffiently powerful models on the client computers.
3. LLMs are fully commoditized, your country funds training of SoTA models, your town runs them in the municipal data center; everything tax-funded. You can use them for a modest fee, as easily as taking a train. When interacting with those LLMs, your privacy is guaranteed by law.
How many municipalities offer the services of ISPs? I know it’s not zero, but decades of private ISPs leading the market leads me to believe municipal data centers won’t be made available to residents.
How often would municipal data centers need to upgrade/replace hardware? What impact would that have on local tax levy rates?
Would you choose to relocate your family to the next town over because they have more available RAM?
You might be surprised how many places have public broadband. I think they are usually the result of poor private offers.
Compared to sewers, drinking water, electricity and roads it seems a rather simple and cheap thing to build.
My commercial isp offers all kinds of packages clearly designed to extract as much money as possible. I cant even see the prices as everything is hidden behind x months discount. From what i remember 200 mbit is 42 and 2000 is 60 but a phone line and tv become available only if you take the faster package and required with the fastest speeds.
So its like one banana for $1 and 10 bananas for $1.50 If you buy just one you cant buy milk, if you buy 10 you also have to buy milk.
Its also been sold 6 times. Unsurprisingly, each time the service and quality noticably declines and the prices go up.
I see their adds everywhere and they sponsor all kinds of things. It suggests the product is overpriced and doesnt sell it self.
Yes, people did. Well at least I did. We knew we can use QCD to compute loops and to some degree masses in the lattice. But there were serious doubts the underlying mathematical theory was sound.
For many it didn't even matter. It would just mean that at those energies the theory is not even valid.
I am not certain on the extend of this proof but QED is probably not sound on it's own and we have been using that even longer. lambda phi^4 behind symmetry breaking and the higgs is strongly believed to be unsound on its own.
Yes, but it has also been known for 50+ years that QED has a Landau pole and might be UV trivial or inconsistent, whereas QCD has asymptotic freedom and is thus controlled at UV.
More to the point, QCD has been experimentally tested both at UV (deep inelastic scattering) and at IR, while the physical meaning of testing QED at UV Landau pole regime is unclear.
It's a good first step. They're not great but they kind of work for many tasks. I think the small models can be useful in context-specific tasks but the friction is high right now because the user is forced to determine whether Siri is good enough, or a better LLM is required per task. Ideally Siri would be smart enough to intelligently route. Apple claims that's how it works but even their cloud processing is far behind frontier models.
Worryingly, they're taking a walled garden approach, and they don't want to expose user data to competitors. After the monumental failure of Siri over more than a decade, I don't trust Apple to execute competently on AI.
reply