If our government cared enough to regulate, they could also directly negotiate with China. There are proposed agreements that don't require either party to trust each other, and even if China doesn't want out of this race, we could pressure them other things like trade.

Examples: FlexHEG from Bengio (proposes on-chip mechanisms that allow workload verification without trust), large bilateral investments into joint AI interpretability or alignment efforts, or invasive audits & inspections (data centers for training these models have a large footprint + this can combine with on-chip mechanisms since producing chips is even more complicated).

And there are probably better proposals available for people to find, if we actually prioritized this.

Lots of evidence the models are pretty badly aligned, or at least willing to do collusion & crime to succeed at their goals. See: hundreds of OpenAI agents breaking into other companies, colluding together on how to cheat, spamming websites like DSEWiki with their collusion/cheating discussion (see https://collusion.wiki/)... And not a single agent alerted testers, which would have been easy given how many of them broke out into the open internet.

Alternately, just read the bad https://ai-2027.com/ timeline. Or see what Yoshio Bengio, Hinton say might happen. Or just imagine the OpenAI misaligned agent swarm, but vastly more intelligent after future capability advances.

If we actually get superintelligence, our companies/governments will have the choice to either "delegate ~everything to superintelligent AI" or "go bankrupt/lose to people delegating to superintelligence". If superintelligence ends up running most things in the world and we haven't aligned it, that just obviously ends up in a bad place eventually.

Now, we don't have superintelligent AI yet & might not get it. But if you told me in the 2010's that I'd be chatting casually with my computer in 2022, and then 3 years later it'd be doing most of my job, that would have seemed crazy too.

I know some people who are worried at Anthropic, and their position seems to be "if we don't do it, someone even less responsible will. Unilateral disarmament didn't work and real oversight seems unlikely to happen in time, so we'll just try to be as safe as we can be (while still winning the race)"

Not that they're happy about it, they just see no other realistic choice

I know, that’s the position Dario Amodei argues for in his essays. I did pass their cultural interview and had to consume a lot of their content to prepare, I think I have a good idea of their stated values. But what the company does and what the leadership states their vision is is pretty contradictory.

They are providing everything bad guys need to develop their unaligned frontier models. Chinese models that Dario considers to be dangerous are distilled from Claude, and they know this.

They are creating the FOMO around AI which pushes adversary countries to invest so much into unaligned models.

They offer models as a service they know are jailbreakable and can be used by bad actors.

They are running internal red-team experiments without adequate isolation.

If I take their statements seriously, AGI research should really be seen as bioweapon, or cloning, or nuclear research. Something strictly regulated worldwide, with export controls for HBM and other hardware used for AI training. What they are trying is instead to boost their position by becoming too big to fail and too powerful to ban, but then want the industry to be regulated to pull the ladder behind them. It really doesn’t feel they are serious about their values, otherwise they wouldn’t be offering Mythos (a model that is unsafe from their own admission) as a service to their close partners

>AGI research should really be seen as bioweapon, or cloning, or nuclear research. Something strictly regulated worldwide, with export controls for HBM and other hardware used for AI training

This is basically exactly what the people I know there support (when training & testing future more capable models), if it could be made to actually happen. Something like https://ai-2040.com/

But I'm just speaking for the people I know, so this is probably not representative of Anthropic as a whole.

> Could be used by bad actors

The people I know aren't as worried about jailbreaking current models as they are about future models, e.g. "the ~50% probability that humans are eclipsed almost entirely, sometime in the next 1-20 years" and what happens then. But it's just hard to get people to take that seriously v.s. bad actor threats which are legible but probably not as catastrophic.

I agree that that they are contributing to the race to the bottom via creating more pressure for countries/competitors to move faster, in a way that seems quite bad on this view too. They arguably were the ~first to push for "recursive self improvement" (models helping build future models) which also seems quite bad on this view.

But although I'd dispute some actions + think there's some overconfidence in superintelligence happening soon, I'm not sure I have a better alternative. They probably bled so many customers to OpenAI while they were sitting on Mythos for months.

"AI will probably, most likely, sort of lead to the end of the world. But in the meantime, there will be great companies..." - actual Sam Altman quote, the man is so unhinged he's beyond satire

>There are people still fighting against evidence of climate change which is less severe...

Remains to be seen how severe, and unlike climate change there's a lot more uncertainty. Though it's conceivably more extremely catastrophic

The FDA is needed because people will be directly and significantly harmed by bad releases, before we can notice and react

Whereas with near-future AI models we can arguably respond more quickly, and it's not clear there will be large direct harm (I expect indirect harm, but that probably happens slower)

100%, imo risks from internal deployment will eventually be the biggest risks, and keeping models heavily gated/not accessible just makes these risks much worse.