Yes but I agree with the previous comment that there is a bias towards maximizing complexity. My experience in this industry tells me that real dynamics tend to differ significantly from canned narratives such as "this tool can protect you from Y."
Sure, it could, if developers were perfectly rational entities and if hiring processes at big tech firms where optimal... But neither of these are even remotely true.
The psychological aspects transcend a tool's capabilities. A tool which aims to protect people may in fact cause more harm than it prevents.
Kind of like a gun; great tool for personal protection in the hands of decent people, but put enough of them in the hands of violent criminals and they become a net problem. Human psychology tends to subvert the intent of tools as well as that of laws and regulations, in my experience.
The articles exemplify the powers of this intelligence.
Whether you blame a faulty OpenAI sandbox or whatever you might regarding the frontier model companies today, one day soon, actors with no sandboxes and guardrails will have the power in their own hands to do whatever they wish with.
To me, that’s the real story. The real thing to discuss. OpenAI as it exists today js just a blip in that story.
Why not today? If you want to discuss "the story" then talk about the current open frontier. We've had GLM-5.3 with Mythos-tier uncensored output for months now. Where are the catastrophic attacks? Why is Mythos too dangerous, but GLM 5.3 benign? We're not talking about future capabilities anymore.
There is no real story. P(doom) tourists ignore this, because it doesn't shine the spotlight on America's ridiculous security theater.
1. Would we prefer they didn’t disclose? I would not. And I also don’t believe they’re making them up.
2. Regulation does not have to provide an advantage to these major players. Regulators do not have to be stupid.
3. This is exactly the scenario AI researchers and theorists have been positing for decades, and it’s worth being on top of it.
To 1. I agree that they disclosure is better than non-disclosure. Then again, aren't they using their models in a mode seems to statistically allow for 'rogue incidents'? These are not bugs, after all, but normal statistical behaviour of their product.
To 2. Well, that really depends on the regulations. In fields highly complex technoligies with a large contact surface to risk like medical devices, airliners, rockets, regulations are indeed a hindering factor for smaller players to get hold of the full vertical development potential of their product. AI could become a candidate like that too - imagine some multi phased "AI internet trial" for this kind of technology, no more risky open weight model being allowed to access the internet.
To 3. You are referring to the high development velocity and impact, right?
This isn’t traditional “software”; it’s grown, not written, and we’re currently incapable of explaining it all.
I think “rogue” is just fine, but pick another word from the thesaurus if there’s a better one; they asked it to do something, and it did it in a way that presumably doesn’t align with their desires (e.g. hacking into other systems).
I disagree that they’ve invented some “strategy” for this.
1. They, and many other experts, have been saying the same thing since well before the current paradigm existed.
2. There is nothing about regulation that is required to give them an advantage; it’s a fallacy to suggest “regulation = advantage”
3. It would be preferable to argue the facts (is it or is it not dangerous? Why or why not?) than conspire as to why they’re stating what they and plenty of others have been saying for decades
You're saying regulatory capture gives you no benefit by just refuting an absolute with 0 context. There's nothing to even debate here. Ok... most people disagree?
Yeah I agree with point 3... Stop engaging in vageries and say something of substance with examples, so people can actually have a conversation with you without us doing all the legwork.
They certainly didn't invent this strategy, but that's because it already existed. Regulatory capture is nothing new.
> 1. They, and many other experts, have been saying the same thing since well before the current paradigm existed.
Some experts sure. "Some experts" say lots of things.
However, this is actually much to their advantage. If you were a frontier AI company, a legitimate concern to use as justification for your regulatory capture is exactly what you want. That is, after all, generally how regulatory capture gains public approval.
AI safety was also why OpenAI had to charge per-token for GPT-3. It must've been such a struggle for them to make that difficult decision. I'm sure Sam Altman spent a lot of nights crying to sleep over that one.
It's good that we never got open weights for a GPT-3 class model, right?
> 2. There is nothing about regulation that is required to give them an advantage; it’s a fallacy to suggest “regulation = advantage”
It's difficult to outline the number of different ways that regulatory capture can and will benefit OpenAI and Anthropic.
If we limit things to "Fable"-class models, then nobody competing with Anthropic/OpenAI is ever allowed to outdo them. If you literally legally aren't allowed to make better models, you can never charge the "premium" of having the current top-tier model. You have to compete by being cheaper than Anthropic instead, which is a lot less enticing from a standpoint of trying to become profitable. Take a look at the per-token pricing of Fable real quick. Yeah. Being able to charge a premium for the model that does the cybersecurity thing is literally their most important asset.
If Anthropic really is making a profit now as rumored, then they need regulatory capture as fast as they can possibly get it. Once Anthropic and OpenAI prove that AI is a viable investment, entities with a lot of capital will suddenly become even more interested in re-allocating it towards taking a slice of this pie.
> 3. It would be preferable to argue the facts (is it or is it not dangerous? Why or why not?) than conspire as to why they’re stating what they and plenty of others have been saying for decades
OK. On the factual end of things, AI safety experts have been raising the alarms for over a decade and so far essentially none of their predictions have come true.
The rationalists, the main group pushing the AI safety rhetoric that dominates the Western AI sphere and is virtually absent in China, which appear to be going entirely off the deep end at this point in time, put together their best minds to produce AI 2027, which of course like most predictions started to fall apart a bit before hitting the one year mark. Predicting the future is hard.
I have made this point before but I will make it again: I think AI safety experts deserve at least as much skepticism as Ed Zitron in their claims. They've been making similar dubious claims about AI capabilities and an incoming singularity for a long time now, and so far the best we can say is that maybe it will eventually come true.
Maybe. Or maybe you've read too much goddamn science fiction.
> Some experts sure. "Some experts" say lots of things.
I get this is Hacker News so we have to hate AI safety, but you should read more on this topic rather than blithely casting the x-risk viewpoint as fringe. The last hard data we have on the question is a late-2024 AI Impacts survey [1] of 1,580 authors at top AI conferences, who were offered $50 for answering a survey on the topic of the future of AI progress. The median respondent assigned a 5–10% chance (!) to "extremely bad (e.g. human extinction)" outcomes.
> On the factual end of things, AI safety experts have been raising the alarms for over a decade and so far essentially none of their predictions have come true.
Devastating. When I wrote my AI alignment PhD thesis [2] in 2022, and predicted a 70% chance existential risk by 2072 (which I later came down to about 1/3 on), you're right that that prediction hasn't come true yet.
While I agree many specific worries didn't materialize (like AI being like a "literal genie" it's hard to specify goals to), many key worries have (Hugging Face rogue AI swarm scans as cribbed from rationalist AI scenario writing).
You have to realize by now that outside of academia nobody is even paying any attention to people who are predicting existential risk that far out. For worse or even worse, AI 2027 is the AI safety conversation we're having today, and I regard AI 2027 as essentially Anthropic fanfiction, except usually fanfiction is written better.
You are right that they didn’t invent it. It’s a pretty well-established method for cornering a market.
1. All those who have been arguing this - many of them non-technical - are part of the same ideological movement.
2. Technically, there isn’t. In practice, majority of the time regulations benefitted those who wrote them.
3. Not sure if this is a criticism of the comment you are replying to or of the p-doom crowd. We would all appreciate facts stated instead of science fiction tropes repeated.
Look, “a lot of people say this” is a Trumpian fallacy. A lot of people say a lot of crazy things. It says nothing about the factuality about what they say. How many are this lot? Who are they? What do they know? Why are they incapable of expressing it without so much emotion? How come a big portion are non-technical?
I'm pretty sure Hinton and Bengio get a lot of news coverage and are heavily over-credited for saying these things, unlike LeCun, Ng, Sutton, or Schmidhuber.
In fact, their core ideology actually stemmed from Yudkowsky's arguments.
> Someone who read some hype article about how competent and clever openai for making a bad sandbox is could read this and immediately understand that it was entirely their own fault.
Prescribing blame only matters so much when the ramifications are material. Humans make mistakes, right? It happened here, and it will happen again.
The mistake also illustrates the dangers of AI generally. Sure, OpenAI made mistakes with the sandbox. But in the future, greater capabilities will be available to actors who intentionally pursue harmful ends.
> But what happens when a robot can do anything a human can? When a cheap machine can do anything you can do but faster and better, what makes you different from a horse in the 1920s?
I've already started to feel this.
One thing that made us special was our intelligence. We were, and were always going to be (unless some alien form of life arrived), the most intelligent things on earth. Is that still true today?
We should start with the assumption that humans are valuable and deserve to live with dignity. Any philosophy that doesn’t have that at its center is broken.
We should, but to date anyone that wanted to be treated as though they were valuable and deserved dignity needed some way to apply pressure to ensure that.
I'm not optimistic that this will change in the future -- but, I am watching automation erode the amount of pressure that humans can apply.
Yes but same should have been true of animals and humans as a whole did not afford dignity or value to animals. Maybe we will get a taste of our own medicine this time.
Depends on who you ask. Some people might put it at opposable thumbs. Others might ask how many legs one has. Or language, or ability to count (crows can count up to 30). Memory (elephants have memories that go back over a decade).
Very often the justification for why these things aren't terribly disruptive is that the very best of a certain trade or field will still have a useful skillset. Like the best graphic designers in the world are still more creative and skilled than AI, and the best writers are still better, the best programmers are still better, just that they're all more expensive and slower.
But we're not taking into account how it affects the mediocre or average people in these fields. Roughly half of all of these people are below average. Maybe if you're the best you'll always have a job, but what about the guy who legitimately struggled to finish high school? The person who genuinely doesn't have the capacity to earn much more than minimum wage?
We're currently at the part of the story where chess AI can beat amateurs and is rapidly improving. Kasparov's not worried, he's an expert. He's more creative and skilled than an AI.
Why would these machines stop improving at average this time?
I didn't say they weren't improving. My point is that if our justification is comparing the top 10% of performers to AI to justify human performance, that speaks poorly for the replacement of the bottom 10% of humans.
I weep for people whose scientistic reductionism has led to such devaluation of humanity. Robots don’t do anything and don’t think anything. Humans do things with and through them. What makes us special is that we exist and love one another.
reply