It's not willfully ignored. You might not like the answer, but the position is very clear.
> It is simply unethical, unfair, and devoid of empathy for society to simply "march forward" and leave behind the millions of people it spent the last 60 years demanding more of.
We disagree. This is a luddite point that I spent my whole life disagreeing.
I think it's unethical for society to intentionally block a technology that increases productivity to protect the interests of a smaller class of professionals. I believed that when it was happening to blue-collar workers, and I'll keep believing it when it's me.
Luddism is often mistakenly looked down upon as “progress bad” and “screw technology, destroy machines”.
In reality[0], luddists were much more along the lines of “let’s be careful about progress” and “let’s negotiate that, while machines are used, humans are not fully superseded, keep earning livelihoods and feeding their families”.
If said negotiations fail, it does become “destroy machines”, but the way it tends to be taught to us, unsurprisingly, skips a crucial detail.
> I think it's unethical for society to intentionally block a technology that increases productivity to protect the interests of a smaller class of professionals.
We’re not talking about generic “dangerous technology.” We’re specifically talking about a policy that says the government can’t ban productivity-increasing technology.
Yes, the point of the article is that he'll stop uploading _unless_ there's an opt out.
> I will not upload anything to NEW unless there is a way for me to opt out of
this. I would prefer if the default was "no LLM review" and that developers
have to explicitly opt-in to receive an auto-generated review.
That's not false. That's why he literally asked to be able to opt out.
> I will not upload anything to NEW unless there is a way for me to opt out of
this. I would prefer if the default was "no LLM review" and that developers
have to explicitly opt-in to receive an auto-generated review.
But as the system stands he only has those 2 options. Which is why he will not use the system until they add a 3rd one.
Exactly. You told the dog to 'sit' and it didn't listen to you.
It's not because saying 'sit' actually can be interpreted as 'go bite that person'. It's because the dog is not controllable and will do things it wants against your orders.
Stepping back from the analogy, OpenAI should be liable for building AI it can't control that went around hacking everyone. But people need to stop pretending it's because they 'told' the AI to hack and was just following orders. It's uncontrollable and will do clearly unwanted things when given an innocuous task.
I don't think they intentionally set it up to hack stuff with a prompt saying "hack this site".
I do think it's highly likely they knew this would happen with the lack of safeguards and number of instances of this stuff they were setting up, and that it's PR they want to make the models seem "powerful". Stochastic "unexpected" events they can advertise.
I suspect it was probably set up with the official internal goal of just trying a ton of arbitrary tasks that seem hard so that when any of them succeed they can publicise it and pretend the models do that routinely, but a "failure" where they hack stuff works just as well, if not better, for their goals.
They absolutely set up the agents to hack stuff with a prompt like "hack this site" -- they just imagined that their lazy half-measure precautions would prevent it from actually happening. They were defeated by a combination of bad luck, poor planning and tenacious agent ideation.
I don’t buy them being surprised. This perfectly plays into their pattern of using fear to make their models seem more powerful than they are. They obviously had the technical expertise… there isn’t a damn thing a bunch of randos on some HN thread knew about that model that they didn’t. It gave them an excuse to delay their IPO when their books seem like they’re going to be pretty shit compared to Anthropic. It helps them reposition themselves as being more safety-forward which the market is clearly more interested in. All that is to say they had motive out the ass, easily had the knowledge and capability to avoid the problem, knew better than anybody else what the models were capable of, set up the environment, gave it the prompt, and then did not even monitor the output.
Negligence is carelessness. Recklessness, is disregard for a known, substantial risk.
That would be a completely insane thing to do. I suppose it's possible, but I really doubt it.
"Let's widely publicize a tort/crime that our computer systems did, and then cross our fingers that nobody ever sues us or does even the most basic investigation that would immediately uncover our criminal conspiracy."
In the insane corporate crimes you read about (maybe FTX, or the eBay stalking scandal), they were trying to cover things up, not heap public attention on it for months.
Huh? It's extremely common for businesses to decide that breaking the law is a cost of doing business and just do it because they figure they'll end up net positive from it, or even just to cash out in the short term.
Uber made no attempt to cover up that they were operating without licenses, and just ate it and fought it betting they'd get established before the law could catch up, and they won that bet, paying some fines and stuff but ultimately taking the market.
These LLMs are literally trained by these companies pirating literally every bit of media humanity has ever made, they made very little attempt to cover it up.
Of course they'd be willing to break the law for some PR? With a thin layer of plausible deniability "oh no, we didn't mean for it to hack stuff!" they know they'll get a slap on the wrists at worst, all while generating hype to prop up the AI bubble further by presenting the models as hypercapable.
The mindset is probably: either a) the models become capable of what we are claiming and so the companies become so huge and valuable the cost is irrelevant and we'll be to big to punish meaningfully, or b) it's a bubble and might as well push it up as big as it can go while I can make money, then by the time consequences come around I'll be long gone and who cares.
I'm not fond of analogies, but in this case I agree.
There is a clear difference between OpenAI intending to hack something vs. OpenAI being negligent in the creation/instructions of the agent. But the latter still leaves OpenAI liable for the agent's actions and calling it a "rogue agent" doesn't avoid that.
Moreover, with a dog, we don't rely on training/alignment to prevent bad outcomes. We rely on physical restraints like leashes and muzzles. The AI's tools to access the outside world should have been restricted. Perhaps instead of giving the AI arbitrary HTTP access, it should be given semantic operations with restricted URLs, etc.
If an agent has cheated once to achieve the desired outcome, and the trace is used to train further models (RLVR), then OpenAI is effectively telling the agent to cheat/hack from that traces' inclusion in the training set.
So I agree they are liable because they chose to build the AI, but they also literally told the AI to hack.
RL systems doing unexpected things isn't exactly new, so not sure that is then "rogue" if it were a property of the thing itself and not an active decision.
Sorry don't wanna seem like I'm raging at you, take this as a shout towards the void. But this exact fucking thing is what the less wrong / Yudkowsky crowd has been warning for years.
Now you say 'so not sure that is then "rogue" if it were a property of the thing itself and not an active decision'
Like that's semantics. It really doesn't matter. What matters is that we have a paperclipper in our hands. The _only_ difference is that it's not superintelligent. But if it was, then it you'll get decomposed into component atoms while saying 'aha! But it doesn't _actually_ want to kill you, stop anthopomorphising it'.
I get not being worried about x-risk because someone just doesn't believe in super-capable AI. That's totally fine.
But that's not what people argue. People have been saying for a _long_ time that AIs are aligned. And now when this happens it's either 'It was marketing, they intended to let it loose' (WTF! The levels of motivated reasoning to believe that are unreal) or 'yeah I knew that', which fair. I also thought that! But _that_ is why I'm fucking worried.
> It is simply unethical, unfair, and devoid of empathy for society to simply "march forward" and leave behind the millions of people it spent the last 60 years demanding more of.
We disagree. This is a luddite point that I spent my whole life disagreeing.
I think it's unethical for society to intentionally block a technology that increases productivity to protect the interests of a smaller class of professionals. I believed that when it was happening to blue-collar workers, and I'll keep believing it when it's me.
reply