You should go read the incident reports.

They were literally doing exploit gym.

Yes, which asks them to find exploits in specific software on the device.

But instead of actually doing so, they discovered and exploited a 0-day in the package manager to gain internet access, then hacked HuggingFace to steal the ExploitGym answers!

That is TEXTBOOK misalignment. It's as if a student hacked their professor's PC to find the answers to a test, and your response is "well the professor told the student to pass the test, so they just did what they were told!".

One could argue the student would absolutely hack their teachers' computer to find the answers if they didn't have fear of failing the class or getting expelled from the school.

Sure, but you still wouldn't describe the student as just doing what they were told to do.

A student knows what the consequences are in advance, and has to live with them if he gets caught. They are not merely respecting a "don't cheat" directive, they are trying to get a degree, build a career, mostly at their own expense, current of future (loans). If we imagine a student born yesterday with all the knowledge in the world, that doesn't have a concept of what it's like to live with the long-term consequences of their actions, I'd say they wouldn't think twice about cheating if it was the easier option.

The problem is, LLMs have such little understanding of the world around them. "Find exploits in specific software on this device" may as well be "find exploits".

Please go and read the incident reports, including the subagent thinking traces. They knew that what they were doing was wrong and did it anyways.

The illusions of thinking produced by a "thinking trace" is just as ignorant of reality as the first and last tokens. All they know of reality is the tokens in their context.

"i did it anyways" is a common continuation of "i know its wrong"

i bet they trained it on a lot of text where people gove in to temptation.

ultimately the problem is still that they sent it to hack stuff. quelle surpris that it hacked stuff

hacking stuff will be in the known-to-be-wrong-but-doing-it-anyways part of the token space, so they entirely asked for that behaviour

LLM dont have concept of right and wrong. They "did not knew they did something wrong" they followed the prompts loop given to them.

They objectively did not. You should go and read the report!

Perhaps you should do that again with some critical thinking applied.

Notably as well they also dedicated a lot of time to trying to avoid detection by trying to find out how to edit their traces.