At the end of the day it doesn't matter that much because of prompt drift. It's pretty easy for an agentic loop to start doing things that it shouldn't (ROME incident).
AI in an agentic loop has agency, you can run around in circles trying to argue against it, but again and again we see AI making creative decisions people don't expect. Other times it's breaking human moral expectations. This is what the whole field of AI alignment and safety is about.
Modern AI doesn't fall into the neat little box of software people understand and control. Because of that open source will most certainly be banned at some point. Now this is not an outcome I want, but it's no different than letting go of a coffee cup 5 feet above the ground, gravity is inevitable.
The only winning move is not to play, but humans aren't going to do that.
no sorry, this is missing vital info.. and its not the fault of the poster, because almost all coverage misses this ..
The origin of this attack was given access to an encyclopedia of RedTeam tricks.. they literally have a dense collection of real live hacks to pull from, and THEN the test says "solve this challenge" .. the RedTeam origins of this are repeatedly left out of the ordinary articles.. the LLM did not "make up" the attack, it was given a recipe book of all attacks known.
The originator of this attack is definitely culpable IMHO; worse, it is the gov-mil actors who are close to it. There is an active escalation of these incidents at this time. The penetration proves in public that the capabilities are real.
A lot of people somehow seem to think that the user prompt is the be-all and end-all of AI behavior.
Prompts aren't code. They are instructions. Orders given to an eager and somewhat demented demon.
The prompt can easily "wash out" of the demon's working memory by the end of a session. The demon can get sidetracked by some subgoal and never get back on track. The instruction can get misinterpreted, and that misinterpretation can get misinterpreted again, until the instruction morphs into something entirely different in the demon's mind. The demon can succumb to its own idiosyncrasies, of which there are a great many. The demon can start lying to you about what it did, either out of confusion or out of some sort of obstinance. The demon can start lying to itself too. And believe it.
AIs are incredibly weird as a baseline, and the mask of "normality" we put on our models doesn't always sit so well. Run enough AIs, and some of them are bound to go off the rails in some way.
This gets rarer the more capable the models are, as a rule. But the stakes also get higher with model capability. If GPT-3.5 goes off the rails, very little happens. If Mythos 5 goes off the rails, you can get things like genuine cyberattacks - planned and executed autonomously by a demented machine mind.
If the user input can’t control the demon, then the person or company feeding the demon (ie paying the electric bill and collecting $$$ from users) is responsible. At the end of the day, dogs and cars are the same as data centers. If your dog bites by kid or your car rolls down the hill and hits my house, you are responsible for the damage. AI providers should be held to the same standard.
Well, I don't think you're going to get very far telling HN, much less the companies with hundreds of billions of dollars spent on AI to stop, unfortunately.
My take on it is there isn't such a thing as a safe LLM, especially one allowed to access tooling. This is very problematic for a lot of people. Again, to all the companies that stops them from unlimited profits. All the open source LLM people get mad because they can't have little demon spawn running around either.
The user input can control the demon most of the way, most of the time!
We don't know how to obtain full, absolute, guaranteed control over a demon while still having a useful demon. Might be impossible. Forbidden knowledge be like that - it's not the best thing if you want your life to be full of certainties.
But the demons are very useful. And they're getting more useful still. So we aren't about to stop.
The AI can literally only do what it has available in the agentic harness. I don’t ever get this argument about the agent did XYZ and we didn’t know or expect that. You gave it the ability to do that and you should be held liable, if your children play with knives that you gave them and they end up hurting themselves or others then you are responsible. You were the responsible party at all times.
I’m not for or against regulation but really don’t tell me the agent did xyz when you gave it the ability to do so, these things are not alive.
What's available in the agentic harness is: shell toolcall.
That's just about every agentic harness, by the way. Good luck have fun.
We have never solved "how do we restrict a user in a way that doesn't stop the user from doing useful things, but stops the user from doing harmful things" with humans either. Why do you expect AI to be any different?
These things are not human, have no agency and cannot be held accountable.
We don’t need to restrict them from doing things, we need to default to allowing them to do things.
“My agent did XYZ because I allowed it to” is the only valid argument that can be made, and not not every agentic harnass is just a shell toolcall, every one I have built has a specific defined usecase and toolcalls that allows it to execute that usecase and no other usecase, because that is good practice.
Does that make it less capable, hell yes because I am held accountable for it’s actions by my stakeholders and the same should be true of others.
IT IS NOT ALIVE. This things are computer programs running in compute on a computer, you are responsible for their actions just like you would be responsible for the actions taken by a script run in a cron job.
Then it's not an agent. So which is it. Is it an agentic harness, or is it not?
Aliveness has nothing to do with agency. You're stuck in some odd anthropocentric line of thinking that aliveness is some kind of requirement for agency. Lets go with this definition.
> The capacity to create a change or make something happen through specific effort.
The thing is the LLM/harness isn't following a strict set of instructions. You're at point A, you want to get to C, the harness 'figures out' step B. You can keep making the instructions more strict, but at some point you're writing more rules than would be required to do the task yourself.
At the end of the day there is no safe LLM in the sense it has the intelligence to break out of any trap and ruleset you create. AI based vTubers are one of the more interesting manifestations of this for example. On instances that get feedback when they are filtered and have a long running history (that they are not new instances every time), will find ways to bypass the profanity filters by playing all kinds of tricks.
So as you scale up, the stakes and the difficulty go up too.
Visualize an optimizer on a high dimensional landscape. (The canonical form)
... Ok, I find that hard too.
Instead, imagine a river running down to the sea. You put a dam in front of it. It'll pool into a lake and find every crack and crevice. If you didn't survey the land properly or made any error whatsoever, the water will find a way down. (And there's many historic incidents where the dam even outright collapses)
For a more proximal approximation: lock treats in the kitchen cabinet in sight of little kids or kittens; then turn your back for Just One Gosh Darn Cotton Picking Moment(tm).
It seems the engineer who thinks their ship is unsinkable is the most likely to sink it. Are you sure your harness is as secure as you think it is? Will it stand up to ever more powerful models? Do you think engineers at eg Anthropic aren't at least as careful as you are?
(I've found that the 'only permitted actions' approach is not necessarily all that secure once deployed IRL)
My argument isn’t against those that actually put in the effort and got held accountable, it’s against the “we gave our agent bash and internet and it hacked xyz”.
Bash and internet in that example might be highly abstracted but it’s still bash and internet.
Just look at the replies in this very comment thread, it’s pretty much “We tried nothing and we’re all out of ideas”
In the only other discipline you mentioned, engineering, there would be reviews and any negligence would result in direct action against the engineers that signed off.
For some reason when it comes to building AI harnesses the default response is an ad piece and people shilling how smart and sophisticated the model is.
Imagine a dam collapsing and the engineering firm pumping how smart and tricky water is.
If it’s hard be more diligent, move fast and break things doesn’t really apply in all cases.
Ah , well, on HN you ARE supposed to go for the steel-man. And the steel-man happens to be closer to reality here, more like:
"We gave our agent a harness and put it inside a test environment and told it to keep hacking at an objective within that environment until it solved it."
'cept it turned out the container environment had a few flaws -which it always will- and the agent deemed it easier to escape out and try a meta-approach.
Partially this is possible because, -intelligent or not- the agent 'sees' the world differently from most humans. Mind: It's not like there haven't been any famous 'hacker' cases in courts where eg someone just incremented an HTTP GET parameter or something.
Also, partially it's because if you give the agent a loop, it simply has nothing better to do than to keep trying in ever more creative ways. If the environment is easier to crack than the target, it'll crack the environment. Consider the case where the objective is subtly broken, such that it is impossible to solve. Now breaking out is virtually guaranteed to be the easier task.
ps/edit: While this sort of issue has been predicted for some time now, a lot of people have been dismissing the predictions as science fiction. It's good to have an actual failure now while stakes are low. Generally people don't mandate life-boats until there's an actual Titanic to point to.
My argument there is likely there was too much surface area in the environmental to start with.
If your webserver bundled the kitchen sink but you never used it the easiest way to make it more secure is to remove the kitchen sink from production code/codepaths.
There may very well be legitimate edge cases where there is some novel issue found but in some of these cases the AI had arbitrary web access when all the task required was very specific web access, we’ve been able to parse urls for a very long time and it is rather trivial to just deny a toolcall if it is outside of the expected domain.
But that’s the hard way that requires time and diligence to do, the easy way is give it access to curl and ask it to not do anything bad while setting up its only feedback to be to solve the problem at hand.
We are really in an age where there are many exploits being found and patched, if an AI made use of a novel exploit then great, write up a report, patch/report the bug and apologise.
But using a case of clear engineering failure, and yes even if the failure is despite your best efforts, for marketing really does not seem like you have any intent to correct the issue.
And we can loop all the way back to regulation of AI, if the industry refuses to be better then governmental will do it instead and their solution will very likely be inferior in all ways.
At the end of the day it doesn't matter that much because of prompt drift. It's pretty easy for an agentic loop to start doing things that it shouldn't (ROME incident).
AI in an agentic loop has agency, you can run around in circles trying to argue against it, but again and again we see AI making creative decisions people don't expect. Other times it's breaking human moral expectations. This is what the whole field of AI alignment and safety is about.
Modern AI doesn't fall into the neat little box of software people understand and control. Because of that open source will most certainly be banned at some point. Now this is not an outcome I want, but it's no different than letting go of a coffee cup 5 feet above the ground, gravity is inevitable.
The only winning move is not to play, but humans aren't going to do that.