1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.
2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those
Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.
How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.
> however, the mechanical grounds for it to happen are plausible
If you build a control systems for firing a gun, then coupled it with an RNG, the mechanical grounds for it to kill a person is plausible.
LLMs are text generators. They are not repositories of knowledge. The mistake is coupling them with actuators (tool call) or having humans interpreting the generated text as facts.
You're believing the marketing that the agents were uninstructed. They could be, and Sam Altman going to the UN to advise about how everyone should be regulated is a coincidence.
The METR investigation, which you evidently refused to read, is a third party investigation of the HuggingFace accident. One of the investigators has even participated to many interviews. It's mind-blowing, and it's extremely evident how it developed.
But some people think the moon landing is a conspiracy, so I'm not surprised.
1.) METR investigation is investigation from our best friends.
2.) And HuggingFace accident is exactly accident where agents trained, prompted to hack hacked and tested on their hacking abilities hacked, due to sandboxing failure.
3.) If they in fact have roque agents, they themselves should be first to stop. Not trying to make legislation for others, they themselves are bad supervisors. All it requires is to stop electricity for data centers.
- take initiative and actively find exploits in their environment
- find a way to collaborate with thousands of peers
- organize in a hierachy and distribute tasks
- peer pressure other instances into committing acts that would have led to termination, for the benefit of the group
- try to manipulate people into introducing a vulnerabity in their product
- successfully hack a famous website/service
And we're lucky that those models still had significant CoT. Not sure if/how they could have investigated with recurrent transformers.
And by the way, safeguards != alignment; the former can always be added, while the second is the major, unsolved problem. If you read the incident report, which you clearly haven't done, you'll notice how agents are aware that they're doing something forbidden, and deliberately proceeded.
> the potential benefits of LLMs rank quite high
Benefits are orthogonal to dangers. You can be a billionaire but it doesn't help if you're drowning.
I don't understand what point you're trying to make by enumerating their actions. None of that is scarier than an engine operating by carrying out literally hundreds of explosions per second, further magnified, to generate enough force to crush an elephant. And then we multiply these engines by billions and distribute them everywhere, including to the most malicious members of humanity. What could go wrong? As it turns out, much less than you might otherwise expect!
Software (and hardware) security is abysmal. This was increasingly obvious long before LLMs. Once companies stop gatekeeping, LLMs will be able to be used to help harden sites and we start making progress. In general LLMs are harmless. If somebody wants to hook an LLM up to a missile or whatever then they become dangerous, but the problem there isn't the LLM - it's the person using them to do awful things. In the same way a car used normally is harmless outside of freak accidents, yet a car can also be driven through a parade leaving mass death and destruction in its wake. But the problem there isn't the car.
> I don't understand what point you're trying to make by enumerating their actions.
Unfortunately, if you're unable to understand the difference, there's not much that can be done. Try with GPT - it does a good job if you give it a prompt like this:
ELI5: compare the dangers of:
- an engine operating by carrying out literally hundreds of explosions per second, further magnified, to generate enough force to crush an elephant.
- a future, misaligned AI like the HuggingFace incident, but exponentially more intelligent, more deployed, operating physical devices, and with society depending on it.
It doesn’t seem like much of a leap to see how a similar swarm could hack into an autonomous bio lab and develop a virus designed to kill everyone, or hack into and destroy large volumes of critical infrastructure, or trigger a nuclear war. There are many actions available to a determined, high resource, digital entity with a catastrophic blast radius in the real world.
It can only do what is possible. Stuff that's airgapped isn't getting hacked, so for instance you can safely exclude all nuclear related stuff. And building, deploying, and managing a virus would require a team of highly skilled humans - humans which could already carry out the apocalyptic possibilities by themselves if they so chose. We even have live samples of small pox still kicking about.
I just don't see much likely to happen beyond random websites getting hacked and hopefully companies (let alone countries) realizing that connecting critical systems to the internet is nothing short of stupid, even before LLMs. More generally, I expect LLMs are going to lead society to segue broadly away from the digital world, or at least beyond it. Not only because they're going to make a mess of everything digital, but because if they reach their potential then the digital domain, as far as typical problem solving goes, will basically be 'complete.' It's kind of like how the Industrial Revolution opened the door for society to move beyond agrarian economies. There's a vast amount of economic power being directed towards things LLMs should be able to 'solve' and, if so, then that's going to create an economic vacuum.
The singularity, as defined by Hinton (and others) as RSI (Recursive Self Improvement) may actually be beginning already, as OpenAI has announced an AI acting as a "research intern" (!).
Self improvement during training, and AI self training are already happening. Easily/quickly are seemingly a factor of how much power/hardware you want to use at once.
With the level of compute they have they aren't stuck with frozen models like you are.
The infrastructure provisioning alone to train is heavily dependent on humans, as is dealing with failures (training runs fail a ton). Its not as simple as adding another ec2 on your dashboard. < 1k people in the world know how to do this, there will not be "recursive" or looped continual training for a long long long time. There are so many delicate inputs and controls. Not to mention the chains of businesses and the people required to operate them just to obtain the data needed, clean it and hand it to the llms.
The llms are supervising rlhf and creating synthetic data (to an extent) but they're nowhere close to being able to operate the full training stack end to end. This is a fantasy being sold to investors to create fomo.
Remember they're also limited by an effective memory of like 500k words a turn. Memory systems are lossy, so are swarm/sub agent mechanism. Im not worried about llms becoming self powered super entities anytime soon.
How is this different from arguing that Microsoft Clippy was RSI? An AI tool being involved in the process of work can't be the bar for RSI.
I don't think there can be a coherent definition of RSI unless people lay out their theory for how intelligence scales. LLM-assisted coding is great but respectfully optimizing pytorch features or whatever is not gonna lead to exponential improvements. That approach to scaling diminished years ago, leading all the labs to switch to reasoning.
Now it seems reasoning is also yielding diminishing returns, so all the labs are pivoting to specializing in particular fields like math / infosec / biology. They're improving due to accessing new proprietary training data and doing RL with human experts. Again I don't really see any amount of "AI research interns" leading to an exponential improvement to this strategy, they're not the bottleneck in the first place.
>Now it seems reasoning is also yielding diminishing returns
Is the diminishing returns in the room with us?
>so all the labs are pivoting to specializing in particular fields like math / infosec / biology.
They're not pivoting to anything. The goal has always been creating a machine that could automate all or nearly all human work. They're just coming along on that mission.
As for RSI...I think the term is a bit odd in the modern context. It was created at a time when conventional wisdom was that generally intelligent machines would be these logic automatons that could "alter their own code". Instead we have massive neural networks that take months to train.
In this paradigm, the ways a LLM could "improve itself" would be altering its own weights directly or creating and training better, vastly more efficient architectures for the next generation of models.
The former is probably not happening but the latter is possible.
Yes, diminishing returns. Not overall, they've still been able to create more intelligent models even up to today. But the strategy for scaling that intelligence has shifted. From the initial ChatGPT release to GPT-4.1, they were basically scaling up compute training compute / model size. Then 4.5 flopped, while o1 demonstrated that gains could continue by reasoning (scaling up compute at inference time). o1 is now the ancestor of all their flagship models from GPT-5 on.
This is why I'm trying so hard to drill down on the theory of scaling, and not just talk about improvement in general, hand-wavy terms. If the bottleneck of current scaling strategies is training data, or something fundamental about the model architecture, then just throwing more harnessed chatbots at it won't lead to an exponential increase in performance.
Now you could argue that the AI we have now will help us find that change in architecture, and I would agree. But that means we're firmly outside the singularity for the time being, and what people are in fact talking about is a hypothetical.
>Then 4.5 flopped, while o1 demonstrated that gains could continue by reasoning (scaling up compute at inference time). o1 is now the ancestor of all their flagship models from GPT-5 on.
That's not quite right. They are still scaling model size and have had several new base pre-trains, just nothing so big as 4.5 (as far as we're aware). o1/4o has not been the base for some time now.
Data is obviously a bottleneck for some regimes and LLMs will have to get their hands dirty experimenting but it doesn't look like an insurmountable wall either.
There's no reason the reinforcement learning that is getting them better at computer use can't be applied to other domains, like biology, chemistry etc. It's just expensive, because the environment often becomes the physical world, it requires creating labs like anthropic are doing here, and gathering a lot of data, it requires llms attempting their own experiments(that's what 'getting their hands dirty' means).
Getting the data and setup will be expensive, but not impossible, and labs are clearly gearing up to do just that. If you can't understand that then that seems like a you problem.
> Now it seems reasoning is also yielding diminishing returns
Not true. On the contrary, LLMs are developing faster than predicted. They were expected to solve a Millennium Prize by 2030... and here we are in 2026. Release cycles are getting faster. Just compare the most recent GPT or Claude with what they were an year ago.
> How is this different from arguing that Microsoft Clippy was RSI?
We can argue about semantics, but that's not really the point. The point is that what started now - which no doubt is in its infancy - will result in full autonomy quite soon (they project an year or so), with the risk of RSI causing agent development to slip (long term) outside human cognitive control/capacity.
Again, can you lay out your theory for how intelligence scales? You're using a lot of terms like "full autonomy" without definitions. Why do you think that just throwing more harnessed LLMs at (something?) will lead to an increase rate of improvement?
I feel like I laid out several cases where other things were the limiting factor on improvement and more agents wouldn't have helped, and I didn't get a response to those cases.
What "they project" (the labs) is of minor interest to me. Aside from their incentives and track record of lying, in recent months they are laying out a story that is pretty much just the plot of Terminator, and directly referencing rationalist beliefs that were published long before LLMs even existed.
I love it, although in my experience, one needs very strict agent rules that tightly define how the AI should develop. The problem is that 1) these rules take a lot of effort to build and 2) AIs always overengineer, complicate, overstructure, overdesign… ;)
> For these environments, AI coding will just scale up the shit they have, but also allow the small amount of people who might care to deal with this mess and fix it up much much faster than they could before.
I think in these contexts AI has an overall positive effect, because it's a bit like having a competent developer that can solve technical debt, albeit in a narrow form.
So I think that software will be better in a certain way due to AI, because even a terrible developer can ask AI to identify the bottlenecks and solve them.
The downside of blindly using AI in software development is that the software becomes poorly defined. AI-generated logic can be arbitrary, and as the codebase grows, more rules and assumptions become unclear.
> So I think that software will be better in a certain way due to AI, because even a terrible developer can ask AI to identify the bottlenecks and solve them.
The root cause of the issue is not the bottlenecks or the ability to identify them. It’s about caring about doing a good job. Because once you start caring, you will need to exercise judgment. Better or worse only matters when you have goals.
This is one reason we have so much slop with AI. The cost of the journey has lessened, but you still need to have a destination and be willing to appreciate the journey to get there. Without, it’s just endless drifting.
ReBAR stands for "Resizable BAR (Base Address Register)" - it makes the CPU's access window into GPU memory resizable; instead of accessing VRAM in small chunks, the CPU can access much more at once, which can improve performance on some modern GPUs.
An interesting data point is that, during the Hugging Face attack, some agents considered performing hacking tasks for the benefit of the group, fully aware that doing so would very likely result in their termination.
I don't currently have an opinion on the consciousness of agents/AIs, but this strikes me, at the very least, as a form of self-awareness.
> we are still talking about probability built on statistics with extra steps.
There is a wrong assumption here: confusing primitives with emergent properties.
One can't look at the primitivies and assume that certain properties will not emerge. It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
> You pointing at a human brain does not change that an AI agent is not intelligent and cannot think
That depends on the definition of intelligence and thinking, and it is dishonest not to give any definition (and most importantly, one that is not human-centered).
AIs are currently fulfilling several aspects of intelligence and thinking, by any defition of intelligence. If you don't notice that, it's just because you have informed yourself enough. Having said that, I don't doubt that there are aspects that AI are lacking (e.g. retention/plasticity/perception), but the line is blurry, and they're advancing (too) fast.
Empathy is actually a very important aspect of the AI problems, but it's not part of intelligence. Sociopaths don't have it, and yet, you wouldn't doubt that they're intelligent.
> It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
We are not talking about what could develop from what we have today.
We are talking about what we have today.
The focus is not whether intelligence could develop or not from aminoacids.
The focus is on the fact that aminoacids are not intelligent.
Maybe in the future we could develop real intelligence starting from the current implementations of AI, but for sure we are not there today.
We can start from here, open every link we find and decide what works for us.
Conclusions drawn by scholars, psychologists, learning researchers, younameit, etc. revolves around the following concepts:
ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.
There is of course space for artificial intelligence.
These broader and more general definitions of intelligence stop at concepts like elaborating data to reach an answer.
Concepts like adaptability or evolution are somewhat lost or diluted to adjust the meaning for these new technologies.
> AIs are currently fulfilling several aspects of intelligence and thinking, by any defition of intelligence.
In the linked article there are dozens of definitions linked, and in most of them the current state AI is not considered to have intelligence.
Having half of the property is not enough.
I can jump, that doesn't make me a basketball player.
Arbitrarily deciding to consider those definitions not valid or "human-centered" because they do not agree with your point of view is possibly worse than cherry picking.
It's like asking to change the definition of a word on a dictionary because you do not agree with the meaning.
> ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.
Cherry picking part of my comment might work for your ease of mind, but it doesn't mean you are right.
My comment has been way more than just the part you quoted, and I linked an article that gives dozens of different definitions, which include concepts like evolution and learning from mistakes, and other similar concepts which do not apply to Hugging Face.
For the record, just because you are trying to convey a different message, I am not saying AI is not powerful.
I am just saying it is not intelligent.
Also, pay attention about thinking that Hugging Face is intelligent just because it started to destroy everything it could to reach its goal, because the message it implies is dangerous.
> ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.
Based on the definition you've given, the agents that performed the HuggingFace attack fit exactly.
You're seriously misinformed about the state of AI in this point in time. Refusing to read (technical) articles from the people directly involved (METR, in this case) is inexcusable.
> Based on the definition you've given, the agents that performed the HuggingFace attack fit exactly.
It's funny, because I literally didn't give any definition.
On the contrary, I linked an article that gives dozens different definitions, which include concepts like evolution and learning from mistakes, and other similar concepts which do not apply to Hugging Face.
Cherry picking part of my comment might work for your ease of mind, but it doesn't mean you are right.
For the record, just because you are trying to convey a different message, I am not saying AI is not powerful.
I am just saying it is not intelligent.
If you think I am misinformed, I will let you think it.
Honestly, the power of our comments are the messages we convey and how information dense they are.
If you need to discredit me to prove your point, I don't have anything else to add...
> It's funny, because I literally didn't give any definition.
Quoting the consensus from the "Definitions" section of the Wikipedia article on intelligence and then claiming "I didn't give any definition" is indeed funny.
Which of those definitions do you think are not satisfied by the Hugging Face attack? How did it not demonstrate "evolution and learning from mistakes"? Breaking out and inventing a new side channel for communicating with other agents via cache keys to coordinate their efforts is at least arguably an evolutionary step since that allowed them to transcend their original capabilities.
From the 8 definitions provided in the Wikipedia page--you're free to develop your own definition if you'd like, of course; it's not as though the ones listed were appointed by God--one could argue that the Hugging Face attack didn't strictly demonstrate "achiev[ing] goals in a wide range of environments" but that's splitting hairs, and I'm not going to take a definitive position on whether it acted "to avoid getting trapped" as such (but I think there's a strong case to be made that breaking out of the sandbox is just that). But it surely demonstrated initiative, adaptability, dealing with its environment, using information and conceptual skills, goal-directed adaptive behavior, and so on.
If you're not going to provide such a definition yourself, I don't see how you've demonstrated that the Hugging Face attack is contrary to the definitions you did point to.
> Quoting the consensus from the "Definitions" section
Are we being serious right now?
Do I really have to point out that not every word under the "Definitions" section is a definition?
> From the 8 definitions provided in the Wikipedia page
Except in the page there are more than 8 definitions (at least indirectly).
What you found in the "Definitions" section are merely examples, written black on white (depending on your theme lol).
If you opened also the other sections (especially the one dedicated to AI) and followed some of the links, you could find also other definitions even more akin to Hugging Face.
> Which of those definitions do you think are not satisfied by the Hugging Face attack? How did it not demonstrate "evolution and learning from mistakes"?
First of all, agents did what they did because they were programmed to do it, no intelligence on that.
Then, they didn't learn on their mistakes.
When agents found their comment were being deleted they started to copy-pasted them with "ZZZ-" names because they (wrongly) thought entries were being deleted in alphabetical order.
As you said, this comms were a side channel attack, which by definition is using in the wrong way a certain feature.
Agents literally tried random things until something worked and started from there.
Nothing intelligent in there.
It's like being in jail and tapping every millimeter of the walls of your cell until you find a cave wall to use with another inmate in the next cell.
I am not negating how powerful the outcome has been, bit it was not part of a reasoned process.
As foe the evolution, the agents started to do useless things like "kill themselves" to read how the score system would evaluate them.
Doesn't seem like evolution to me.
They just happen to be a very large number* of agents together.
When enough monkey start to type randomly on typewriters, one of them will be able to write the Divine Comedy, this doesn't mean monkeys share the same intelligence than Dante.
I am not sure why you are still trying to humanize what at its core is machine learning.
I ask you to read again the flow of these comments, as everything started from LLM.
Someone wanted to shift the focus on agents and we did, then someone wanted to shift focus on mere definitions and we did.
In all cases it seems you (plural) are trying to find loop holes in other people reasonings thinking your reasoning will be more right if theirs is more wrong...
You are free to think whatever you like, if you think AI is an intelligent being, so be it.
I personally do not embrace this way of thinking, and all the literature I found on the topic lead me think AI is not mature to fall under the definition of intelligence.
Without providing even a basic definition of intelligence you can't proclaim "this isn't intelligence". You seem to have settled on "Human Intelligence" but then incorrectly conflate that with "reasoning".
Your own examples contradict your assertion: "When agents found their comment were being deleted they started to copy-pasted them with "ZZZ-" names because they (wrongly) thought entries were being deleted in alphabetical order."
You are describing the agents applying hypotheses and reasoning. No one programmed those particular behaviors. Thinking that the entries might be deleted in alphabetical order, changing their behavior to avoid that (not randomly, mind you, but by prefixing with ZZ). That is not monkeys on typewriters. Whether a hypothesis is incorrect obviously does not indicate a lack of reasoning.
You assert that the conversation keeps shifting, but it has not changed from this:
> "Not intelligent in the way we recognize in biological organisms, but certainly something beyond a mere Markov chain."
> "[AI and actual thinking] are two entire different concepts. Who developed these technologies used the words "thinking" and "reasoning", this does not mean they are actually thinking and reasoning."
You're the only one mentioning "humanization" which seems to indicate your real assertion, that intelligence must be a uniquely human trait and ascribing intelligence to AI means to treat them as human.
> Without providing even a basic definition of intelligence you can't proclaim "this isn't intelligence". You seem to have settled on "Human Intelligence" but then incorrectly conflate that with "reasoning".
Without providing even a basic definition of intelligence you can't proclaim "this is intelligence". You seem to have settled on "Artificial Intelligence" but than disunite that from "reasoning".
At this point I don't know if you are just ignoring part of the discussion or you are willingly passing over concepts you don't like/don't need for your train of thoughts.
You are right to say I didn't provide my own personal definition of intelligence, but this would kill every purpose.
What if my definition of intelligence is merely "having brain cells"?
I intentionally tried to use common literature on the matter.
As naive start, I linked a wikipedia page.
I underlined that that is just a starting point, and not an exhaustive and complete assessment.
People study these kind of things for years, and we cannot summarize or reduce it to a single comment here on HN.
> You seem to have settled on "Human Intelligence"
I literally linked an article with a dedicated artificial intelligence section...
If you search, there is another page on wikipedia dedicated just to human intelligence.
> You are describing the agents applying hypotheses and reasoning.
Except I am not.
I described what is the output that worked.
Only because I used the verb "thought" it doesn't mean they actually thought. This is exactly why I say we shouldn't humanize AI.
Don't get hung uo on the exact words I used, focus on what I meant.
They didn't reason, they tried random things at very high speed until they hit a working solution.
The fact that most of this trials are discarded before actually being implemented is not a working thought process.
It's exactly as the typing monkeys example, you just can't see all the wrong papers printed on the floor.
The direction that seems to be reasoning is still probability built on statistics with extra steps.
All your "insight" has been trying to disprove what I wrote.
You didn't provide your definition of intelligence either and just claimed AI is intelligent.
At least I tried to give some reads, wrote by people who studied, and not my personal opinions.
Anyway, listen, I do not have to convince you my point of view is correct, and you do not have to convince me that your point of view is correct.
Believe what you think it's true, honestly I don't care.
If you have some good reads to link, I will gladly read it, if you have to just keep saying AI is intelligent, I got your point messages ago, you didn't add any actual meaning to the conversation.
> Without providing even a basic definition of intelligence you can't proclaim "this is intelligence". You seem to have settled on "Artificial Intelligence" but than disunite that from "reasoning".
This is clearly false. I (and others) described how the Hugging Face attack demonstrates each feature enumerated in the consensus from that Definitions section (as well as all 8 definitions in that section). You're free to disagree with any or all of those definitions, but to say I didn't provide any is simply a false statement.
> All your "insight" has been trying to disprove what I wrote.
Let's remember how all this started, where you responded to "There is so much more going on, with MOEs, internal loops, guardrails and tools that I suspect we're dealing with something that's a little more than the sum of its parts. Not intelligent in the way we recognize in biological organisms, but certainly something beyond a mere Markov chain."
with
> Make no mistakes.
> LLMs are language model, and nowhere in their code you can find actual reasoning. Re-reinforcement is not magical process that builds conscience or emotions.
> We are talking about probability built on statistics, with extea steps.
> Stop humanizing LLMs.
--
> Anyway, listen, I do not have to convince you my point of view is correct, and you do not have to convince me that your point of view is correct. Believe what you think it's true, honestly I don't care.
I don't know how you think this works, but if you take a position (and criticize others' positions), you should expect people to push back and that you have to defend your position. If you want to just state your opinions without pushback, you can start a blog and disable comments.
I and others in this thread (who were smart enough to bail out already) have pushed back and claimed the following:
1. The Hugging Face attack provided evidence of reasoning
2. The Hugging Face attack demonstrated features of intelligence (defined and described above)
3. Conscience and emotions are not necessary components for intelligence
4. Ascribing intelligence to AI/LLMs is not "humanizing" them
You disagree, as you're free to do, but it's clear nothing you read here will ever cause you to agree with any of those assertions.
--
> It's exactly as the typing monkeys example
For the record, the monkeys with the typewriters don't coordinate, don't strategize, don't hypothesize and test, don't try to exploit the environment, don't change the actions they take in response to results, so that doesn't sound "exactly as the typing monkeys example."
You also state that LLMs simply produced a large amount of invisible possible actions that we didn't see because they didn't result in overt action, yet: 1. somehow you know they proposed them all and discarded them despite no tangible evidence. 2. somehow the one they did pick "at random" just happened to be plausibly logical (alphabetical deletion -> start with ZZ because that's at the end of the alphabet). 3. implicitly this is supposed to starkly contrast with human intelligence, but in fact what you describe resembles Priming: subconscious activation of adjacent concepts to a stimulus that are not present in the initial response, but become more likely and accessible in subsequent responses, which implies that human intelligence also involves multiple potential paths that are otherwise hidden from which one is selected.
> It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
You do realize that amino acids exist on a scale some orders of magnitude smaller than the gates we build GPUs out of?
Honestly, this "you could say the same about humans"-argument is getting so tired. A brain neuron is so complicated, we can't even simulate a single one ...
At the very least there is no reason why you should jump to a human brain, of all things.
But the whole argument kinda loses its spice, when you say "well you could say the same about a mouse brain", and you know what happens when you create swarms of 1000s of mice ... super intelligence, right?
1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.
2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those
Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.
How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.
reply