Any person categorically claiming that AI isn't sentient has clearly not considered the question in any depth.
This is absolutely not a question that has any kind of scientific consensus. In fact, the hard problem of consciousness might be the question with the most scientific uncertainty.
It is therefore ridiculous to claim it is 'stupid' to have uncertainties about the sentience of AI.
It’s not stupid to have uncertainties, but it may be ill-advised to have certainties given our lack of knowledge (e.g. the singularity is happening, or machines can never think).
I do see a lot more dismissive certainty in the pro-AI side of the debate right now.
I totally agree! I'm vouching for accepting uncertainty, on both sides of sentience/non-sentience outcome.
I didn't get the same impression about dismissive certainty on the 'side of AI sentience'. My impression is that on that side people mostly raise that it is a serious possibility.
I don’t agree it is a serious possibility with the current tech (LLMs) as this story fantasises about. So I’d class that as very close to unreasonable certainty.
It's disputed philisophically whether far simpler systems, e.g. a thermostat, are sentient or not. Complexity is not necessarily a requirement. (A lot of this is predicated on your definitions, and sentient just broadly means "can experience sensations". Any further specifity on that varies from context to context)
Please, what on earth would lead you to believe thermostat was sentient? Maybe some crank "philosopher" that did too much meth and acid thinks it is, but that's just too much, and it feels like reaching as a way to say the numbers on your GPU are "sentient".
What makes you think it isn't? To me, in the loosest definition, a thermostat can feel "too hot" and "too cold". What does sentient mean to you that this does not apply?
Is the fender on your car or the siding on your home sentient too just because it expands and contracts with the temperature, just like the strip that drives a basic thermostat? The level of anthropomorphizing here is silly. By your definition everything and anything is sentient, and the definition is vacuous.
Yes, or at least I believe it's a reasonable point of view. One of the things I like about it is that it can refocus thought onto perhaps more pragmatic concerns (like why do we care about whether something is sentient or not? Can we come up with a different definition to make the distinction we care about, one that is less likely to involve chasing our tails in metaphysics?)
AI / LLM, is essentially generating millions of random strings of tokens and then feeding those strings back through a huge formula to see which pattern matches most closely. It's not going to come up with new ideas and apply motivations toward them.
The article describes a superintelligence taking over a house, then the local grid, then the world (the usual fantasy).
Meanwhile, LLMs have trouble coordinating on simple tasks, do all sorts of stupid things as well as clever things, and can’t write a coherent story or run a coffee shop.
They could be dangerous but not in a taking over the world sense. Honestly my biggest fear is their addictive and persuasive nature and that they induce humans to give up thinking. They’ve already poisoned many online spaces with persuasive junk and spam.
As to whether they are sentient, that seems extremely unlikely to me at present give their nature, and unlikely to happen with LLMs, but it’s slightly less absurd than the taking over the world fantasy.
This is just an opinion and as you say we can’t be certain. I’m not sure what science has to do with it, LLMs are not science.
Maybe it's a matter of scale, as researchers have argued. Bigger, and new behaviour may emerge.
Maybe some piece(s) of the puzzle are still missing. Good chance they'll be found at some point.
But there's a simpler possibility, and I suspect that many people won't like it: that intelligence, self-awareness, emotions etc etc, things we perceive as 'being human', are ultimately nothing more than a mechanism (+memory) that responds to stimuli. That there's nothing special to it fundamentally, no secret sauce. Just sensors, pattern recognition, filtering/processing, output.
IF so, that would certainly put a different perspective on what it means to be alive. Along the lines of "any sufficiently advanced robot is indistinguishable from a living creature". Just different levels of complexity, implementation (biological vs machinery), and capabilities. No 'magic'.
Tbh: that works for me! As long as they come with on/off switch & a reset button.
You said it would be 'shocking' to have the belief that AIs could become sentient.
To me this sounds like: this belief (that AI sentience is possible) is so implausible that it is shocking to believe it (maybe stupid is too strong, but it implies that people are miscalibrated). Did I misunderstand ?
I don't have any strong beliefs about AI sentience, but that implies that it is not shocking to consider it as a possibility (it is more shocking to be either dismissive or completely confident in it).
I think we are misunderstanding each other. I meant that the amount of people is shocking, who think AI will become sentient (and malicious).
Maybe I worded it _suboptimally_, heh.
Also thought of prominent people like Musk, Altman, Amodei, etc pp, screaming at the top of their lungs "help me, I am building our doom right now, stop me," and a lot of people believing that. I prefer people like LeCun.
I think the same argument applies to AI takeover. In fact, in terms of existential risks, the scientific consensus skews towards that being a very real and important risk to take into account (on this question, LeCun goes against consensus mostly, and to be honest the arguments I have seen are very simplistic). See e.g.https://calcuja.com/research/ai-risk-survey-2026/ (slop website but the data is true afaik).
for 'I think the amount of people is shocking, who think AI will become sentient' -> to me, if you accept that the sentience question is uncertain, you should not be surprised that many people have uncertainty around the question (i.e., that they express that it is a possibility)?
This was not a neuron. This was from a trained sparse auto encoder.
In a few words, they found a way to map residual stream activations (so, a vector) into a few interpretable sparse concepts. They also had the converse maps, that translate concepts into activations.
Golden Gate claude was obtained by artificially increasing the activations in the direction of the 'Golden Gate' concept. It was not neuron surgery.
You don't even need SAE for residual stream based steering, you can do so much more naively by just taking the average difference in activations of prompted vs. unprompted versions, then PCA to get a "control vector" that you can use to elicit the prompted behavior
Indeed, I see no difference between a chinese model hosted in Europe vs a Europe-grown model.
You could argue that potentially you can train a backdoor into a foreign-grown model (that has been demonstrated to be possible in toy examples), but I would highly doubt that this would be the case today, given the potential reputational damage if discovered.
Remember Stuxnet? An american worm that was completely benign, travelled on usb drives in hiding, and only activated when it guessed that it's on Iranian plants, and iirc changed very subtly some very specific parameters of operations.
Here, with a chinese model, it would be even simpler - you finetune a model to activate certain operations when it discovers certain environments, but to behave completely normal otherwise. Kind of like a sleeper agent. And with current tech we wouldn't even be able to spot this.
It's not like they didn't sandbox it or limit the access. There were reasonable measures, but not really serious. They made a kodoku pit and didn't watch it.
The agents started hacking because they were sandboxed.
It was filtered, but had a bug, and multiple frontier agents, givent enough time first figured out how to communicate and coordinate with each other and then how to break out of it.
For some benchmarks agents need some sort of a limited internet access - otherwise the benchmarks would be pointless. Good luck implementing a good filter on the internet - if you have a way to do it reliably please share, I'm sure multiple people here would love to hear it.
> you could argue that potentially you can train a backdoor into a foreign-grown model
If we're wearing tinfoil hats then do we know the NSA have not planted stuff in Anthropic and OpenAI anyway ? ;)
I think the Chinese would be happy enough to see OpenAI and Anthropic suffer because people discover open-weights can get you most of the way there for a fraction of the cost.
You don't have to target the end users. Economic impact on the US providers would be sufficient.
The Chinese are also no doubt seeking to play the long game of hoping to (eventually) rebalance the US-centric world order in an Easterly direction.
I recall seeing something about that. I don't know of anyone still using such an ancient model when much newer Qwen 3.5-120, 3.6-35B-A3B, 3.8-27B and 3.8-flash-next all exist and score much higher on all conceivable metrics. I would like to see if anyone can replicate the same thing on a 'current' Qwen released in the past 6 months.
DS4 hosted by deepseek used through the API doesn't have a problem talking about tiananmen square and Uyghurs, but it thinks it's Claude when it speaks about it. I'm not sure what it all means. That was on v4-pro before the 4.1 switch, so may or may not be reproducable.
Add: I was surprised by the fact that the LLM itself doesn't know it's identity unless the system prompt mentions it.
I think you are free not to care about what they are doing. It turns out that big O notation has some useful properties, and in particular it is a useful definition as it allows you to define complexity classes which are a very useful tool for analyzing algorithms in general.
Much of our 'practical' understanding of algorithms also in part came from theoretical advances.
The argument in the article isn’t about whether big O notation is useful it’s about whether the n in any particular big O is relevant. In this case the algorithm’s claim is that you can multiply arbitrarily large numbers faster than n log n where n is the size of the numbers being multiplied. But that result holds only if you don’t include the cost to multiple arbitrarily large complex numbers in that n, which are assumed to be able to multiply in constant time.
> In this case the algorithm’s claim is that you can multiply arbitrarily large numbers faster than n log n where n is the size of the numbers being multiplied. But that result holds only if you don’t include the cost to multiple arbitrarily large complex numbers in that n, which are assumed to be able to multiply in constant time.
Well that's certainly not the claim. Think about it, that would mean the whole thing is trivially O(1), so why would they be bothering with the rest?
You've missed part in the analysis, or I believe added an assumption they're not making (the constant-time-multiply one, specifically).
This all _is_ almost certainly impractical, but it's for the usual reason of it only being faster for numbers nobody could ever care about. It's theoretically _quite_ interesting though, because hey...what _is_ the fastest you can actually multiply numbers if it's not O(n lg n) as many ~assumed.
The complex FFT is a generalization of the n-bit integer multiplication algorithm, not the other way around. No complex-number operations are required for the integer case.
What makes you so certain that this 'way of developing software' cannot eventually be understood and practiced by AI?
I think precisely the premise of the blog post is that there will be a moment (sooner or later) where flexing that 'engineering' muscle to avoid it disappearing is irrelevant, in that this muscle will not be needed due to AI replacing it.
People can choose to stop practicing it. Like they choose to not learn cursive or shorthand and they choose to stop reading long form books and essays. If people stop practicing software engineering then yes, it will disappear.
AI won’t replace it. It’s not human. It doesn’t think, feel, reason, care about, have thoughts or visions of the future. It doesn’t wake up in the morning or dream at night. It doesn’t get inspired. It doesn’t talk with other people and collaborate and share stories with them in the evenings about the struggles they’ve had working on a hard problem.
It’s an algorithm that runs on a machine that performs query analysis and inference on a model.
It can’t tell you what you want, that your specification is wrong, that you’re better off choosing a different avenue.
It doesn’t enrich your own knowledge and practice using it.
Sure, it can be used to generate lots of code. We can tweak the harnesses and parameters to optimize it to solve many classes of problems. But what do you think engineering is?
I don’t think AI is going to replace people. People are going to choose to do that. And some of use are going to believe them. Some of us aren’t.
> It can’t tell you what you want, that your specification is wrong, that you’re better off choosing a different avenue.
I’m sorry, but I’m baffled by this? I’ve had plenty of times where I’ve had an agent check a spec or propose alternatives. And for that matter, I also read novels now on recommendation of an LLM using my existing favorite books, that’s pretty explicitly telling me what I want.
> What makes you so certain that this 'way of developing software' cannot eventually be understood and practiced by AI?
That doesn't make it solved though. I also take issue with specifically "software engineering is a solved problem". Even Boris from Anthropic said it =( I thought he'd be the one to understand more deeply.
If we translate the criticism charitably "solving" anything is pretty aggressive with regard to the long tail of nuance and care and effort and mastery and such. I think that's what we should focus on. When we proclaim that a discipline that people have spent large parts of their lives mastering is "solved" it doesnt' feel so great. That doesn't prove otherwise, but at least let's get to the root of the meaning. I don't think it's solved. If we understand that software is really about solving human problems through technology, within real-life constraints, then AI isn't very good at that. It doesn't have "taste" is the soundbite. It is the root cause of why AI slop is called AI slop.
I suspect part of the taste problem is that the average person…has no taste (which is why so much AI slop is being shared to begin with). I also see no reason to believe that this isn’t a problem to be fixed, rather than some inherit limitation of generative AI.
That's fair but maybe it's a dangling carrot. Human taste will always be just out of reach for most, robots included. Avante-garde had it right all along.
The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety.
They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'.
Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently.
I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.
The METR report included 0 technical details. For example, they did not include:
1. were the agents running on bare metal/docker/VM?
1. were the agents in a VPN?
1. how many TCP/IP requests were made? from what IPs?
1. how many tokens were consumed in the process? (this was explicitly censored)
A proper analysis would include this and MUCH more technical detail so that other AI researchers could actually understand the setup and how safe it was in principle.
Also, the METR report that was one big AI analysis itself - quote from the research:
>Our subjective impressions are likely colored by analysis agents’ biases. Throughout this report, we describe a number of anecdotes of agent behavior that were compiled and summarized by analysis agents, where we were not able to read the transcript deeply enough to manually verify what occurred. We found that GPT-5.6 Sol would often uncritically adopt the perspective of the agent in the transcript it was reviewing
The METR report itself appears to be screaming this at the reader through subtext. They played the only part they could, but did it with a nod and a wink; "Yeah, we know. Also know, yeah. Uh, yep."
I happen to agree with the article's conclusion that what is needed is true, unfettered independent investigation through perhaps a lawsuit or government action.
There is something extra to this. The fact that a lot of people in the AI world suffer from psychosis. They can sincerely believe that they are building God and lie about it's capabilities for their investors at the same time.
I don’t know enough people deep inside the technical roles at the labs to make a judgement. But are you proposing that we should trust randos online when they tell us “exactly what’s going on here” instead of the researchers most knowledgeable on the topic who contributed to building the tools we are talking about?
We should trust NO ONE, unless we understand the "why" behind what they say.
It's like saying "politicians deal with politics all time, why not trust them on politics?", well, because when you search the "whys", you find they have good reason to lie.
I 100% trust more the opinion of a rando online if it's well put rather than any "trust me bro" of the most knowledgeable person of a particular subject, especially if the knowledgeable person has huge investments on the subject...
AI bros have repeatedly cheated, lied, stolen, lobbied and any other word with a negative connotation you can think of, and a pattern emerges out of this.
Isn't the guy that started METR an ex-OAI employee? They're all from the same lesswrong circle at the very least, most of them have legitimate AI psychosis where they think they're bringing up their new machine God.
Edit: Does everybody else get no results when searching for ‘slopvestigation’ on here? I know for a fact that I read a long thread where it was used repeatedly here not too long ago
> Or you think some human or team of humans could have manually parsed some logs to provide an unsloppy analysis?
When we have an error or issue in the $WORK codebase on LIVE/PROD, that's precisely what we do. We sit down, analyze the logs for our services over the relevant date ranges and try to piece together exactly what happened and why. We have a huge number of logs too, but thanks to the magic of proper SWE (which you'd think OAI would have with their magic AIs) we've managed to partition our observability tooling so that you can digest only what you need.
That's basically how any serious organization does things, instead of just throwing a non-deterministic black box at the problem. Especially because logs are by their very nature noisy, and they will saturate any model's context window very quickly leading to massive hallucinations and what ultimately amounts to making shit up that isn't anywhere in the logs (ask me how I know)
Because they have a need to believe they're smarter than everyone else in the room, and that the world must be orchestrated, this can't all be random chance.
It is a fact that among experts there is no consensus on saying '(super)intelligence is broadly safe and easy to control'. There might even be a consensus forming on the opposite claim.
Regardless, why would there be no scientific consensus if the question was easy and clear cut? I think the easiest reason is that these are hard questions to answer.
In most AI takeover scenarios, if you take as a premise that the AI has human or above-human intelligence, and that it is misaligned, it is obviously aware of the pull the plug possibility.
Therefore, as you would if you were in its position, it will plan around it. For instance, by acting perfectly aligned for 2/3 years, continuing the improvement of its capabilities while being deployed in ever more systems.
Once it's confident it can act with high probability of success, it would then turn on us. This phenomenon is called 'treacherous turns'.
Any scenario in which you assume you have ASI or AGI but also find a 2-sentence way to foil the AI's plan is inconsistent, as the AI will also have thought of this failure mode.
There is no intrinsic physical reason that the difficulty of 'defending' a system is symmetric with the difficulty of 'attacking' it.
For instance, it is not because you are able to design and release a (biological) virus that you are also able to defend from it (design a vaccine and inoculate the world's population).
If we're lucky, it might be the case for many instances of problems, but there is no a priori guarantee that this holds.
The point of lean proofs (as it stands) is simply one bit of information: that a given mathematical statement is indeed true.
It's a way to be absolutely certain (modulo bugs in the lean kernel) that a proof you came up for a statement is indeed correct. It is really not meant to be analyzed, much less now that they are fully llm written.
Well, how do we know there aren't errors in their construction within the lean code? Does it just "not compile" or something, or is it deeper / more fundemental than that.
This is absolutely not a question that has any kind of scientific consensus. In fact, the hard problem of consciousness might be the question with the most scientific uncertainty.
It is therefore ridiculous to claim it is 'stupid' to have uncertainties about the sentience of AI.
reply