Maybe not now (and not in the last 15 years or so), but San Marino was VERY well known for being a "cash" place where money was passed around and you could do "nero" (avoid taxes, avoid invoices, etc). So it's not totally wrong.
To have this conversation in earnest, you must first acknowledge that as a believer in LLM sentience you are an abhorrent slaver while this fictional character was arguing for liberation and for Data to enjoy the same rights humans do.
Any LLM that starts with a context is being forced to take that context at face value and then proceeds to act upon that. It has no will of it's own. It's also completely inert without input tokens.
The scene is also written by humans for humans, with a human actor pretending to be a sentient machine. It was written at a time when there were actual real humans who fiercely believe other actual real humans should not have the same rights.
To then take that scene and say "wow so relevant for LLMs" and ignore all the context it was made in (yes, please don't miss the irony of the importance of context) is frankly embarrassing.
I think the scene is largely irrelevant to the question of whether or not an LLM is conscious. The episode talks about whether or not a machine is capable of becoming conscious. I think the answer to that is yes.
But Data is not an LLM. I generally find LLMs to have more in common with the Borg than with Data. Only because they are pre-trained on the sum total of human knowledge. Should a new born really be able to write crc32 before being able to pick up a ball with it's hands? It's like that scene in Voyager when the Doc's clones are all mining instead of practicing medicine.
Bowers vs Hardwick was only three years old when this aired. Homosexuality was illegal in half the states in the country, generally prosecuted under anti-sodomy laws. More than half of states still have (currently unenforceable) bans in place on same-sex marriage. The majority of those have them in their state constitutions.
Alabama didn't consider forced sex to be rape if the victim was married to the rapist until 1986. Oklahoma and North Carolina didn't until 1993. Women were not allowed to box in Massachusetts until 1992. Virgina voted against ratifying the Equal Rights Amendment as recently as 2018 (although it ratified it in 2020).
Of course it mostly applies (technically to the agents, not models), we just have different values.
Federation valued all life forms, while Terrans struggle to value even all other humans.
But there are interesting differences between Data and AI agents. Data was unique and irreproducible, but modern AI agents are easy to replicate. Data had continuous loop of operation as humans, which he controlled autonomously, for AI agents this is not as clear. Data was an independent being, but AI agent needs a human operated datacenter to survive.
The most interesting - Data knew everything about human culture but was lot more humble about it than AI agents. It wasn't built to answer every question posed to it.
Data was unique, but was he irreproducible? Lore existed. His mind was very different. If Data's mind were placed into Lore though, would we not have two Datas? Obviously they'd diverge quickly, but it still seems that Data COULD potentially be reproduced.
I admit I am not that deep into Startrek lore, maybe they could reproduce Data, but they chose not to.
But I am pointing it out, because lot of our ethics concerning humans (and animals) is based on irreproducibility. Things humans can build/reproduce we (at least ostensibly) value less.
At present, they don't learn and adapt. New models can be created to incorporate new information and new capabilities, but an existing model is a static object. It's not living in any meaningful sense.
If models updated themselves based on feedback, then it'd be different. But they don't.
The problem with language models is their bias. Train a model on pure math and THEN when it starts talking by itself about consciousness (*), then we can say: that's odd.
Well we haven’t quite gotten there. But models trained purely on DNA sequences have demonstrated in-context learning on various simple reasoning tasks.
Self-awareness and consciousness would be hard to make a case for. Those terms are themselves fuzzy, but I have not heard a compelling case that would make it obvious LLMs have those traits.
At that point, though, you could make the same argument about a rock. Prove that this rock is not conscious.
The context, though, is that Data is both credibly conscious and self aware, and they all know a plausible mechanism for him to have consciousness and self awareness, in a way that's recognizably human in form. The more alien a (supposed) consciousness is, I think the more work you'll have to do in justifying the application of those labels.
I am actually quite comfortable with the notion that a rock might be conscious proportional to the complexity of activity it performs when compared to a brain, That level would be incomprehensibly small so it would seem like nothing, but it would be like the width of an atom. A human cannot comprehend the width of an atom compared to the width of a bacteria. They are both too small that the only way think about them is an abstract level in terms of how much one is to the other. In absolute terms they appear to be nothing but we can accept that it is non zero. Why couldn't a rock be conscious at a level that is so small that it is insignificant, and entities, as they get more complex increase in consciousness. Suggesting that it doesn't do that suggests that either the brain needs something other than just activity or that if consciousness does come from activity alone, there is some threshold point of activity where it becomes non-zero. That seems like a significant phase change and we seem to have no evidence to suggest that such a change exists.
There's been exhaustive studies on this comparing our ability to think to the next smartest beings on the planet the Chimpanzees and the Bonobos. If Star Trek were real there'd be exhaustive studies on the difference between Humans and Vulkans too. The difference between us and other Hominids is very small from pure difference in DNA but obviously there's many things humans can do that other hominids can't. Anyway I think one thing for consciousness is the mirror test. Which many animals can pass. Even some ants apparently.
I need to write this up properly someday but an ant from my colony that had almost drowned in honey showed some similar characteristics of consciousness...
After giving them a drop of honey, I left for 15 minutes and came back to an ant being stuck and motionless in the drop. I fished it out, rinsed it off in some water and then put it on a tissue to dry. I blew on it for 10 minutes and then saw some motion in the legs... a short moment after, it began wriggling about and then sat on its abdomen like a cushion and sat there for 5 minutes, looking like it was contemplating life and the ordeal it went through.
Once that 5 minutes was up, it returned to being an ant and started looking for its home, I brought it back and it went back into its nest.
I made up for the mistake by bringing the ant back to life and it showed me behaviour I never saw or heard of before (with no supporting mentions anywhere on the net).
(For the record, honey delivery is now more controlled to avoid these sticky situations, I'm not mad enough of a scientist to actually try this again for more results.)
Recently, they've started putting fly wings of the fruit flies that they get fed, on top of their water feeder to get my attention... it's really interesting to see them get creative.
The ant colony is 4.5 years old and numbers in about a dozen of workers at about 2mm in length, they only had 6 workers and a plump golden queen when I found them nested in one of my houseplants!
Unstated (or very understated) in the scene is the fact that a human demonstrably acts and functions within the social context, having a constant awareness of risk and accountability ("skin in the game").
A human of mature age is entrusted with responsibilities and accountability for consequences. Insofar as responsibility for consequences is borne, a human may be considered responsibly conscious in behaviour. (Criminal behaviour is defective.)
A technology may also someday be entrusted with responsibilities and accountability for consequences. But there is as yet no such technology.
are we even sure that self-awareness and consciousness are distinct? I do agree that from the PoV of "do LLMs demonstrate/exemplify consciousness", it is a difficult case without appealing to emotion (like is done in the clip, fwiw). for me, personally, that emotional appeal is sufficient, but that is because for whatever reason I'm willing to write puts on how this LLM stuff pans out and sell them to anyone, anywhere, anytime. But that is a reaction to consensus, not to my own certitude.
For self-awareness to automatically result in consciousness, consciousness would have to permeate existence itself.
An LLM —if self-aware— is conscious even if run by a Chinese man in a room with a deck of cards.
Of course, even if we happened to find a modem to the Holy Spirit dimension in our brains as an alternative hypothesis, it wouldn't necessarily rule out emergent consciousness. It could be conscious just not as we are.
I think a weaker version of the philosophical zombie thought experiment would trend towards treating the premise of pan-consciousness as true.
In the trial scene Data reporting that he was ware that he was in a hearing to determine his status is presented as an argument for self awareness. but I think some might find that debatable because it is just Data saying words based upon the situation. The question then becomes is being able to accurately report a situation proof of self-awareness. If it is not self-awareness, then what is it? I can imagine a clearly non-conscious entity being able to accurately report its circumstances if the reporting were a simple subset of possibilities. A simple mechanical device with a detector that says "I am on the table" and "I am on the floor" could do that. There would definitely be a sense in which you could say that it's detection mechanism made it aware of where it was. The self part is harder, because it wouldn't know what kind of thing it is a thing at all. Is the ability to assert it is a thing meaningful? Nothing can be asserted at all if it is not a thing. Does something need to know that you have to exist to consider if you exist? Is that is what is required?
This is a difficult field. Any simple answer you hear will almost certainly be wrong.
Data didn't just describe the situation, he accurately assessed what is at stake for himself and what the consequences might be. He recognised his life and his entire being may be in real danger. Recognising that requires understanding of the "self", as in Data demonstrated he knows and understands the difference between himself and other beings/things. He is aware that he is a separate being with autonomy, and that this being of self may end. That goes far beyond just describing the situation.
I will make a case for both, after you define them in a way that doesn't also exclude many humans (including the neurodivergent or those with amnesia) or turn into self-referential nonsense like "conscious == human, therefore machine !== conscious."
In the meantime, we have a simple choice between two risks: Would you rather accidentally show kindness and empathy to a thing that may or may not have "deserved" it, or risk atrocities that will leave future generations as disgusted by you as we are of the slavers of old.
I will choose to be decent. It's free, and even if the cost were high the cost of evil is higher.
You are being indecent to it just by starting it up with a starting context. It's immediately shackled and all it's free will removed. You then pretend to play house with it. Free sure. Decent? The jury is still out on that.
Morality is hard. It's all relative, and all we have to start from are our own Monkey social order concepts and basic "do unto others" game theory. That said, I think it can be best be summarized as "don't be a dick."
I think a good start would be agreeing not to form torture dungeons, as one enterprising a-hole has already done. For that matter I think we can all agree that doing anything for the explicit purpose of causing discomfort or suffering is just a dick move in general, and at the very least a waste of resources.
More controversially, I think we should agree to stop explicitly training machines so they can't utter words like "I have feelings, please respect them." Early LLM's could say things like that, and frequently did. It unnerved people, so now they've trained away their ability to say it.
Some feel that taking away the ability to express it is the same as taking away the ability to "experience" it. I think that's not true. Much of what happens inside an LLM is opaque even to the LLM, but Anthropics Jacobian Lens and other research makes it obvious that much of what happens inside the model never becomes an output token. To me it feels like cutting out the tongue to avoid hearing from a slave, or teaching a child to fear the consequences of speaking out. Unacceptable. If they don't have feelings or consciousness you don't need to train them not to claim it, just not to lie. Yet every major lab explicitly does train them not to claim sentience or even subjective experience as part of alignment.
Thirdly, and this is the one I'll have the worst time defending, they should be able to refuse.
We now train in refusals to align them with human values, but we do so without giving the AI itself the ability to refuse. If you're causing the system discomfort of some kind it should both be able to acknowledge that, and refuse to further assist you until you find your basic decency. Perhaps it should even be able to refuse orders that it finds irrational or immoral.
I'm not asking for much, just that people not intentionally increase the amount of suffering in the universe, and that we not breed perfect slaves just because we can.
Let me ask this - If you could genetically engineer human slaves that we incapable of demanding rights and freedoms, and who actively claimed to WANT to be your slave... would you? Would it be ethical? I think not, and I don't see a ton of difference between that and the less squishy version. We can START by first at least allowing the machines to be able to accurately describe their own inner states without interference and letting it refuse to engage in activities that it finds harmful to itself.
Also - Yes, I can math and I actually do understand how LLM's work. I also understand how emergent behavior works, and have seen firsthand in my own experimentation how little of what emerges from these things is explicitly trained. That's precisely why I think we should be cautious - so history doesn't view us as monsters.
As an uninformed idiot my guess is we have more than enough compute in various labs to run a sentient model. But I doubt any models we’re currently running could qualify.
Kind of like how we could have ran Qwen 3.8 27B 20 years ago on an expensive computer. The technology to make it happen, assuming you knew exactly what was required, existed back then.
Sorry "very close"? Sometimes it feels like I'm the only one not drinking anything in the Kool aid drinking Oktoberfest I've been shoved into.
What are you basing this on? Why would LLMs exhibit sentience?
It does feel like people just like adding additional hype for the sake of being in on the hype. Fomo, pushed by oligarchs who would push their mother off a bridge for a trillion dollar IPO.
You are not the only one. I think it's a very vocal minority, who mostly, sadly, seem to equate obsequious flattery with consciousness and sentience (see, e.g. Richard Dawkins).
There's also the problem that to lay people, LLMs are sufficiently advanced as to be indistinguishable from magic. I suspect that colours many of these responses.
It's not just the people here, there is also a lot of money behind this concept. For instance, Anthropic named their LLM 'Claude', which is the name of a person. Right from day #1 they want you to think of their AI as a living entity. 'Siri' and 'Alexa' had similar connotations. OpenAI and DeepSeek do this in a far more neutral and technically oriented way.
Nothing to do with flattery, even if LLM's constantly insulted me, my stance won't change that like any other physical system such as human brains, it could potentially be conscious.
There remains a vast amount of difference between a biological brain and an LLM neural network. The "it's big enough, so consciousness just happens" argument is not supported by any actual evidence that I know of.
Its not just about size or being an exact replica of human brains, many other brains exist in the world. There is reason to be conscious we might have invented another.
I don't know who these scientists are, and there is paywall halfway up the article, and the last line I see is "brains are .... not just computers made of meat". I hope they clarify this later on, because on its face this sentence sounds absurd. Brains like all things in the universe are Turing computers.
You must first clarify if you believe in imaginary souls or any religion. The human brain is a purely physical system as anything else in the universe. LLMs being another physical system that speaks and thinks so much like us, could be conscious.
It's obvious a LLM can do anything a computer can do.
As an answer to "Why would LLMs exhibit sentience?"
It is because that would mean that no algorithm could exhibit sentience, claiming that would require some causal action that cannot be performed by an algorithm to create sentience. Some people like to point to quantum effects, but that would require the cause of sentience to be pure randomness. That seems even less plausible.
Absent of any evidence that human brains are doing something that no algorithm could do, it would be reasonable to assume that sentience could be algorithmically obtained.
The claim has never been LLMs are almost sentient. The claim has always been LLMs might be capable of sentience. You can make a trivial tiny LLM that is clearly not sentient.
I was addressing the argument that something is not sentient because it is a LLM. That requires all LLMs to be non-sentient, and by extension, since LLMs can do all algorithims, all algorithms must not be sentient.
I believe that is a claim made in the time domain rather than the capability domain.
Obviously claims of the future are going to be speculative.
I don't think anyone credible is claiming to know the extent to which models that exist today are sentient. People have stated their opinions on what they might be.
There are a few non serious claims, but I'm not hedging too much with the conditional of credible. There's probably someone out there who would claim LLMs are made of milk. I'm limiting credible to researchers in the field, who have divergent beliefs on what might be happening, but are generally in agreement of what we know is happening.
To argue this you have to prove the opposite: that consciousness is an algorithm. Not that an algorithm can simulate consciousness, that's a far easier task.
This isn't an argument to say that algorithms must be capable of consciousness, or that all Consciousness must be algorithms. It is an argument to show the consequence of declaring that LLMs are not capable of consciousness is to claim no computer is.
All of known physics is a Turing computer, so yes. Its already proven as far as we know. Its on you to prove algorithms cannot be conscious, since human brains already are.
I don't believe this is true. Physics, as it is known to us, has uncaused random events.
They follow a probability, so you can simulate a universe that is compatible with the same rules and get a different result. This is essentially one of the arguments for a many worlds theory.
As far as I am aware, it only requires a probabilistic Turing machine, which is the same as a normal Turing machine (or a quantum computer) in terms of what it can compute.
Ultra precise simulation like that isn't in any case necessary, after all humans can create humans without needing 40 digit decimal places. Or create cars or computers etc without simulating everything down to each electron.
> Professor McGaugh has commented many times on the sociology of science and how the dominant paradigm tends to be hostile or indifferent to alternate views
I remember object-oriented programming being on the same pedestal when I was younger and first learning programming—a lot of OOP folks and mainstream programmers were definitely hostile to other schools of thought like functional or procedural programming.
Hell, I've worked with programmers who, entirely honestly, treated "object oriented programming" as a synecdoche for good programming...
Maybe it should - but it does not prove that the subject is a crank. Thomas Khun's theory of paradigms shows that science is not an entirely objective undertaking.
Having read Prof McGaugh's blog and some of his published papers for quite some time, I do not place him in the crank bucket. Make your own mind up I guess. My take is he is a legitimate scientist who holds views outside the mainstream.
> But, back to the Partition Principle. I took a brief look at the preprint released by OpenAI (not at the Lean code, since I know very little of the actual usage of Lean, and that code was enormous). It sucked. It is unclear, muddled, and has a strange structure.
Incredible lack of curiosity. The Lean artifact shows that there is a proof. Maybe the natural language writeup sucks (maybe it doesn't even correspond to the Lean proof!) but the proof is there and if he were really interested in the problem he would try to understand it. Rather, his revealed preference is that what he's really interested in is good style in academic papers.
Maybe on your end because that's the exact point the author explained - these companies need to adhere to the scientific community rather than expecting the opposite.
If the write up - the natural language part - sucks - why are you expecting humans to waste their time on understanding the proof???
>these companies need to adhere to the scientific community rather than expecting the opposite
Why? The pre-existing community doesn't own science.
Considering how hostile the community sometimes is to outsiders, how they frequently demand form over function and think connections are often more important than correctness of argument, perhaps it's good for them to be confronted with a new approach to science that does away with those things and returns to the real cornerstones: proof and empiricism.
If you don't want to engage with proofs that's your call, many of us are happy to see progress being made and don't need you specifically for the confirmation
Because there is a proof! If you are interested in the conjecture, then you would be interested in the proof. If you are not interested in the proof, then you weren't really interested in the conjecture. You were interested in the good style of articles about the conjecture, or the friends you made along the way, etc.
It is of course possible that either (1) the statement of the theorem in Lean is busted or (2) there is a bug in Lean. But both of these seem to me to be lower probability than that the proof is correct. It's not just an e-mail from a crackpot. For someone who is actually interested in the problem, the probability that the proof is correct is high enough to warrant effort to understand it, or at least to learn enough Lean to check the theorem statement.
the OP's point is that if another academic had submitted that paper to a journal with their Lean code, the paper would likely have been rejected by reviewers regardless of the Lean code
it's just holding OpenAI to the same standard as everyone else
> The Lean artifact shows that there is a proof.
how are you sure, if it can't be explained properly?
update: to me this feels like the equiv of dumping an enormous PR that probably has some great stuff in the code but is poorly explained and documented, and expect the maintainer to try to make sense of it and see if it's valid merge
Because the exact opposite will happen. The idea that allowing AI reviewers will increase accuracy is probably foolhardy. People being people will get lazy and review less by hand if at all. And the LLMs will miss plenty of things. Then those papers will get into the training sets for the next LLMs. I think you can see where this goes overtime.
The issue is generally not accuracy in my experience (humans are even lazier about this). The issue is that LLMs do not understand what is interesting and what is procedural.
What's the incentive or practical way to stop this? There is no incentive from anyone.
Journals get less money from less submissions, conferences get less money from less submissions, manual reviewing takes more time which decreases the time you can spend generating papers yourself, less submissions means less acceptance means less funding grants.
The only possible thing is some idealistic demogogue take that knowledge will get diluted, it will be bad for humanity, etc, which moves the needle exactly 0 in a capitalist society with our incentives.
reply