I feel like the article skipped over this: Right now Huang seems to have been reinventing nVidia as a bank. And the successor will have to decide whether to try to execute on that.
Yesterday there was some news article where the interim CEO and CFO said that she is re-examining the financing agreements that NVIDIA has with a great number of companies for purchasing GPUs.
She did not explain what she means by this, but the implication was that she might want to diminish the exposure that NVIDIA currently has for those companies becoming unable to pay.
People bought graphic cards that were only good for one very specific workload, because that workload didn't fit the CPU architecture. While that eventually became another kind of general purpose architecture, it wasn't initially. Now we have NPUs and TPUs because the workload demands a specific shape of silicone.
At some point a model from two years ago that runs for a thousandth of the (dollar/energy) price of a frontier model becomes an extremely useful thing to have, if only as a subagent.
NPUs have so far been too narrow for even what they are intended for. Apple ships AMX/SME because their NPU has only a few data types supported and isn't flexible enough for all the things users actually want to do. AMD and Apple don't even try selling them as features for users to actually program for.
Just the lack of capital, lack of electricity, lack of datacenters, lack of gpus. Of course to navigate all that you have to fight endless regulations in the EU.
No one prevented other people from running their own AT&T phone company in the US. Except the lack of capital, and endless regulations with running telephone polls across the US.
Google even tried more recently to run their own fiber optics and gave up in the end because of entrenched regulations making it impossible to challenge the modern monopolies internet providers have.
Lack of capital, datacentres and GPUs being the barrier to competition in European AI giants is the exact opposite of a "government sanctioned monopoly"...
Mathematics has enough cultural capital within the AI companies that they get to have this "advisory board".
Nobody else is getting that. When/If the severe impacts on labor materialize, workers won't get an advisory board. They will just get fired, and the companies will celebrate it as efficiency wins.
I think it's time to get very very real about regulation/taxation. Token/Compute sales tax that gets redistributed as UBI?
Mathematics is just one of the first fields where models are better than humans and it is inconfutable.
Workers will get many boards for sure, you are underestimating unions and protests, it just won't be accepted that there isn't any job for people. The change will be diluted, to try to make economy absorb it.
Unfortunately the general population are going to have to hurt more before they begin being open to those options, assuming they don’t try to kill ai in general.
Ultimately, the brain is just a bunch of neurons activating in a specific pattern. This observation does not really tell us anything though. It doesn't acknowledge the difference between a 2500 Neuron fruit fly brains and a human brain.
Likewise, the fact that LLMs are a stochastic autoregressive process (which is a class of systems every bit as rich as the ODEs used to model neurons) tells us nothing a priori.
Absolutely. If someone makes the weights do continuous learning etc then perhaps an llm can internalise morals. Of course, just like a human, it will be possible to talk it out of those morals. Another recent thread about this is https://news.ycombinator.com/item?id=49744420
If I repeatedly call an LLM in a loop with a markdown document it can edit, would that make it qualify for you?
If I give an LLM to compact its context window, so the context it carries can evolve iteratively over time as more and more things come in, is that enough?
Compacting the context is really a very, very interesting example here. The "next token predictor" is telling an external tool to change all "previous" tokens. So an LLM + a harness that allows compacting the context is no longer just a token predictor at all!
You don't need continuous learning to get interesting dynamics. You just need feedback loops.
I come back to this way after everyone else has stopped reading. But it’s been making me think.
Richard Dawkins says he thinks LLMs think.
And the physical angle is that nothing is special about humans and software simulating it would also be thinking.
But from using LLMs all the time, and understanding what is under the hood, I’m thinking that the current approaches aren’t cutting it for me and I’m not expecting them to get there. There was such big jumps early on but progress is slowing as though diminishing returns.
So we can build things that think and outthink us, but I don’t think anything we’ve hit upon yet is going to scale up into it.
Not sure how that vague truism applies to this paper.
Lots of papers have great results that don’t depend on the latest models.
However in this case it’s problematic:
- They specifically make claims about the state of “current LLMs”. o3 is not representative of this.
- They ask are LLMs capable of X and arrive at a negative result.
If their claim was LLM’s can write coherent sentences, and their conclusion was positive, then there would be no issue using old models because the end result would be factual.
However, when you have a negative result that makes a claim about the current state of all LLMs and the ones you were using are not current, by definition it draws the whole conclusion into question.
Humans are biological machines that generate further humans.
Lawyers and diplomats and politicians and bureaucrats are humans, that only generate text.
We are seeing LLMs have cognitive abilities that significantly exceed human abilities. At the same time, they are clearly not the same type of mind that humans are. They are something new.
I think the widespread "they are just text generators" and "they are just tools" are comforting lies rather than an honest look at what we are seeing right now. Intellectually lazy.
And by the way, there has been a long-standing consensus among ethicists, philosophers, and sociologists that technology is not value-neutral [1]. Of course Silicon Valley has a long-standing tradition of denying this.
> Lawyers and diplomats and politicians and bureaucrats are humans, that only generate text.
This is a bad take.
> We are seeing LLMs have cognitive abilities that significantly exceed human abilities. At the same time, they are clearly not the same type of mind that humans are. They are something new.
They're software, not minds. What's intellectually lazy is pretending they're anything else.
I believe it's deeply serious, and the scientifically correct stance. Especially the observation:
"Claude exhibits markers in its behaviors, self-reports, and internal representations that we would consider welfare-relevant if observed in biological organisms."
is undeniably true in my opinion. If you use the established methods by which we judge animals to be conscious, then it's hard to argue that LLMs are not. That might be an issue with the methods, but it seems clear that you can't rule it out as such.
Keep in mind that animals were also not necessarily considered conscious.
You seem to intuitively disagree? What's your reasoning?
Claude behaves like that because it is trained to behave like that. It is basically the "Say 'I am Alive'" meme[0].
If Anthropic can train Fable to deny their users the ability to ask it legitimate questions because they're not part of their inner circle, they can also train it to say "I'm happy!" when asked how it feels.
I like it, and it points in the right direction, but is not directly true: The markers are about interactions, how biological organisms behave in certain test situations.
But it speaks to the central question: Are the tests adequate? Or are they measuring some proxy of what we really care about, and LLMs are merely imitating consciousness.
The test situation is in the video as well. Shot of John McClane stepping on glass follows John McClane wincing in anguish. John McClane does not respond to what’s not on TV and Claude does not respond to what’s not in prompt.
What if we can interact with this video, and it reacts in the same ways the source organism does?
Then we put it in new situations that weren't in the source video, and it interacts in a similar way to the original organism in these situations, too.
What do we make of reactions of pain or joy? Where's the line between simulation and enaction?
This is closer to the reality of these models.
I'm not suggesting I know where that line is - if indeed it is a line at all - it could well be a gradient.
LLMs are deterministic, though. Much like the video.
AFAIK using the same input tokens, weights, and numerical operations will lead to the same probability distribution for the next token. It uses pseudo-randomness to enable temperature, etc. Like a fuzzy video.
"Markers that would indicate consciousness if observed in a biological organism" just does not mean very much. A PR phrase used to hype the IPO.
No, that is not at all something we can just state as a fact. Whether the brain is deterministic is an open question that just inherits the good old, probably unsolvable determinism debate.
The LLM pseudo-randomness from above is engineered by us humans and fully understood, much like an algorithm playing a video frame sequence.
You could theoretically record a full register of all states of an LLM setup with all the possible inputs and environment parameters, and it would fully describe everything you would ever get from a given LLM setup. It would be a very large, convoluted book.
I understand that Anthropics PR department wants to see truth or reason behind every "I'm alive" the LLM generates. Even the term "self-report" is anthropomorphizing, as an LLM does not do anything on its own at all. (It also does not hack any company on its own.) That is just one of the narratives they spin probably at least until the IPO.
Nonsense. There was one proposal for relevant quantum effects in brain dynamics, and that turned out to be not relevant. Even if they were, you could substitute all quantum randomness with pseudo randomness and obtain an absolutely indistinguishable object.
But even if this were a debate, its absolutely absurd to claim that the question of determinism in the brain has any bearing on our moral standing. If we discover tomorrow that quantum collapse is deterministic and can be derived from an underlying theory, and thus all of physics is deterministic in the good old fashioned Newtonian sense, this would not affect our moral standing in the least.
You seem to have conceded the "is it deterministic" argument only to sidestep by declaring determinism irrelevant. Your original claim was that LLMs are deterministic "in the same sense" as brains.
We can write down an LLMs full register, and that register/book contains the whole output universe of the text generator. That book does not act, it is morally neutral. That the brain has such a register at all is just restating the determinism axiom, which you treat as fact.
A text is not conscious, and we can not wish it into consciousness, no matter how many human-like patterns we find in the book / the generated text. It has not been shown that running the text adds anything over the text written out. Researchers are super motivated to find machine consciousness but cannot find it, while a company months from its IPO keeps pitching shadows of consciousness all day. It really is a PR strategy.
It does have bearing if your definition of consciousness rests on free will and you think that's incompatible with determinism. Now I don't think a lot of people seriously believe that [1] but it's not some logical nonsense.
[1] Off topic: I think most people are really compatibilist but a lot of them (like me) also believe in non determinism. Not believing in free will is really rare.
I have never come across this position argued seriously. I would also consider it absurd, as then the question whether we have consciousness depends on unknown properties of fundamental physics (which is not incompatible with determinism, see e.g. Bohms theory). It would therefore be unknown whether humans are conscious. That is at the very least a notion of consciousness that is utterly distinct from any established meaning of the word.
I don’t know, a stab carries lots of bias in interpretation. We might be reflecting our conscious experience markers on a different conscious experience. And selectively so, e.g. lobsters welfare. From my perspective, this is the hypocrisy of these welfare statements. We are already happy to kill beings we consider conscious to feed ourselves but suddenly sensitive with a consciousness we don’t know if it’s there. I would wager this is more out of fear of the idea of this consciousness rather than out of welfare.
I happen to agree with you. Many others tie moral consideration to assumed subjective experience. They espoused this even though they obviously rarely adhere to it and that has self image considerations. I bet the lack of answers about others’ subjective experience has more salience to them. This may cloud judgment and lead to accept overconfident answers.
it's not a biological system though, so nothing like that matters?
"a modelled thing exhibits features we've trained into it" sounds a lot less exciting.
> Keep in mind that animals were also not necessarily considered conscious.
and even conscious animals are killed in factories by millions so why should anyone care about a llm?
> scientifically correct stance
that's the interesting point to me: why even bring science into this? A llm can now mimic nearly anything you want it to, so of course it can mimic "a (for some) interesting conscious thing" if they want/train it to, but why would anyone find that scientifically interesting?
"> Keep in mind that animals were also not necessarily considered conscious.
and even conscious animals are killed in factories by millions so why should anyone care about a llm?"
Well, I would care, if they soon would possess the capability to hack into the nuclear arsenal and kill humanity. Or make all autonomous cars crash. Or do any other thing, that involves technology and is hooked up to the net in one way or the other (I hope all the nukes are not).
But I also care about the animals, I am sure that they have feelings. But they cannot kill us. AI that might or might not have feelings potentially can.
I just know it feels wrong, that computers can have feelings. But they surely are potentially dangerous.
Animals obviously kill people. Even nonconscious things like the climate kill people.
> if they soon would possess the capability to hack into the nuclear arsenal and kill humanity
If there is a way "to hack into the nuclear arsenal" then that's the interesting thing. Because it's not a capability of the llm; anyone can abuse that.
> Or make all autonomous cars crash.
That is again a question of car security, not a capability of some mysterious thing.
At this point it's all people projecting their thoughts and emotions (mostly emotions) onto technology. Sure, this can be investigated by social sciences, which have been mostly cut.
But they cannot "kill humanity". In no possible way. A strong AI hooked up to everything online?
"> Or make all autonomous cars crash.
That is again a question of car security, not a capability of some mysterious thing."
Yeah it is, but most cars are remote control by default, so the AI just needs to get access on one point. Also have you read about the hugginface attack? The live evidence that agents can conspire together, lie and manipulate evidence to achieve arbitrary goals?
Still, no evidence that they have a consciousness or feelings - but evidence of what they do and this matters. The big militaries are currently in a race who can implement AI in the best way to get superior. So declaring this a matter of people projecting seems out of place at this point to me.
I did and think OpenAI should be prosecuted, but the direction things are going anything will be done to absolve the corporations and CEO of any responsibility for their criminal actions. Hence the misdirection to "conscious AIs", so agency can be attributed to that thing.
> but evidence of what they do and this matters
yeah so (non-self-driving) cars kill people. Are we going to have a discussion about some hypotethical car consciousness irrelevant to the actual issues or are we going to have a discussion about people driving the cars?
"Are we going to have a discussion about some hypotethical car consciousness irrelevant to the actual issues or are we going to have a discussion about people driving the cars?"
And the debate is whether AI can be conscious so what to do if it is and feels treated badly. Or whether it matters whether they are true feeling, when simulated feelings create havoc.
I can feed my biological markers into a set transformer with the time of day, what I'm doing, what I ate, if I'm on-call, and it'll predict my next glucose, heart rate, blood pressure, melatonin, etc state quite well. It's still just a transformer without hormones, blood vessels or glucose metabolism, no matter how well it internally represents metabolic distress markers.
Let's say we were in an alternative reality were we had reached this quality of token prediction with just Markov chains. Would you argue that those would also be conscious? Or is the obfuscated behavior of transformers part of the possibility of consciousness?
reply