Right if it was literally the processing they make the claim about and not the molecular ingredients. Then they basically have a result that points to there being something fundamentally wrong with our model of physics.
You cant have a non mediated cause according to our current model of physics.
Dont be fooled. Reasoning does not happen in the prediction of the next token, it happens virtually in the text that is created.
The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.
All these "AI can't" arguments seem to secretly rely on the assumption that human reasoning/sentience/whatnot is dependent on a soul (or any equivalent metaphysical entity). Everything else is handwaving.
I agree - many are. But there's another camp to watch - LeCun/etc don't believe in a soul but also don't believe that LLMs are capable, because they don't contain the right neural network.
I'd argue that they're still intuitively trying to keep intelligence in the gaps, as humans like to do. Or maybe they're mad they bet on the wrong horse... or maybe both. Hard to say.
> because they don't contain the right neural network.
I'm totally onboard with that as a plausible argument, but I think the architectural limitations with respect to consciousness are very deliberate rather than some lack of technology. We've invested huge sums of money and human effort to create tools with explicit goals that are completely at odds with consciousness or AGI. If we had invested similarly with the clear goal of creating something with agency, self-determination, neuroplasticity, etc. instead of controllability, repeatability, reliability, I think we would be there already.
This is trivially false, the text gets transformed into activations for the weights. If there is reasoning it's in the connection pattern of the weights.
The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token
When i look at the things said by LLMs i clearly see stuff that i would call reasoning.
Its reasoning does however have certain "bugs" that a humans reasoning would never have.
My guess is that much of those bugs appear cause the reasoning that a LLM does is not itself built on a self stabilised system. In humans you get coupling between levels of self stability which acts as a constraint, stabilising the system even more. The next level being predictive coding modelling the world. There is no next level in a LLM; they are trained as refiners in teacher forcing mode, a paradigm where self stabilisation is not a driving factor.
Clearly.. this is not some new revelation, it has been the default assumption for a very long time, probably over a hundred years.
Damage to the hind brain results in behavioural changes that are blatantly different compared to when the "newer" parts of the brain is damaged. The cerebellum literally even has a different color.
Biological minds in biological organisms are self referential in the way that you have a neural network that forms a model of the world. That model then "discovers" that it is "it self a part of the world" so it tries to model that part of the world (model it self). In this way some type of self referential "awareness" (or whatever you want to call it) is formed. That self model that contains awareness is then used to guide organisms behaviour. Causal Transformer LLMs don't work in this way, they have theoretical knowledge that they exist but its selfhood is not in this described way built on the self modelling that biological brains do.
All this being an empirically unproven theory/hypothesis. But an extremely strong one (if you ask me).
Alan Watts says, "Mind finds itself in a strange position. It (ie universe) is in me (modelling wise). I am in it (physically)." But the model only has access to its modelling of both it and itself.
(1) LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time.
This is a consequence of how they are trained.
> LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time.
LLMs are likely to get stuck even with sampling if asked to generate tokens on their own long enough, though sampling does tend to stretch out the time before that happens (as do other techniques that don't involve sampling, like applying repetition penalties directly to token logits). But LLMs generally aren't left to infinitely extend their own output, and the length response typically needed in the use case is much shorter than the would result in collapse given the kinds of inputs expected in that use case, the existence of the theoretical eventuality may not really matter.
You know you can just try it and see on any inference system thst has this knob, right?
Related: if you don't have a limit on sampling (top-K or top-P), eventually you'll hit one of the really unlikely tokens by chance and then the model will switch to Japanese because the most likely completion after a random Japanese character in the middle of an English sentence is more Japanese writing, not a reversal back to English.
That could be how it works, but in practice it takes into account all previous tokens when producing the next-token distribution to sample from. So a switch back is more likely than your explanation supposes.
No, if you switched to Japanese the LLM wouldn't ignore it, it would "assume" there's a reason for that. The same if the previous iteration of the LLM switched to Japanese. Else you're expecting an LLM to ignore its own previous outputs and restart "thinking" from scratch with every token?
It's situational and I suspect there are situations where it would and others where it wouldn't. Would depend on the almost infinite variables of how training was done. You'd be right that it'd be likely to switch but while it's possible it's due to temperature, there are just so many things going on. But it would be one sensible explanation among many.
Interesting topic. That said I don't know how useful this is since LLMs are primarily trained using mode-covering training rather than Mode-seeking(RL) training, which means LLMs can not form (and does not have) the same underlying structure to their models of language that humans have.
A LLM does not learn topic by topic, it learns everything all at once and slowly integrates it in to a single knowledge system.
I thought it was sorting an infinite set of infinite strings as the first step in the "algorithm" that seemed sketchy. [0]
It's definitely not a constructive proof, even though it pretends to be; none of the mathematical objects can be constructed, nor can any of the algorithmic steps be executed.
That said, it's far less well-known that the workaround (if you find it to be true) is trivially easy (from Alfred Tarski), making it kind of a useless theorem in practice.
[0] You might think, well, I'll just write out the strings in order by using a generator! No sorting needed... But you have to write the strings down to perform the algorithm, and it takes infinite time to write down the first string, so you'll never even get to the others which is when you do the diagonalization trick. Like I said: it's not constructive, none of it can actually be done.
They work because you are infinity percent more likely to buy something you know exists vs something you don't know exists.