“A soul”?

As popular as I know the rhetorical tactic is on both sides of these discussions about LLMs, I’d still thank you not to strawman me.

Don’t even bother. These people almost always have some goofy ass, non standard, fluid definition of “thinking” or “reasoning” that cannot ever be met.

My opinion of their reasoning capability is based in part on (proprietary, non-published, only internally peer reviewed) experiments on GPT-series models’ ability to perform a suite of formal and informal inference and deduction tasks.

Perhaps you could argue that “appropriately applies syllogism to arrive at correct conclusions” is too high a bar to set, but I don’t think it would be fair to call it a “goofy-ass”, “non-standard” or “fluid” element of a reasoning capacity assessment.

Yeah I guess the Jacobian conjecture must have been disproved without reasoning.

It’s hard to say. But supposedly the counter example wasn’t found by an agent running in full auto; it came out of a bunch of back and forth with a human operator. Without, in addition to the aforementioned access to currently non-public information about these models, a detailed transcript of the chat sessions leading up to the discovery, it’s hard to ascribe the reasoning steps involved to any source in particular.

Part of my concern here is that simply pointing out that LLMs appear to be performing tasks that can be done through reasoning, and using that in and of itself as evidence of reasoning, is affirming the consequent.

But you're not just saying they are insufficiently good at reasoning, you're saying they're (probably) not "doing actual reasoning". So we need to know how you are defining "actual reasoning".

I don't think the bar for an actual reasoner can possibly be 'always appropriately applies syllogism to arrive at correct conclusions', because in that case nobody in the world is an actual reasoner. And if your bar were 'sometimes appropriately applies syllogism to arrive at correct conclusions', it's hard to understand why the current generation of AIs doesn't meet it; they are clearly capable of doing so, at least to all outward appearances. (Maybe you think their apparently successful demonstrations of reasoning are illusions, but again, you would need to define what counts as "actual reasoning" vs. a superficially convincing simulation of it.)

Like I said earlier, it’s proprietary work and I’m not at liberty to discuss in too much detail. So all I can really say in response to your specific criticism is that one throwaway example for the purpose of responding to someone else’s attempt at denigrating me is not the whole of the thing.

More generally, it’s important to keep the burden of proof in the right place. The idea that this particular neural network architecture is performing reasoning tasks is really quite extraordinary, and we should demand an unusually compelling case to be made before accepting it.

Compare to the example of Douglas Hofstadter and computer chess. In the late 70s he predicted that computers would need general intelligence to compete at the top levels in chess. When computers did start competing at that level, he did not automatically assume chess programs are intelligent. He took a look at how they are implemented, decided that the idea that that particular algorithm was doing abstract reasoning was implausible, and instead re-evaluated his assessment of what it takes to compete with a top tier human at chess.

I tend to align with Hofstadter’s approach to the question. Like I pointed at in a cousin comment, arguments that what we see represents reasoning tend to hinge on a common logical fallacy.

>More generally, it’s important to keep the burden of proof in the right place.

The extraordinary claim here is that what looks like a duck, quacks like a duck, acts like a duck is in fact not a duck even though you are simultaneously completely incapable of proving it is not a duck.

Just because you have placed humans on a pedestal does not mean you are not the one making the extraordinary claim. Hofstadter is entitled to his opinion and while you are free to follow his 'approach', you'd do well to consider that neither you nor Hofstadter in fact have any idea whatsoever how 'abstract reasoning' is implemented in even the systems you are certain posses it. Don't be confused that you are doing anything more than begging the question here. You already assumed your conclusion and are working backwards to justify it.

Well, like I was hinting at above, it's actually based on a close analysis of all the ways it doesn't quack like a duck, together with some serious consideration of how plausible I believe it is that a decoder architecture would be capable of performing generic symbolic reasoning tasks. The crux of the problem is that general reasoning involves actively selecting and then manipulating a symbolic model at least semi-independently of language generation, and that's a very different shape of information flow from next token prediction, even with MHA.

I would certainly be open to hearing a detailed explanation of how that kind of thing might actually support reasoning, but, absent that, pointing out that I don't understand how brains do it (which is true) is the pot calling the kettle black.

I actually want to turn the tables on your claim about humans being on a pedestal, and this is another spot where I believe I side with Hofstadter. We humans have a tendency to believe that certain behaviors are so special that the only way to accomplish a similar outwardly-visible behavior is by using similar underlying mechanisms. I see a healthy measure of chauvinism in that assumption.

As I mentioned earlier, these LLMs are trained on vast collections of both the inputs and outputs of genuine human reasoning, and potentially have enough parameters to record a large proportion of that information in the form of probability distributions. And even something as crude as a singular value decomposition on a term-context matrix is enough to produce generalizations that rather spookily resemble human semantic models. Neural models do the same thing much better, but I'm still not prepared to see much in that beyond more evidence in favor of the distributional hypothesis. Which means that there's plenty of room to believe that what seems like reasoning on the part of the LLM is actually more akin to a projection from a sort of "holographic" representation of written artifacts of human reasoning.

That's not placing humans on a pedestal. That's just thinking hard about possible explanations for a phenomenon and then provisionally choosing the one I believe is most parsimonious. As any believer in scientific skepticism should strive to do.

>Well, like I was hinting at above, it's actually based on a close analysis of all the ways it doesn't quack like a duck

I doubt that, if your descriptions are anything to go by. "appropriately applies syllogism to arrive at correct conclusions" is just not something humans reliably do, so if that is the bar here then it's not worth anything, no matter what secret benchmark you think you've cooked up. Will you say humans cannot reason as well?

There's the classic example of the Wason Selection Task. Most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters.

>I would certainly be open to hearing a detailed explanation of how that kind of thing might actually support reasoning, but, absent that, pointing out that I don't understand how brains do it (which is true) is the pot calling the kettle black.

Not really. You're the one making a positive mechanistic claim here: that reasoning requires "actively selecting and manipulating a symbolic model at least semi independently of language generation" and that a decoder transformer therefore probably isn't doing it. The claim needs justification and it is once again another example of you assuming your claim and working backwards towards it.

"Next token prediciton" describes the training objective and the interface through which the model's computation is expressed. It does not tell you what computation the network must perform internally to minimize that objective. There is nothing contradictory about a system learning internal representations, manipulating them, and then using the result to predict the next token.

In fact, we already know transformers are capable of considerably richer internal computation than your description suggests. Much richer than we can even begin to understand.

We don't possess some mechanistic definition of human reasoning which we've opened the human skull and verified. We infer that humans reason largely from behaviour.

So if some other system exhibits some of the same behaviours, you need some non-question-begging criterion for why they cease to count as reasoning in that system. Otherwise the position just becomes unfalsifiable: when the model succeeeds on a novel reasoning problem, that's merely a "projection from a holographic representaion" (this is a meaningless statement by the way); when it fails, then you jump to claiming its evidence that wasn't/cannot reason at all (which is just silly and bad science).

>I actually want to turn the tables on your claim about humans being on a pedestal, and this is another spot where I believe I side with Hofstadter. We humans have a tendency to believe that certain behaviors are so special that the only way to accomplish a similar outwardly-visible behavior is by using similar underlying mechanisms. I see a healthy measure of chauvinism in that assumption.

Nobody is requiring the same mechanism. That's my point. I'm perfectly happy for transformers to reason in a drastically different way from humans (though i suspect it's not so drastic). You are the one requiring that it must have a particular internal form - one that cannot even be validated to exist - and then rejecting systems whose architecture doesn't obviously resemble that form.

>As I mentioned earlier, these LLMs are trained on vast collections of both the inputs and outputs of genuine human reasoning, and potentially have enough parameters to record a large proportion of that information in the form of probability distributions. And even something as crude as a singular value decomposition on a term-context matrix is enough to produce generalizations that rather spookily resemble human semantic models. Neural models do the same thing much better, but I'm still not prepared to see much in that beyond more evidence in favor of the distributional hypothesis. Which means that there's plenty of room to believe that what seems like reasoning on the part of the LLM is actually more akin to a projection from a sort of "holographic" representation of written artifacts of human reasoning.

That doesn't distinguish reasoning frm 'non reasoning'. Learning from artifacts of reasoning is perfectly compatible with learning procedures that reproduce the underlying structure which generated those artifacts.

Appealing to the distributional hypothesis doesnt help much either. The fact that relatively simple statistical methods can recover meaningful latent structure from text shows that structure is present in the data; it does not establish that a much more powerful model trained on the data can do nothing beyond "projecting" stored patterns.

>That's not placing humans on a pedestal. That's just thinking hard about possible explanations for a phenomenon and then provisionally choosing the one I believe is most parsimonious. As any believer in scientific skepticism should strive to do.

Your explanantion isn't more parsimonious simply because you labelled all succesful reasining like behaviour "projection". All you've done is add an unobserved distinction. It's a meaningless semantic game. You call it a projection because you want to believe LLMs don;t reason, not because it's a meaningfull term that adds some helpful falsifiable criteria.