> they found bits of code that are associated with "math" and the supplied arguments
How is this different from a human using an algorithm they have memorized, or reading it from a reference site written by a human and then writing the same formula into a custom one off piece of python code?
I could have gone and spent a couple of days teaching myself the math behind Karney and reading its reference implementation (very possibly just copy/pasting big chunks of it to save time) and writing a wrapper around it. It would have produced the same result.
>> they found bits of code that are associated with "math" and the supplied arguments
> How is this different from a human using an algorithm they have memorized, or reading it from a reference site written by a human and then writing the same formula into a custom one off piece of python code?
Humans identify which "algorithm they have memorized" to use beforehand, due to the problem to be solved being defined by other humans, which leads to...
This doesn't make any sense at all. Was this supposed to be a gotcha? An LLM is trained on problems defined by other humans, and identifies which algorithm it must use based on pattern recognition. The pattern recognition is also particularly compressed into its most sparse and fundamental components, as this is key to generalization. This is not a sensible difference between human and LLM learning, we do the same thing.
I'll give you a recent example from my usage. Pi harness with extension for learning Chinese. When using it to feed drill questions to me and rate answers it would sometimes get lost in the sauce and start generating user aka me answer and then rate it and comment it. It's trivially wrong to the point that if a person would do that, they would be considered for some serious psych issues.
And it gets even better since when called out it wouldn't just take my word for it but only acknowledged the issue after parsing the log with clearly delineated user and model output.
So yeah while impressive things are able to be done, the current models are also dumb AF and an idiot savant is a pretty good label for them.
This is often (but not always, as it is often run with positive temperature which deliberately shifts the generation path) because of disconnected context and the issues around context compaction. Long context has always been critical to important work, but it remains a substantial challenge due to computational bottlenecks. It isn't a fundamental issue with the architecture, moreso the tricks to make them cheaper to use. In the long run, taking various concessions to make them cheaper to run is likely to be the limiting factor that prevents this "runaway explosion" that many people are worried about. That doesn't mean that the model design is inherently limited, like a lot of people seem to assume based on current failings.
Besides, have you spoken to someone lately that everyone would calls a genius? They can say the most braindead stuff sometimes. I wouldn't use worst-case performance as an indication of general capacity.
>>> How is this different from a human using an algorithm they have memorized ...
>> Humans identify which "algorithm they have memorized" to use beforehand, due to the problem to be solved being defined by other humans ...
> This doesn't make any sense at all. Was this supposed to be a gotcha?
No, it was meant to be an explanation as to the difference between "memorization" and "understanding." In this context, people pick the algorithm they determine applicable and then the question of memorization is relevant.
> An LLM is trained on problems defined by other humans, and identifies which algorithm it must use based on pattern recognition.
Funny that you make this argument here, where when I wrote elsewhere in this thread:
[LLMs] are statistical token generators whose results are
dependent upon their training data set and involve a
degree of randomness.
Nothing more.
...
It is pattern recognition, a task in which ANNs excel.
To which you replied to the above with:
During conversation, we are statistical token generators
whose results are dependent upon our training set.
Seriously, write that definition out rigorously. It
encompasses virtually everything. It is totally
meaningless. So to say "nothing more" is effectively also a
tautology.
This argument was asinine in 2024. It is insane to be
saying these things in 2026. Where have you been?
...
It absolutely understands how to do math, by whatever
reasonable definition you want to provide to the word
"understand".
So which is it?
Are LLMs ANNs? Which themselves are pattern recognition algorithms (hint: they are)?
OR (setting aside the ad hominems you kindly provided)
Do LLMs possess "understanding" of concepts such as abstract mathematics (defined and interpreted by humans) and we, as simple humans, nothing more than statistical token generators as you assert?
It is both. I do not understand why you would assert that both cannot hold simultaneously. Pattern recognition becomes "understanding" once individually recognized concepts become sufficiently sparsified and compactified. Or, at least, that is to my knowledge the only mathematically valid definition of "understanding" one can produce at this scale (it is valid under Solomonoff induction). Philosophy is fine, but we need to have a consistent definition of what "understanding" means, or we will just talk past each other. I argue that for any proper definition you provide which humans satisfy, a strong LLM is very likely to satisfy that as well.
I also would not argue that humans are "simple token generators". That is not what I said. I said that just about everything can fall under the classification of "statistical token generators" at an abstract level, so it isn't a useful distinction. We are not talking about a Markov chain generator from the 90s, so if that is the frame of reference, I think we should all get that out of our heads.
> Pattern recognition becomes "understanding" once individually recognized concepts become sufficiently sparsified [sic] and compactified [sic].
Understanding is a state of mind. As such, it exists entirely within an individual and nowhere else.
For example, take any two university professors who teach the same subject where one only speaks Arabic and the other only speaks Vietnamese. Each will not be able to understand what the other says, regardless their understanding of the shared topic.
> I argue that for any proper definition [of understanding] you provide which humans satisfy, a strong LLM is very likely to satisfy that as well.
This is demonstrably incorrect as detailed above. There is no "understanding" LLMs can satisfy as we know it, since to certify said "understanding", it requires interpretation by a person to "know" an LLM "understands."
> I also would not argue that humans are "simple token generators". That is not what I said.
That is the essence of what you wrote, unless you object to my use of "simple" instead of "statistical". In this context, I postulate this is a distinction without difference.
> I said that just about everything can fall under the classification of "statistical token generators" at an abstract level, so it isn't a useful distinction.
This only holds if one subscribes to statistical token generators being a/the fundamental underpinning of "everything". Here is a proof by contradiction:
If everything can be classified as a derivative of
statistical token generation, how does one explain
quantum physics?
We cannot engage in an intellectual discussion if we do not agree on definitions. So far, your definitions of understanding seem to be whatever vibe you are going for in the statement and I would urge you to think about what a sensible mathematical definition of understanding is so that it can sensibly be assessed on neural networks. Otherwise, this isn't science, it's a debate about personal experience.
> Understanding is a state of mind
This is meaningless, it is a circular definition at best.
> It exists entirely within the individual and nowhere else
Then why are we talking about it? What is the point if it is something that can only be defined per individual?
Regarding your language example; this is a case of missing vocabulary (excluding grammar of course, but I feel like that is second-order), which is not the same as conceptual understanding. We are often able to translate because we have shared concepts. Those concepts are what we really care to assess with LLMs.
> requires interpretation by a person to "know" an LLM "understands."
We are still not getting anywhere because you have not prescribed criteria to determine whether it understands. If it is a "know it when I see it" situation, that clearly isn't working. For example, if you say that you need to dig into its internals and figure out whether it is breaking things down appropriately, that doesn't work because you probably don't have the expertise to do that. The experts that do are telling you that it very likely understands because it pulls apart most concepts in the way we would expect.
I do object to the use of the word "simple". "Statistical" is so broad to be almost meaningless; it merely means that a prediction is being made in the presence of data which possibly contains some degree of uncertainty. "Simple" encompasses that which can be understood readily by a non-expert.
Quantum mechanics is statistical (this is literally the Born rule), but evolutions are not operating as stochastic processes in the sense of Kolmogorov. That is very different, and not relevant to our discussion.
> So far, your definitions of understanding seem to be whatever vibe you are going for in the statement and I would urge you to think about what a sensible mathematical definition of understanding is so that it can sensibly be assessed on neural networks.
Any reasonable definition of understanding is not dependent upon "whatever vibe you are going for", but instead must include at least an English dictionary definition of "understand" such as:
to grasp the meaning of[0]
And, for further clarification, "grasp" can be defined as:
to lay hold of with the mind[1]
Which makes an equivalent term-expanded definition of "understand" to be:
to lay hold of with the mind the meaning of
As such, there is no "sensible mathematical definition of understanding", unless you possess a complete mathematical model of the human mind.
>> Understanding is a state of mind
> This is meaningless, it is a circular definition at best.
See above to as to why there is meaning in what I wrote.
>> It exists entirely within the individual and nowhere else
> Then why are we talking about it? What is the point if it is something that can only be defined per individual?
I like to think analyzing fundamental premises, often implicit, explicitly can help to identify fallacious positions.
> We are still not getting anywhere because you have not prescribed criteria to determine whether [an LLM] understands.
My apologies for being opaque. Let me clarify:
LLMs do not "understand". People interpreting LLM output
are the only entities involved which can "understand",
because "understanding" exists strictly within each
person who possesses it.
>Understanding is a state of mind. As such, it exists entirely within an individual and nowhere else.
For example, take any two university professors who teach the same subject where one only speaks Arabic and the other only speaks Vietnamese. Each will not be able to understand what the other says, regardless their understanding of the shared topic.
I wrote a program that statistically generated tokens to consistently factor latge RSA numbers. Does this program actually factor numbers or is it just a statistical next token predictor?
Frankly, I don't think you understand what "statistically generating tokens" actually means. Write that definition out formally. Then compare that operation to what a human does, assuming no revisions. It is the same, and that is my point. If you believe that humans understand, then "statistically generating tokens" cannot be disjoint from understanding.
You have observed nothing more than that a human can turn a shaft the same as an electric motor, and that an mp3 player can say "hello" the same as a human.
I think you're moving back to the "external calling" argument again, which my last few comments really have nothing to do with. But assuming your examples are not exclusive to that situation, that observation is my point. The definition of "statistically generating tokens" is too broad as to be meaningless in this context. So using it as a reason for lack of understanding is ridiculous.
If by "result" you mean the final code, then just asking someone else who understood the math to write it would also have achieved the same result.
On the other hand, if by "result" you mean that you gained knowledge or understanding of the code in a way where you could personally tailor its behavior to specific circumstances without asking for help, then it's not the same result at all.
I find a lot of the arguments that having LLMs write your code is no different from copy/pasting Stack Overflow answers to be specious. They blur the line between asking for help and asking for someone else (or something else) to do the work for you. What they ignore is that doing the work yourself has ancillary benefits and is a valuable end in its own right.
> How is this different from a human using an algorithm they have memorized, or reading it from a reference site written by a human and then writing the same formula into a custom one off piece of python code?
And how is _that_ different from making the human memorize a billion weights and do matrix calculations in their head, in order to generate tokens?
How is _that_ different from a hive of bees trained to do the same?
Everything is modelable by a turing computer as far as we know. So yes, I don't see why these processes can't do the same, its on you to show why such a process implemented in any of these isn't a form of thought.
How is this different from a human using an algorithm they have memorized, or reading it from a reference site written by a human and then writing the same formula into a custom one off piece of python code?
I could have gone and spent a couple of days teaching myself the math behind Karney and reading its reference implementation (very possibly just copy/pasting big chunks of it to save time) and writing a wrapper around it. It would have produced the same result.