Yes, exactly. And this is why I don't think that the LLMs can make significant process beyond what humans have done and published.
"But the math proofs," people will say. A lot of those seem to be spam-solving things with a huge swath of existing lemmas, and a some of these are being debunked and retracted.
Just today I was quizzing ChatGPT about a basic grammar question for a language that has huge training data but for which the grammar was not well documented. It kept giving me confidently wrong answers until I drilled and drilled it and then finally it found/gave back an explanation that perfectly fit a pattern given in one particular grammar, citing that as a source. It doesn't appear to have been able to figure out the inner structure on it's own. It appears only able to pattern match and put things together from what humans have already discovered and written.
Right, there's a difference between statistical interpolation and semantic induction. The whole point is that LLMs can't reason from first principles to drive missing rules. It keeps confidently feeding you approximations until it collides with some source that already mapped it.
Interesting take, given that the whole reason LLMs are interesting is that they're the first system we have that can work in semantic space. Statistical interpolation, that we've solved long ago.
I think that's equivocating on the word semantic. Word embeddings map concepts like "king - man + woman = queen" or cluster synonyms together in high-dimensional vector space and the ML literature very loosely calls this "semantic space." But a high-dimensional topology of token co-occurrences isn't semantics in the sense of computation or formal semantics. It's still "just" measuring distributional similarity. Some vector that represents "thread deadlock" lives near tokens like "mutex" and "race condition" and "starvation," but the model itself has no concept of concurrency and contention.
Claiming we "solved" statistical interpolation long ago just means curve-fitting and basic regressions on structured data. Transformers are a truly impressive achievement, scaling all of this to unstructured high-dimensional text topologies, but it's fundamentally the same math operations on statistical proximity.
Like how do we explain hallucinations here? Tokens that are hallucinated are semantically "close" in that vector space but they're completely false in reality. If LLMs operated in a true semantic space they wouldn't hallucinate CLI flags that don't exist.
> Some vector that represents "thread deadlock" lives near tokens like "mutex" and "race condition" and "starvation," but the model itself has no concept of concurrency and contention.
I propose that concepts of "concurrency" and "contention" are themselves vector in latent space. All concepts are. Recall that we're talking about a 10^4 - 10^5 dimensional space. You can fit in pretty much any conceivable association as some direction in there.
And try to zoom in on any concept you know. If you do, it should quickly become apparent that there's never any concept you can give a closed definition for. We can only define concepts, and we can only learn them, through generalizing from examples. Which is conceptually (pun not intended) regression - finding a vector along which examples live.
"But the math proofs," people will say. A lot of those seem to be spam-solving things with a huge swath of existing lemmas, and a some of these are being debunked and retracted.
Just today I was quizzing ChatGPT about a basic grammar question for a language that has huge training data but for which the grammar was not well documented. It kept giving me confidently wrong answers until I drilled and drilled it and then finally it found/gave back an explanation that perfectly fit a pattern given in one particular grammar, citing that as a source. It doesn't appear to have been able to figure out the inner structure on it's own. It appears only able to pattern match and put things together from what humans have already discovered and written.