I guess unpopular opinion but (1) I doubt this person is actually an expert and (2) I think the actual problem is that being anonymous is a coddling blanket that absolves you of the social cost of anti-social behaviors (i.e., acting like a dick) or holding an unpopular view. Obviously anonymity has uses, too, but that is now how it is actually used on the Internet most of the time.
The only reason I bring this up is because I think it is counterproductive to view these people as experts when they are likely not.
Written by Walter Murch?? Like, the guy who is considered maybe the best editor and sound designer of all time, who was nominated for 9 Academy Awards, and won 3??? (Editing perhaps second only to Thelma Schoonmaker.)
Walter Murch has been working in film for more than 50 years, as an editor, sound designer, writer, and director. He edited and sound-designed many films for Francis Coppola (The Godfather Trilogy, The Conversation, Apocalypse Now) and Anthony Minghella (The English Patient, The Talented Mr. Ripley, Cold Mountain). His interest in science can be seen in his editing of Mark Levinson’s film Particle Fever (2013), about the search for the Higgs Boson.
Yes, That Walter Murch.
In The Blink Of An Eye is on the bookshelf just behind the monitor I'm typing this on, with my headphones and BMD Speed Editor slightly blocking my view of it.
Perhaps this will be the push we need to finally, seriously consider the possibility that our formalisms of “efficient” and “efficiently verifiable” algorithms correspond to nothing resembling reality, and P \in NP is actually ill-posed.
This has been marginally debated since the mid 90s, when Impagliazzo published the “5 worlds” survey. Occasionally it is brought up in talks about average case analysis. There is occasionally a workshop about it. But it has never really got the attention it deserved. How many more of these results do we need to take it seriously? (Assuming we do indeed certify it.)
> Perhaps this will be the push we need to finally, seriously consider the possibility that our formalisms of “efficient” and “efficiently verifiable” algorithms correspond to nothing resembling reality,
What do you mean? Do you mean that our heuristics shortcuts of treating P as efficiently solvable (and NP as efficiently verifiable) are not warranted? So far pretty much all algorithms in P that we found have small exponents. That hasn't changed, but it might change in the future.
Or do you mean something else?
Keep in mind that even most instances of many NP hard problems can be solved very efficiently in practice, because most instances ain't hard.
However we know problems that are expected to be 'easier' than NP that are hard to solve in practice. These are pretty much all the problems we build cryptography out of, because essentially these problems are in the intersection of NP and co-NP and that's expected to be smaller than NP. And cryptography has to be hard to break on average to work: a random key needs to be a good key.
> and P \in NP is actually ill-posed.
Oh, that's very simple to answer: P is not an element of NP. The question people care about is whether there's any element in NP that's not in P. In LaTeX notation P \subset NP vs P \subseteq NP.
I'm not sure what you mean by ill-posed? The question is very clear. What are you trying to say?
"Ill-posed" here means that the formalisms we've chosen to represent "efficient" and "efficiently verifiable" algorithms could turn out to not correspond to anything humans would consider meaningful ("we posed the problem poorly"). For most people, a solution to an NP-complete problem that's barely-subexponential would be clear evidence of these formalisms being ill-posed.
There are many ways it might turn out to be ill-posed, this is just one of them. Impagliazzo's "5 worlds" survey is in some sense a catalog of some of these ways, and includes some of the ways you seem to be referencing (approxmations, probabilistic techniques, average-case efficiency; depending what you mean this could lie in the "Heuristica" world).
> "Ill-posed" here means that the formalisms we've chosen to represent "efficient" and "efficiently verifiable" algorithms could turn out to not correspond to anything humans would consider meaningful ("we posed the problem poorly").
OK, that's a reasonable worry and one that practitioners share. Thanks for explaining. Though I don't think ill-posed is a good term for this, the technical meaning of the term points in a different direction.
So far, as far as I can tell this new result gives much of any extra credence to that worry.
It is completely acceptable in math to use "ill-posed" to indicate that something is insufficiently specified, improperly formulated, or ambiguous. Math people do this all the time.
You can hold that position if you like but you should do so knowing that the vast majority of people (including math people) would say that a formalism that classifies a technically-subexponential solution to an NP-complete problem as “efficient” is in fact improperly formulated, because the goal was specifically to formalize our intuition of efficiency. Our formulation in that case would have failed to do the single thing we laid out for it. Textbook improper formulation. There is not another way of putting this.
The technical result has nothing to do with "efficient" which according to textbook CS theory roughly defines it to be the set P, the strictly polynomial time algorithms. Has nothing to do with the ETH result which is still an exponential, 2^(n/ log n), just not a pure exponential.
I'm also confused what you are saying down here. Though I can see how "sub-exponential" is easily misleading terminology.
The point is not that ETH has anything to do with P or NP, but that the bounds theoretical CS chooses for its conjectures tend to be extremely round and convenient. Does reality actually prefer such nice, round bounds in general? Does reality prefer them in the case of P and NP?
In essentially all cases, we have no real reason to believe or disbelieve. We choose these bounds because we can't do better. It's often the case that we later prove them wrong with some extremely strange result. Here, we have weird bounds for 3-SAT, and a few weeks ago we had one for integer multiplication.
But this has also happened repeatedly throughout history. In the 50s Kolmogorov conjectured that n-bit integer multiplication was \Omega(n^2). Karatsuba produced O(n^{1.58}) during Kolmogorov's seminar on the topic. Strassen discovered matrix multiplication is O(n^{2.81}) while trying to prove it is O(n^3). And on and on and on.
Maybe P and NP are well-formulated and they do perfectly capture our intuition of "efficient" and "efficiently verifiable" algorithms. Or maybe they don't. We are probably going to keep finding extremely weird, cheesed-out bounds for famous problems, though. You can choose what you want to believe, but I think it would be more remarkable if the reality of P and NP turned out to be as clean as the bounds indicate.
It is true that CS had to make theoretical concessions. Just as Newton did. We don't criticize Newtonian mechanics (or, I don't know, Galileo?) for not being realistic about friction or idealized balls rolling down ramps. Their anti-empiricism was a starting point for physics.
To wit, the mathematical rationale for P-NP was precisely that that under polytime reduction this is still closed in P. That is a very reasonable starting point for a theory. This is also the reason why it is tripping up a large swath of the comment thread with people asking about subexponentials which do not preserve that closure
The opposite narrative is that in science, these idealizations yield profound intuitions and insights.
Furthermore, how far are you willing to go? Do you think Turing machines are invalid, that Turing-equivalent computation is pointless and unrealistic? So we are never allowed use that theory to convey the essential idea of in principle all computers being fundamentally functionally the same and that anything claimed to be more powerful than that had better have some incredible evidence, lest it be appeal to magic. Computability and complexity have a shared foundation so how much of that foundation are you going to reexamine?
Whereas if one wishes to be like a covert finitist/intuitionist, they ought to be open about that instead of accuse asymptotic complexity (or analgously, Turing computability as above) of trying to "cheese" complexity theory as a research programme. Meanwhile, it is true that P vs NP could have large exponents or constants in P, e.g. O(c*n^100...), literally this is the first thing taught in any complexity course or textbook. Students and practitioners remember this admonition well, maybe they just don't talk about it publicly as much.
One can go even further back. Cantor was marginalized for his ideas about infinities.
The difference with people like Lipton, and Knuth, is that the critique is coming from inside the house. In that they know the theory and the importance of it, but also hope/aspire for some new way to address the open question. They are keenly aware that P / NP is a problematic question, but not for lack of trying or stupidity on the part of other scientists. What is more likely is that the 21st/22nd century math to pose the better question does not exist yet (or may never be accessible to humans, another question well on the minds of many today).
Here's something to ponder. It is often observed that despite being NP-complete, many actual human instances are in fact tractable. E.g. routine airline scheduling depending on SAT solvers. But is this evidence that NP-completeness is a wrong theory or evidence that human-sized problems are a special case that becomes tractable (anthropocentrism), or some other more complicated reason?
Tech corporations have always been enormous stores of (1) capital and (2) people. MSFT's flavor of this has always been "tech Berkshire Hathaway": a holding company for a diversified portfolio of a couple-dozen billion-dollar businesses. MSFT in particular is willing to park ~300 people on a seemingly-dead-end product, just waiting for the incumbent to mess up. Sometimes it eventually succeeds commercially (Teams, PowerBI) and sometimes it doesn't (or hasn't, yet, see Bing).
What is happening here is a sign they are changing their view on how this capital allocation works.
To what? We don't know, and I think they might not know yet, either. If you are spending > $4M for a top-end engineer and $8M on their tokens (this refers to a person know personally, it is not hearsay), this will necessarily change the how this allocation works. Maybe it doesn't make sense to park 300 people on tomorrow's PowerBI any more.
What is clear is that we are not in anything approaching an equilibrium state. MSFT (and others like them) clearly do not have enough conviction to fire all the line engineers and only keep the so-called "elite" ones. It's not even clear that they think we're on that trajectory. I think what we are seeing, though, is that they have enough conviction to try to clear the way for more of this kind of spending on these so-called "elite" developers.
I'm not sure what you mean. Are you saying you think I'm suggesting they are extremely sophisticated actors on some dimension? I'm saying they're best thought of as a Berkshire Hathaway-style capital allocator, not a tech company.
Almost everything you'd want to say about it would spoil something interesting about the film. (This is not hearsay. I've seen it and I can't think about how to say anything meaningful about it without spoiling it.)
So if the movie can be "spoiled" it's at least....entertaining? I'm not sure to think if people are just leaving disturbed and mildly traumatized or what
I felt the movie was entertaining, but also somewhat unsatisfying. You don't really get answers or catharsis from Holmes being further confronted and exposed. To me, the movie was about media narratives and how deceptive lies of omission can be.
So we have here a massive evidentiary brief, the point of which is to argue that it is easier than we thought to, like, "sequence the AI genome" (in my estimation). We revisit famously-messy neurons, examine various tool failures, check places where messy things still hide, and catalog how "polysemanticity" is represented variously in the network.
Once sequenced, the thinking goes, we'll be able to do things like: explain and anticipate failures; predict what happens when you modulate the pathway an input takes on its way to becoming output; reconstruct algorithms the models have learned.
I'm not 100% sure this will work out like the mechinterp people think. If the neural representations and pathways are "clean" (organized into a a meaningful-to-humans hierarchy), it seems at least very plausible that the behaviors that AI xrisk people are worried about will mostly boil down to straightforward synthesis (i.e., mostly-linear-interpolation) of mostly-well-cataloged knowledge. And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
What I envision is the ability to find the circuits and neurons evaluating morality and ethics and being able to figure out how they work at the boundaries and ensure safe deferral to human judgement or stopping outside of in-distribution behavior, which further is vetted against actual human values.
I don't see a lot of hope for that, because it's not a well-specifiable quantity. I would say we currently have no real evidence that you can even do this even for concepts that are much more well-specified like, say, dogs. Maybe you can! But that is not affirmatively known, and nothing we have is on track to supporting that, right now.
I think the paper says that while there do exist dog neurons that carve out the difference between a "Boston terrier" and a "terrier", those neurons actually only get to carve that fine line because they're part of a whole flow control system where activation has already flown through much coarser grained discriminator neurons (like a "V" neuron), including sometimes through the negative side of activations, before the terrier discriminator ever played a role.
So, if you try to draw a circle around the "dog" part of the network, do you circle the Boston terrier vs terrier neuron only? You'll be missing most of the information that represents the "dog" category and that same dog category is also involved in shunting activation away from things that are toward-being-a-dog in whatever sense is being discriminated in that moment.
You’re only considering the front half of the network though, classification is a helpful example for explaining the concept - but if it follows through to the output then you could quite easily imagine just shunting some activation from one branch of the representation tree to another to steer behavior. Which we already do with steering vectors - but if you could read off the actual topology you could do that with much higher fidelity.
The hard part of AI alignment has never been making models follow instructions. It's that not everyone agrees on which and whose instructions AIs should follow.
I'm not sure it will work out either. Or that it extends beyond image classifiers, which have the nice property that you can use them as image generators and find their ideal match picture. But this looks like work others could replicate.
It's an objective result.
The relationship with "alignment" isn't clear. This may help predictability, but alignment is a few steps further along from that.
Some people seem to be upset about this. I'm not clear why. Some are followers of Yudkowsky. (Whether that's a cult is an open question. I've encountered them online from the sex side. Actually, I think the hookers are saner than the rationalists.)
Is there a "there are some things man is not meant to know" position on this?
People who want neural nets to be a black box, forever inscrutable? The people who talk about neural nets being "holographic" may incline in that direction.
The essay says he has extended it to LLMs, although not as much.
The connection to alignment seems pretty clear. Steering DNNs is well understood, the bottleneck is identifying what to steer. Once you can easily identify neural circuits then turning them up/down to achieve specific results is straightforward.
Note that this is already being done even without the kind of precision that Nick's work says is possible. Morality abliterations of open weight models are easy to find on HuggingFace. They compute the steering vectors by just presenting the model with lots of disparate examples that have the single moral decision as the commonality, then finding the shared vector directions.
> And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
...wait, what? Removing a bunch of knowledge seems like a fantastic and obvious solution to me. My AI coding bot doesn't need to have knowledge about making nuclear weapons, the customer service AI doesn't need to have knowledge of dangerous viral synthesis techniques, my email receptionist bot doesn't need to know about C4.
If we can easily ablate dangerous knowledge from these models, we're sorted.
Why always nuclear weapons in these conversations, lol. Is there something in this community that makes this example particularly interesting?
Go read the transcripts where the AI is “unaligned” and has some very bad impact, like encouraging kids to kill themselves. There is not some obvious piece of knowledge to remove to make it safe. You’d have to basically lobotomize the model to make those conversations impossible. People won’t use a model that’s lobotomized.
I disagree: You could lobotomize all the social-interaction parts of a model and it would still be very useful for practical purposes. Personally, I think it would be wonderful if we had models that knew how to debug javascript or look up a missing shipment at the warehouse, but didn't know how to pretend to be someone's friend.
If, as you suggest, fewer humans find it compelling to spend hours talking to a machine, so much the better.
There are maybe examples where your point stands but that’s probably the worst one. The point of that knowledge is actually to answer questions. That is where social skills in the model come from: training it to answer questions.
I don't think a model needs to be a parasocial companion in order to answer practical questions effectively. It doesn't need knowledge that allows it to mimic empathy, or simulate emotional engagement to answer "how can I repair this lawnmower?" or "why is this SQL query slow?"
> “Google Translate is amazing, but it has taught us essentially nothing about linguistics,” Strogatz said. “We don’t understand why it works. It just works. That’s the Twilight Zone. It is knowledge without understanding.”
This is not true, and it is the crux of an enormous debate we're now having across many, many fields.
Classical Chomskyan syntax (which models sentences as parse trees) has been enormously, astonishingly predictive of the rich structure in real-world human language. But we are quick to forget that we invented concepts like "words" and "noun phrases" specifically to help us analyze and predict things about human language. What LLMs and modern MT shows, definitively, is that there is another, totally separate way to describe these phenomena, which is both vastly less descriptive, and also vastly more predictive.
That these appear to trade-off is unfortunate, but it is NOT the case that we "don't understand why it works." We absolutely understand why it works, if we didn't understand we would not be able to improve these systems. What we do not have is a clean, descriptive theory that boils this down to something that is formally specifiable and analyzable. But those are different things, and this is the root of the disconnect between many communities working in and around these spaces (linguists vs computational linguists; mathematicians vs computational mathematicians; mech interp people vs empiricists).
"We absolutely understand why it works, if we didn't understand we would not be able to improve these systems" - Understanding how to improve something does not require understanding how it works.
Growing a plant does not imply understanding soil minerals or photosynthesis.
Training a dog does not imply understanding it's brain.
Using LLMs to improve a codebase does not imply understanding C++.
Improving a PC with new parts, does not imply understanding RAM.
You can understand abstractions about the system, without understanding the system.
Without gradient descent or other forms of automatic optimisation, do you think humans could build something as capable as GPT-1?
Not sure if this will bake your noodle or not, but yes this is basically my point. Saying I understand how dog training works has very little overlap with saying I understand how the dog brain proteome works, even though they're related, and even though I might (or might not!) understand better how to train my dog for having fully understood the proteome.
Saying we understand training and deploying machine translation systems that have strong predictive (rather than descriptive) power, apparently has very little overlap with defining a formal, analyzable, mathematical model of language. Mathematician who refer to "understanding" generally mean the second thing, but the first thing is also a kind of understanding. Just messier.
I think they're referring to the fact that neural networks are more or less black boxes. Their internals are blurry and not explainable in the same way that traditional linguistics tries to explain things - with rules.
Of course, ultimately that's because language itself doesn't follow simple rules like linguists would prefer. They're more like guidelines.
Yes, that is one of the points I am making. Chomskyan syntax is a strongly descriptive theory that is entirely ex post. MT and LLM systems are strongly predictive and have no descriptive theory. It is completely wrong to say the first is "understanding" and the second is not. They are both different types of understanding. In particular, Chomskyan theory clearly does not represent enough "understanding" to predict real-world English sentences with similar lucidity to that of LLMs.
No, they're not predictive at all. They're generative. A prediction is a forecast based on a theoretical model about a state of affairs in the world. Making a guess what the weather tomorrow will be is a prediction, and a prediction can be true or false. An LLM generates text based on certain parameters, and output it generates aren't true or false, they're simply whatever the system is tuned to produce.
LLMs are terrible at making forecasts or world modelling precisely because they don't actually operate on much of any model or theory, they have no notion of truth or falsehood, they simply generate whatever you want them to generate.
If you'd like to use that definition of "predictive" in this discussion that's 100% fine with me. But, you should do so knowing that in ML and AI "prediction" is a term of art that means "a statistical model's output given some input." LLMs are one example, in that they take as input a sequence of tokens and predict the next token.
In this specific context, the point is that LLMs and MT "predict" lucid-sounding sentences that incorporate linguistic structure (syntactic and semantic) that are well beyond the predictive abilities of the Chomskyan theory of syntax (which is itself a generative model). In other words, the Chomskyan model of syntax is descriptive of much of the rich variation in real-world human language. But it is not predictive in the sense that it does not provide a constructive algorithm to emit lucid-sounding sentences (and does not attempt to, either).
For those who are not math people, this comment is referring to the fact that "attention" is formally equivalent[1] to Nadaraya-Watson kernel smoothing, which is a kind of weighted average. Yes! Incredibly, attention is just a weighted average.
Imagine you have a series of (x,y) coordinates. You receive a new x coordinate, and you'd like to predict the corresponding y. So, the absolute dumbest thing you can do is find the nearest x, and then report its corresponding y as your prediction. The second dumbest thing you can do is find the n closest x coordinates, and then average their y values together, and output that as your predicted value. And the third-dumbest thing you can do is that n-closest-average thing, but give the "closest" x coordinates the most "weight" when you average them. (Technically, not always "closest": you have some "kernel" function which does the weighting, but "closeness" is a good kernel so I will use it here.)
That third-dumbest thing is Nadaraya-Watson kernel smoothing. And that's what attention does. You have a big list of vectors. You compute a weighted average of those vectors. You now have a new vector. You can do what you want with it, but LLMs will generally turn that into a distribution over token types, from which you'll generally want to select either the top one, or weighted-randomly one from the top k (known as "increasing the temperature").
With all that said, I'm not sure "Kernel smoothing is all you need" is actually materially better than "attention is all you need" for 99.999% of humans. :)
In this context, what does "closest" mean? Most similar token embeddings? Most similar Q embedding to prior K embeddings? Most similar positional embedding?
Are you, in essence, implying that the Key-Query attention mechanism is basically a neural function approximation of the Nadaraya-Watson kernel or something very like it?
Yeah, I mean, a lot of things can express attention, because attention is mechanically extremely simple. The "trick" of attention is that it's "relatively easy" to train. Or if not "relatively easy", we at least have a lot of collective experience doing it.
The crucial question, however, is "Where do the weights come from?" :-)
btw, "kernel smoothing" etc. are still fairly complex terms that sound a bit like math magic: 1-nearest-neighbor (1-nn), k-nearest-neighbor (knn), and weighted knn would be even simpler and possibly more descriptive.
Yeah but Nadaraya-Watson kernel smoothing isn't actually 1NN/kNN/weighted kNN.
First of all, the trick relies on imputed 1NN/imputed kNN/imputed weighted kNN. And second of all, you still have to explain how imputed weighted kNN relates to the problem of taking a bunch of x/y pairs and predicting a novel y for some novel x. When you start to unpack all of that it does not look simpler to me than what I wrote.
For the weights, I'm not sure that's as mysterious as you make it seem. The most important thing they learn is, basically syntax. So you have 40k token types to choose from at each prediction, and the vast majority of them are statistically implausible because they are syntactically invalid. At some positions this restriction reduces entropy from ~15-16 bits (or 40k choices) to 2^8 (~200 choices) and in some places it reduces it "only" to 12 or 13 (~8k choices, common when the next token is "approximately a noun").
Once you winnow it down, it's much closer to a moderately big multiple choice problem. And much of that entropy lies just in the model's representation of word co-occurrences.
Attention is not based on Euclidean closeness, therefore it is not kernel smoothing. Saying otherwise would be the same as saying kernel smoothing is just an average (e.g. omitting the important part that it has to be distance-based-weighted).
First and most importantly, attention is, formally, kernel smoothing with a particular exponential kernel function, and if you don't believe me, you can read the paper I linked, which proves it.
Second, and less importantly: kernel smoothing absolutely, definitely does NOT require a kernel function representing Euclidean distance, and in fact the entire point of kernel smoothing is that you can pick basically any function that fits your needs. Many use cases find it helpful to have a real-valued kernel function that is positive and semidefinite, or nonnegative and integrating to 1, or whatever. Different kernel functions in these spaces have different properties and trade-offs. The exponential kernel attention uses (if memory serves) is (1) finite and nonnegative, and (2) has a positive total weight so you can normalize.
Third, and least importantly, you definitely can have a kernel function that is "uniform" (i.e., "just" an average) although it's generally not very helpful (e.g., not positive semidefinite).
> First, kernel smoothing with a particular exponential kernel function
Well, here's the thing. Kernel smoothing uses Euclidean distances. "with a particular exponential kernel function" is the novel bit. I even gave an example of what absurdity you will get if you allow arbitrary addition like this to the term.
> every assertion here is mistaken. ... Second, ..
Never made that assertion.
> Many use cases find it helpful to have a real-valued kernel function that is positive and semidefinite, or nonnegative and integrating to 1, or whatever
> you definitely can have a kernel function that is "uniform"
You did not get that point. See response to your "First". I was explaining why the reasoning that led you to "attention == kernel smoothing" is faulty - if you apply it to averaging, you will get "averaging == kernel smoothing". Because kernel smoothing is "just" averaging with distance based weights. So attention is "just" kernel smoothing with Q-K based distance function. The issue is in both cases the part after "with " does all the heavy lifting.
Your writing takes a lot of work to follow, I think because English is your second language. I will try one more time.
For those following along, when this user says that attention is kernel smoothing with a "Q-K based distance function", I think (THINK!) what they're trying to say is the second part does more work than it gets credit for when we say attention is "just" a weighted average.
Both of these things are actually true. Attention is indeed "just" a weighted average (there is a proof in the paper in my original reply), AND ALSO, the Q/K are learned, and the similarity between two learned vectors suggests there is more going on than might initially be obvious. (I actually argue in the top comment that the positional information is the #1 reason why attention works, so I agree with this.)
Let's see why.
What are Q and K? The idea in the original paper was: we have some learned embeddings for our tokens, and some "context" we'd like to use to enrich those embeddings. Stuff like the positional information, context accumulated through layers, other learned stuff. So we do this enrichment, and that produces the so-called "query" (Q) and a "key" (K) vectors. Attention is STILL a weighted average, where the weights are determined by how "similar" those vectors are (this is what we're talking about when we discuss the "distance" provided by the so-called "kernel function").
Anyway the reason it's hard for me to parse this reply is because I think a lot is getting lost in the transition from a second language to English. For example, "distance" in this context means something specific, and to be precise the exponential kernel provides a similarity measurement (dot product can encode distance but generally requires fixed-length norms). Similarly, I am confused by their use of "Euclidean" distance (kernel functions do not require this, see p. 203 of the seminal Elements of Statistical Learning which provides a general construction of anisotropic distance-based kernels which is well-accepted), I'm not sure how to interpret "what you feed to the kernel is Euclidean distance" (a pairwise kernel should compute the similarity, not be fed the similarity) and so on. None of this is the end of the world, I am just having a very hard time figuring out what they meant.
The authors did not know that it was Nadaraya-Watson kernel smoothing when they wrote the paper. Not recognizing that the novel thing you are doing is secretly another, very old, very common thing, turns out to be an extremely common ailment in modern ML/AI literature. :) I suspect this is true also of physics.
Wouldn't it be better to express it like this: the authors haven't realized that the attention mechanism can be represented as a special case of Nadaraya–Watson kernel smoothing, a representation that hasn't resulted in practical improvements in attention mechanisms, probably indicating that the connection isn't that important in ML?
Like an accountant not realizing that addition and multiplication are special cases of a monoid operation.
No. It definitely has resulted in practical improvements to attention. One practical improvement is that kernel smoothing is vastly easier to explain than "attention", which I think I have already demonstrated here.
Aside from that, kernel smoothing has roughly 70 years of literature devoted to every practical aspect you could imagine: approximation, accuracy, pedagogy, behavior under sparsity, relationship to other well-known theories like Bayeisan learning, and so on. Variants like Performer and elliptical attention, all directly arose from the knowledge that attention is intro-grad-stats kernel smoothing. A large portion of our understanding of ICL is directly or indirectly resultant from the fact of this correspondence.
As an NLP person it's very hard for me to endorse this explanation, because the Vaswani et al transformer (as it is described here) simply does not work until you add the most important ingredient of all: the positional encoding.
Most of the semantics of a sentence are indicated by the precise order the tokens appear. This is easiest to see in a language like English, since it is visible in two very prominent ways.
First, the morphology is fairly derivational, meaning you can keep adding suffixes to things and English speakers will generally get what you mean. For example, add some more suffixes to that word: derivational-ish-ly. You've probably never heard this specific variant, but you still probably have some idea of what it means anyway. The transformer can only capture these semantics if it has some representation of the fact that the suffixes inflect the meaning of the word's root (presumably something like `deriv`). English encodes this entirely by the position of the suffixes relative to the root. Without the positional information, there is no way to recover it, and sentences would lose much of their meaning, since English depends heavily on this construction.
Second, English has relatively few cases, and thus relies very heavily on the order of words to indicate what is the subject and what is the object. English specifically is Subject-Verb-Object, and the instinct of native English speakers to use SVO ordering is so strong, you will find them inserting nouns and pronouns into sentences that do not actually have subjects at all. "It's raining outside" is the most famous example. This sentence has no subject at all, and yet native English speakers feel a strong need to place "it's" at the beginning.
If the transformer did not have positional encodings it would be completely hopeless to recover what is the subject and what is the object from English sentences. There simply is no other way to reliably tell, other than inspecting the word positions.
It's worth noting we also do have the counterfactual. Generative language models that explicitly exclude positional information, like Latent Dirichlet Allocation, are only able to (essentially) cluster word co-occurrences. That is not as useless as it sounds—if I say "fire house burning ambulence" you kind of get the point even without a subject or an object—but it certainly does not generate the lucid, provoking sentences that modern LLMs do.
> "It's raining outside" is the most famous example. This sentence has no subject at all, and yet native English speakers feel a strong need to place "it's" at the beginning.
French is interesting because it evolved this so-called "dummy subject" pattern (which is actually called "subject-prominence") for an entirely different reason than English.
French is descended from Latin, which is "pro-drop" in the sense that you'd drop the "dummy subject" and simply write "pluit." Later, Old French, like English, lost most of its cases. Because of that, many word endings started sounding very similar (a problem known as "agreement syncretization"). The pro-drop "null" subject disappears in the record roughly as that happens, presumably to help disambiguate.
This is an entirely different path than English took, and it's also not a path any other Romance language took. Spanish isn't even subject-prominent. It fully allows the "null" construction, e.g., "llueve", which literally is just "rains."
The sentence does have a very obvious subject. The environment is producing rain. The subject is the environment and its condition is being elaborated. It = environment.
The rain is not magical, the environment is producing it. The environment is raining. It's raining outside means the environment is producing rain outside.
Who could possibly confuse that as lacking a subject.
And with this reply we begin one of the permanent debates on this subject. Regardless of where anyone here comes down on this particular case, "it" is an indisputable truth that not all English sentences have subjects. Yet, many of them have pronouns where a subject should be. Maybe the famous example isn't one of them, but "it" is not at all hard to come up with a sentence where that is true. :)
Italian, Spanish and Latin (for example) can generally avoid to express explicitly the subject in a dedicated pronoun, as the verb already encode the it through specific flection. In contrast in French and German only in imperative tense one can formally make such an ellipsis (colloquially of course one can encounter more nuances).
In Ukrainian it’s «дощить» (from “дощ” — rain) — the form of the word tells you its function in the sentence. Same happens in other Slavic languages: Polish, Belarusian, Slovakian.
Latin languages rely on the order of words to explain their function.
So much so, English speakers don’t say “blue big ball” in German, time before place is mandated.
>Most of the semantics of a sentence are indicated by the precise order the tokens appear.
I would rather say it’s derived from contextual element. And in linguistic context which encodes more interpretation cues, order of morphemes drop symmetrically in semantical importance, though it can still affect interpretation on pragmatic level. Latin for example already offers more flexibility regarding word order, though it’s used mostly in poetry/stylised-rhetoric. And as German show having the case encoded in the nouns don’t even prevent mandatory rigidity in sentence structure.
It's hard to speak for every language on Earth, but generally languages that have less flexible word orderings will have correspondingly more complicated morphology. In those cases, order does obviously still matter, but instead of in word orderings, it would be in things like prefixes and suffixes.
There isn't really a way to get around this. Natural language IS structured as a recursive tree. Our job is to extract this tree from a flat stream of tokens (or similar), and straightforwardly the only way to do that is to make the ordering of the tokens meaningful. There is no other option! This is how 100% of parsing works in any context whatsoever, both theoretically and practically.
There is nothing like a valid criterion to discriminate "natural" in languages. All human languages are praxeostratic, they result from the cumulation of practices through time. Some might be more or less blank slate founded through theoretical jump, but even burning the bridge doesn’t mean creation ex nihilo. I mean, the only thing that would not be "natural", would obviously be "supernatural", like if one pretend a language to be some divine creation and not result of human contingent existence.
And just because utterances can be more or less matched with recursive tree doesn’t mean there is an ontological absolute truth about language there. Even "language" refers to some vague abstraction. Just like the "it" in "it rains" mentioned in this discussion: no entity out there actually respond to "language".
At introspection level, I absolutely don’t interpret utterances like recursive tree. They are more like cues going through living graphs more or less already connected and being able to change topology on the go. Impermanence of interpretation results is the only constant.
Now it’s not to dismiss that some other people might feel like "Natural language IS structured as a recursive tree" is the only way. People have different perspective, different brains, different life experiences. One who dwell in aphantasia¹ don’t perceive the world the same way as people that have hyperphantasia². Different of course here doesn’t mean better/worst or superior/inferior.
¹ A word I discovered in some HN comment
² A word that I had never red before but just popped up in my mind as I was thinking about this sentence, but which is actually already is in use in literature apparently: https://en.wikipedia.org/wiki/Hyperphantasia
A tree is a graph so I’m not sure I understand your idea. But, at the end of the day, the point of the tree is just to represent the relationships between the tokens. There IS a relationship. As long as your theory represents that as precisely and with as much predictive power as trad linguistics, it doesn’t really matter how.
I’m also not sure why you are talking about why natural languages are not different from other ones. That is what I’m saying, it’s the reason behind the comment that all parsing works the same way.
Yes, as I mentioned to another commenter, I'm talking about the original formulation because that's what the article talks about. We have more options now, although (as you seem to be suggesting) they do all necessarily model position. They have to—there is no other way to do things like recover the subject in an English sentence without positional information.
AFAIK, positional encoding is not that important for the performance of causal transformers (e.g. all important modern LLMs). The causal masking itself supplies necessary information implicitly.
This has been addressed several times now in the peer comments to this one. I am addressing the original formulation of the problem because the article is about the original formulation. There are other options available now, including causal masking, but in the original formulation positional encodings are THE most important single ingredient in success, because many aspects of semantics cannot be recovered from anything except position. One example is the subject in an English sentence.
You are making an excuse for a poor observation. "the Essence of Intelligence" applies to the current models, and they do not require positional encoding, but they do require attention.
> because the Vaswani et al transformer (as it is described here) simply does not work until you add the most important ingredient of all: the positional encoding.
This is factually correct. The Viswani et al transformer DOES NOT WORK without positional encodings. A transformer architecture that involves (e.g.) causal masking is a different architecture. If you want to separately discuss one of those architectures I am happy to do that too.
But positional encoding by itself also won't work nearly as well.
And if you look at how transformers work you will immediately realize there's nothing a transformer can do that an RNN cannot do, and frankly that an RNN cannot do 1000x more efficiently at inference time. It's just that training even the best known RNNs is going to take a lot longer because it can't be parallelized. It cannot train on 100 GPUs, in fact it cannot really use a GPUs advantage.
Plus equivalence proofs will get you a lot further than RNNs too. Kernel methods will obviously work as well, for example.
They don't really have that many different characters, it's just that the way Chinese script was adapted to computers use a rigid set of that many pre-composed characters.
To clarify, I'm talking about the original formulation of the problem, where positional encodings are crucial, because the article is talking about the original formulation of the problem.
But these days, there are many more options. They all maintain this positional information in one way or another. The one you cite (as I understand it) drops the positional encodings after pretraining, and keeps (e.g.) the causal mask.
You could be right, I have not kept up with the literature very closely! I'm just trying to help people understand that this article does not capture something very important about the Vaswani et al formulation of the transformer architecture. :)
The only reason I bring this up is because I think it is counterproductive to view these people as experts when they are likely not.