Because the degree of realism in a video is determined by implicit knowledge of concepts like "in front of", "behind", "next to", "inside", "outside", "between", "occluded by", etc. as well as distances, angles, and the relative size of objects when viewed from a perspective.
I guess I just don't have an intuition for why that is different from what it does for non-"spatial" things. Like the fact that it works is still because it produced the code it did token by token. Whether its dealing with, e.g., "beside the rock" or "in the array", it's doing the same kind of inferential activity.
> I just don't have an intuition for why that is different from what it does for non-"spatial" things
A big reason people separate this out is because it was only a short time ago that AI models were noticably and uniquely bad at this. I don't say this to be like "oh so imagine where theyll be in x amount of time", rather that this thing that was once a serious limitation of the technology is slowly being compensated for by larger and better trained models
If you're at (-2, 3, 5), pointing in the (2, 1, -1) direction (as a direction vector), then which is in front of the other from your point of view: the object at (4, 5, -8) or the one at (6, 4, 3)? (or neither)
You can't just defer this decision to the three.js program because you need to understand this sort of relationship yourself if you're going to suitably place objects (and your own view) in a three.js scene in the first place. Building a complex scene involves making hundreds, or more, of decisions about where to place coordinates in 3d space.
This example is underspecified (e.g. what are the shapes and sizes of the objects and what is the angular width of your view) but it illustrates the problem. Even with a good intuitive understanding of 3d space, we would struggle with this. LLMs are not calculators, so it's surprising if they manage it.
I think of all these responses this articulates my point the most. If humans would struggle with it, why is it still supposed to be "spatial reasoning"? And underspecified or not, you frame this exactly in the way an LLM would tackle this: if I am making a game, I don't place objects in arbitrary positions in a space and try to keep it all in my head, but in relations to one another. Then, the point in front of another is the one where A-B>0.
>Whether its dealing with, e.g., "beside the rock" or "in the array", it's doing the same kind of inferential activity.
Are you sure about that? Presumably the spatial concept of "inside" and the computer memory concept of "inside" have different contextual embeddings in an LLM, not so different from how our brains have different neural activation patterns when we use different concepts. Unless you think that "the same kind of inferential activity" also applies to human neurons firing.
The problem i have with this argument is they still struggle to generate images of so many basic things that should be explained in their "world model" (text, fingers, reflections, faces) - so clearly they dont have a "world model" in the way that we have one.
And if you use image generation capabilities, you clearly see that LLM's suck as understanding "next" to a thing.
See 3D examples from earlier models and how they try to do it but things are stacked on top of the same x/y/z point or mushed together or parented wrong or disconnected.
It’s something that specifically stopped happening in the smarter models.
Same way the “pelican test” became worthless beyond a certain point and just some idea of “I like this one better” not technically more competent in any way.
quite loaded words there. "knowledge", "relationships".
the llm must be able to output the correct code when the input says something like 'place object x to the left of object y' vs when it says 'to the right'. there is an absurd combinatorial space of the possible inputs vs outputs it must generate -> the whole set of these, interpreted from a human point of view, you could call knowledge. the [x] in input x output you could call the relationships. And the fact that the LLM doesn't have to brute force represent all of them (impossible in the limited embedding vector x internal representation state) you could interpret as 'understanding'. But again, these are loaded words that are somewhat meaningless when looking at what an LLM does in a literal way.
They have plenty graphics code to train from, so this structure will be directly or indirectly available in the training data in a very plentiful way.
In literal llm transformer terms, the embeddings must have some of their components statistically represent spatial structure in some way that later in the internal layers of the LLM give some statistics of how likely it is to occur for certain code tokens to appear relative to the input of spatial wording in the token stream.
it is likely that their internal representations encode something more general than specific input x output stream combinations (because we already know this is the case for regular words and concepts). if not in the embeddings, then a couple layers into the network for sure.
