It's certainly possible that "the special sauce" that provides human consciousness and sentience (that I believe is not there for LLM-based AIs) provides some key ingredient to some forms of intelligence (particularly when it comes to originally) which LLMs would not be able to duplicate.
For example, much has been recently discussed about how OpenAI's math results, as impressive as they are, remained within "the convex hull" of existing math literature. Thus, some have argued that the way LLMs work will keep them squarely within this convex hull, and also there seem to be clear examples from human history where humans came up with breakthroughs that were outside the convex hull of ideas that existed at that time.
There has been a lot of recent debate, both for and against, this idea that current AI approaches are limited to this convex hull. So I'm not going to try to convince you one way or the other, but I do think it is at least a plausible explanation of how/why current approaches end up hitting a wall.
I'll also show my bias which is to say I certainly hope they hit a wall, because I don't understand how anyone can think that humans won't be truly and completely fucked if AI ends up far surpassing human intelligence.
the convex hull argument has recently been under scrutiny as well.
some proofs of famous conjectures (for example unique games conjecture) have been described as creating new, insane, almost alien ideas. Scott aaronson has a blog post on this.
Edit: OK, I found his rationale in the comment section, and I don't find it particularly convincing:
> (1) Einstein and Gödel and Gauss and Shakespeare “merely filled out some points in the convex hulls” of everything that came before them, if you take an expansive enough view of what those convex hulls actually were. What a pathetic piece of cope.
> (2) Dana tells me she now mostly understands the proof of the UGC, and is amazed by the new ideas in it, and wants to give talks about it soon. So, maybe people can listen to that and then lecture her on how it’s “just a point in the convex hull” of the ideas that she’s worked with for 20 years!
First, to just use Einstein as an example, I remember reading many years ago (like at least a decade plus) that many scientists felt like if Einstein had never lived, the theory of special relativity (i.e e=mc^2) likely would have been found pretty quickly in any case - the work had already been done putting the nail in the "luminiferous ether" coffin, and people were starting to connect the dots. I think most people would agree that special relativity was within the "convex hull" that existed at that time. However, I've also heard that general relativity was so far outside the bounds of scientific thought of the time that if Einstein hadn't discovered it that in may have been 50-100 years or more before we figured it out - it was far outside the bounds of the convex hull of that time.
All that said, I'm not a mathematician qualified to determine whether what has been discovered by OpenAI is inside or outside the convex hull, and I do think that the vast majority of human intelligence operates within the convex hull in any case, and what AI can do is hugely, immensely powerful. Still, I'm more inclined to agree with this other commenter on Scott's blog:
> I am a little more in the “convex hull” camp when it comes to AI creativity. In my own interactions on research problems, I find the AI to be singularly uncreative. But these are not math problems.
> From my own introspection, my creative ideas come from a kind of visualization: situating myself in a quasi-physical abstract place that is very difficult to explain. But from there I can simply “see” things by “looking around”.
> This seems to me to be a byproduct of being an embedded agent, not just a prompt-responder. Embedded agents reflect and cogitate using their model of the world. Yann LeCun in particular pushes this as a necessary requirement for true intelligence.
> That said, I don’t agree with LeCun’s claim of necessity. Just as reasoning turned out to be something mysteriously emergent from language in the LLMs, so too could this kind of creative behavior. But I have no idea how.
Interesting - thanks for your perspective! I just took Scott's comments mostly at face value.
In any case, I think most researchers have been filling the convex hull (and have been enjoying doing so?) for most of their research. So even if Einstein provided something truly necessary, there's a reason there's only one Einstein (or grothendieck or insert famous name here). I know that's moving the goalposts but yeah.
Yes, I agree. The reason we highlight truly original, unique bits of genius is that they're so rare. So while I hold on to the idea that there is something humans will still exceed at, the vast, vast majority of things we do obviously aren't of that nature, and AI will vastly change the nature of work and society in the very near future.
For example, much has been recently discussed about how OpenAI's math results, as impressive as they are, remained within "the convex hull" of existing math literature. Thus, some have argued that the way LLMs work will keep them squarely within this convex hull, and also there seem to be clear examples from human history where humans came up with breakthroughs that were outside the convex hull of ideas that existed at that time.
There has been a lot of recent debate, both for and against, this idea that current AI approaches are limited to this convex hull. So I'm not going to try to convince you one way or the other, but I do think it is at least a plausible explanation of how/why current approaches end up hitting a wall.
I'll also show my bias which is to say I certainly hope they hit a wall, because I don't understand how anyone can think that humans won't be truly and completely fucked if AI ends up far surpassing human intelligence.