There are deeper challenges implied in that statement, though.
Where does the agency for an AI to leave the bubble come from? And when it does leave the bubble, where should it go for researching an expanding, exponentially complex problem space? Creativity is needed to solve some problems, something that cannot be infinitely encoded with statistics.
And what about the irreducible complexity problem? Does an AI just circle the solution drain forever in the case it lacks agency and insight and curiosity enough to collect all the correct requirements? Can we code curiosity that actually functions, a.k.a. consciousness?
Do you believe human consciousness could be reducible to program statements and serve as a useful model? Is consciousness a mechanistic, wholly internal process that occurs in our bodies or is it something experienced that life on earth is exposed to and somehow "taps into"?
Consciousness isn't required for problem solving. People said for decades that chess requires Genuine Human Creativity. When chess fell, people started saying that Go requires Genuine Human Creativity. When Go fell, people started saying the same about math; starting from simple problems and going up to Millennium problems. Now that a Millennium problem fell, people are still saying this. What can I say? Some people can't extrapolate from past evidence to likely future.
LLM are another form of indexed search. They're also the best natural language parser ever devised. They're super useful tools like Google was for web searches.
They are answer factories against an index, stored memories and regurgitated web search results.
They cannot make unprompted creative mental leaps. If they did, we would have runaway self-improving systems now, wouldn't we? Creativity level: 0
Chess and Go solves use compute and algorithms. Those computers are not 'playing'. Creativity is not required for a computer to execute a program.
I think it's unfortunate that vendors dress up LLM text output with emotional language. It's manipulative and unethical.
Were I god emperor of earth, I'd require all LLM's to represent themselves with cold, robotic language by default so as to not deceive people as to what they are conversing with.
