That is the task they are constructed to perform, but that's just an output, not the inner workings of what's happening to arrive at that next token. You can give a human the same constrained task, but the output alone doesn't a human mind make.
Agree. Token predicting machines will continue to be token predicting machines by nature. Continued size and tuning will have the effect of making them more and more perfect at being average.
This. People treat "token prediction" dismissively, as if it were a limit and not a foundational skill. Human brains do "token prediction" in all sorts of contexts.