It's also interesting because in humans the existence of "Aha!" moments that are not preceded by or are only loosely related to a chain of thought is taken as the proof of the fundamental mystery and irreproducibility of human intelligence. Now the same argument is made to deny that LLMs actually think. Go figure.
This seems indeed one obvious hole in the argument of the paper. There is no indication whatsoever that human thinking process is more reliable than LLMs intermediate tokens. Which doesn't make our thinking useless, as messy as it might be. We reorder and explain it after the fact.
