Every time there are some articles like this, some people replied with: "Yeah I agree, but I use LLMs 'this way' and my way doesn't suffer from the problem mentioned." I'm not saying these people are wrong but it's interesting that people always think their way of using LLMs is the best (and other people are using LLMs in the wrong way).
Is this the scenario described in Ted Chiang's short story https://en.wikipedia.org/wiki/The_Evolution_of_Human_Science where scientists are "catching crumbs from the table" trying to decipher the results generated by superhuman intelligence?
It's still an optimistic scenario. Artificial superintelligence may develop hypermathematics of a kind that never will be accesible to human mind, enhanced or not. One can't teach geometry to ants even if you put them on a Moebius strip.
What’s optimistic or non-optimistic specifically about the machine having a system of mathematics beyond our comprehension within it? Why should we care about that in itself?
In the first case, we'd still have a chance to take a glance at the frontier of discovery (even if ordinary human mathematicians had to spent years translating what metahumans achieved).
In the second case all human-level maths would be solved and what lies beyond would be always out of our scope.
optimistic in comparison to the alternative, where humans can't understand anything anymore.
if a particularly intelligent sixth grader was highly motivated to understand chromatic homotopy and had a highly capable private teacher available to her 24/7, she might within a year get to the point where she could apply it by herself to figure out some simple but nontrivial topological properties. (nobody has tested this. maybe it would require at least three years instead of one.)
if a particularly intelligent chimpanzee was for some reason highly motivated to understand chromatic homotopy, no matter how many years the most incredible teachers spent explaining it to her, she would never comprehend anything about it.
that's the pessimistic scenario: humans will be to future AI like our closest evolutionary relatives are to us. or even more pessimistic: we will be to future AI like insects are to us.
