Plus everything important is always written down.

It’s more true in aviation than anywhere else (except maybe space flight), but still.

And if it's written down, it's accurate. 100. Percent. Accurate. /s

You know, if I had to make that bet, I would choose airline maintenance manuals as the test case.

Have you seen the maintenance manuals on an aircraft production line? They're in 3-ring binders, so you can take out and replace pages. They are constantly being corrected and updated. The industry recognizes they're never at 100% (and to their credit mitigation is part of the process).

And because it's accurate the AI will know how to weigh that accurate information against other accurate information in such a way that it will give you a great answer without any help. In other words: any amateur will be able to do aircraft maintenance! What can go wrong? /s

GP did not suggest anything of the kind, though. They talked about "figuring out where to look" which very different. In that kind of use, it doesn't really matter that the model might hallucinate or draw faulty conclusions, as you're not using it figure out what to do, how or why. The only thing it is telling you is where some information is located, after which you go check it out from the actual source of truth. If the reference turns out to be wrong, you'll notice immediately.

I hate AI as much as anybody, but claiming that it's not useful for "distilling volumes of data into a short list of places to look first" is silly.