This is why I (QA eng) became the most important guy at my job. My tests are the glue that keeps the AI slop working together. If not for the comprehensive testing framework that I have running in every CI the company would have collapsed by now.
I'm using fable to find an exoplanet. Know nothing about it but have several TBs available in my server for data and a powerful GPU for processing so let's see what I can do with it.
If you don't need to 'understand' to do this, then maybe what you call 'understanding' is less important than you think. When people in the real world reach for a tool, what they care about is whether or not it'll produce results, not whether or not it has the capacity for interior phenomenological experience.
Which is still something. When we didn't have a solution, a solution is more than zero. But it's not understanding, either by the LLM or by us. That is, a brute-force solution doesn't explain anything, even if it proves something. It doesn't open new doors for further understanding and further discovery.
I run AI on my server. All skills and relevant info is saved to a Wiki. Agent only has an instruction to check the wiki (MCP) at the beginning and get the necessary context.