eh, they are also used extensively in medical research using alpha fold, and in math, and in legal for one thing. And this is of course only the beginning, and IMHO, it'd take a lack of foresight to not start putting layers and layers of safeguards now.
Yeah, to me the hugging face hacking is a harbinger of what's to come, not just in the world of tech but eventually to other fields too.
If I described a computer to you in 1998 I'd have said "they are also used extensively in medical research using alpha fold, and in math, and in legal for one thing." Maybe not alpha fold, idk if that's some new thing. But you get my point? They're a very useful tool, sometimes it feels like magic what we can do with them... But it's not magic. That's why I said magic wand, the OP was implying LLMs can (or soon can) manipulate matter at an atomic level. That's orders of magnitude removed from where we are and I disagree that there's any evidence that LLMs will inevitably reach that point in our lifetimes.
Given enough smarts, you can simulate anything. Quantum AI is an active goal.