There's no reason why LLMs can't be hooked up to a physical world feedback loop though, in fact that's already being done with chemistry research https://www.youtube.com/watch?v=AYSR02tcwes