Sure, nobody knows the exact future. But you still plan for what's probably going to happen. I don't know if social security is going to be around when I retire, and all the experts are saying it's probably not going to be. So, I'm planning a retirement without. Similarly, all the experts are saying AI (<- that word is not "LLM") is probably going to improve. I personally think it's irrational to ignore experts.
I think that a person (me) who has worked directly with frontier AI for over a year now for many hours a day may have gained some expertise in what this technology is capable of and thus what is dangerous about it, but I wouldn't speculate too hard on what superintelligent AI would or could do because we have not yet seen such a thing nor is there even an established criteria as to what that would look like or how it would be valued and in what capacity (hint: humans are still the final evaluators of what is valuable)
The simple fact that I have to write every control scheme in the book to keep the thing on the goal path (and not lapsing into laziness or dishonesty) is itself possibly a safety feature, if you look at it that way
> The simple fact that I have to write every control scheme ...
Look three years ago for a reality check of the progress. The LLM you're using now isn't the LLM you'll be using in 5 years. It probably won't even really be the same architecture, just like those from 3 years ago are VERY different than the full systems we use today (harnesses and all). What you're doing was literal science fiction not long ago.
