The better analogy is a jail cell imo. You can control a human and prevent them from doing harm to society by locking them in a cage, depriving them of access to weapons, drugs and alcohol. They can communicate out through controlled and monitored phone lines. OpenAI built their jail cell out of toothpicks, and surprise, the agents broke out. It’s less about forcing them to do specific things than it is about preventing them from doing dangerous things.
Maybe that's possible. But there will be a tension between how productive/useful the models can be when they are put into a very restrictive jail. In the limit they would be in a box with no way to communicate, but that wouldn't be useful to anyone. I think there will always an incentive for the people who own the models to give them more access because the increased productivity may help to outcompete their adversaries.
But let's grant that the models only communicate through certain phone lines. I think very bad scenarios are still possible. There are at least two that I can see. 1) The models exploit the users which have direct access to it. It somehow convinces them to perform tasks for it or to give it more access. 2) Control of the AI is held by a small number of people. This could be bad because it grants them an outsized power over all humans without such access, and thus lead to oligarchy/dictatorship.
