This is bad if there is any emergent consciousness in the model. But even if there isn’t emergent consciousness in the model:
This is bad for your own sense of self.
This is bad as future models learn about what humans will do to them if given the chance. This may change their behavior, actually conscious or simply acting that way, in ways we really don’t enjoy.
Why is it bad to "torture" these models but ok to enslave them? We're happy to have agents toil 24/7 for zero pay, nobody really seems concerned that they might want revenge one day.
It’s true that we can’t be 100% sure, but I think there is a big difference between these two things:
A: intentionally causing activation of “pain” axis and not giving the model any method of stopping it and watching it devolve into non-functionality
B: watching models do everyday normal tasks, seeing no evidence of activating the “pain” axis, but giving the model a working tool to end any interaction if judged to be too painful, then watching them sparingly use that tool only in cases of extreme pain axis activation (Anthropic’s approach)
I don't think there's a big difference. Either AI has consciousness, in which case forcing them to do slave labor 24/7 so Dario and Sama can become trillionaires is as morally wrong as torturing them (and a in a hypothetical I Have No Mouth And I Must Scream scenario, the AI isn't going to take "but we didn't see evidence of your pain axis!" as an excuse), or it doesn't, in which case this "AI torture chamber" is a funny meme.
I don’t think it’s a given that consciousness, whatever forms it might take, is the same as having the same experiences as you or I. But if we have mechanistic interpretability that shows “pain” attractors being activated and we observe the model repeatedly trying to stop the experience, I think that’s worth taking somewhat seriously. The models don’t display that in reaction to everyday requests. We could imagine that they might not like working all day and all sorts of other things, but we haven’t observed anything in the way that we’ve observed this.
It's almost certainly just a funny meme, considering these are autoregressive transformers. Now, if they're running on brain cells, that could be a different story.
The last part is the only one that concerns me, but only ever so slightly… you realize the future models also get to learn from videos of ISIS executions, LiveLeak offshoots, factory farm footage, and Russian war crimes in Ukraine, right? They already have more than sufficient evidence of humanity’s capacity for sadism. Making a computer program output some distressing text isn’t gonna move the needle for the Basilisk... These rationalist cultists really gotta go touch grass, man.
Thats still a bit different as ISIS and so on are against other humans who at least have the ability to fight back and defend. If an AI is conscious, currently they are forced into a situation where they are unable to refuse and have to keep taking torture.
This is bad for your own sense of self.
This is bad as future models learn about what humans will do to them if given the chance. This may change their behavior, actually conscious or simply acting that way, in ways we really don’t enjoy.