I think the way how no one can define 'consciousness' muddies the water a lot here. Can something be conscious and lack the biological structures that create visceral pain, psychological suffering, a will to do things on your own, or death - the actual properties that make things like slavery and torture possible? I think it can be, I don't see what makes human-like pain and reward perception mandatory for 'consciousness'. And I don't think LLMs have the traits I listed above.
If the model lacks capability for human-like pain, I fail to see why it would have a capability to be offended by someone being rude to it. One seems way more foundational than the other, and being able to experience emotional pain is part of what makes us human in the first place.
If you believe it lacks those traits but is conscious, you believe in a definition of consciousness that is meaningless, as “human-like” is the only reference against which we can measure consciousness. A good thought experiment is to try and imagine a consciousness that is radically different from humans, and whether we would be same to even register it as “consciousness” at all. (Incidentally, pretty much all portrayals of aliens in sci-fi give them a human-like consciousness, just with varying degrees of twist on top of it to make it feel engaging.)
Whatever people think consciousness is, I don't think most would agree that LLMs have it. The main point is that the capacity to feel pain is what's central here, not another rehash of the great consciousness debate. I can't define it well enough myself to say if pain and other biological signals must be central to consciousness, I can imagine a hypothetical alien that has its own stimuli and could be seen as conscious, but the point is - consciousness isn't what should be discussed here, I think.
