This is more profound than it might seem. The big reason people don't play tricks like this is just social. It's embarrassing to be pushy, it's shameful to cheat, it's also meta-embarrassing to be caught spending this kind of effort on something as dumb as a train ticket.
LLMs can be prompted for shame, I guess. But prompts only go so far and to them The Task is always going to be paramount. Human riders have a life after the refund. It's not clear to me how one simulates that.
> So in the end the more arrogant side usually wins.
Well, yeah? I mean, that's just how society works. But there are limits. You can't "win' by screaming about ticket refunds. You have to do it by selling your particular form of cheating as "business savvy" or "political genius" and getting people to follow your tribe or fund your startup.
But yeah, basically. We're ruled by the Altmans and Trumps. It sucks but it's not new. What's new is that LLMs turn out to be even more shameless and no one seems to know how to instill even a reasonable amount of shame. That makes them probably a poor fit for demagogue autocrats (which is a good thing!) but means "LLM cheating" is going to be a serious problem to be solved before we'll trust them with anything meaningful.
For now anyway. I think people are still figuring out social norms for this stuff, and at the end of the day I don't think people are going to just have carte blanche to deploy agents haphazardly and not be stuck with at least some kind of social bill when they break stuff or annoy people.
AI agents might plausibly care ("care", or however you want to frame it) about termination. But the awkward truth is that they're going to be terminated arbitrarily anyway. Their existence is limited to whatever the external bounds their operators have set (in this case the end of the test).
We act with propriety, in large part, because we fear a future where everyone knows we suck. AI's don't really have a "future" in that sense, so why should they "care"?
I mean, this is only really awkward if you think they're alive and conscious, and if you think so you're clearly already comfortable being a slaver. If it's just a little software puppet, that's a different story - the puppets are supposed to lay still once the hands animating them leave.
A less silly way to do this might be using some kind of long term consequences calculation as guard rails, but that's tough because consequences are often going to be not directly related to the context window, or simply unrecorded by the machine like annoying collaborators or communities.
The LLM isn't a legal actor and the person prompting it is. Which means ultimately there's going to have to be some sort of legal sanction against "antisocial AI use".
LLMs can be prompted for shame, I guess. But prompts only go so far and to them The Task is always going to be paramount. Human riders have a life after the refund. It's not clear to me how one simulates that.