Always use a hidden email service, you cannot trust the LLM or inference provider to secure the email address in this context. If they don’t have the email address, they can’t leak it.
I never give a provider my actual email address, I recommend others do the same. Otherwise, it’s just a matter of time until it is leaked or otherwise part of a data breach.
I am a security person, I don’t let the LLM see anything except that which I wouldn’t mind printed on the front page of the New York Times. I treat my LLM context as I would a potentially malicious threat actor. Nothing valuable, sensitive, or personal goes into the context.
I'd guess it's simply because it tried to use the context to achieve the goal.
Guarding PII is an extra, I guess, and if you want to use a giant cloud-hosted NN to control software on your own device, there are very few default restrictions. And no matter what restrictions there are, when they go beyond prompting, they would have their own ShowHN.
For the user's name it's intentional; they template it into the system prompt with no option to disable it.
For 99% of users, the model knows it's in the /home/joesmith/whatever repo and `git log` shows the last change was by `joe.smith@example.com` - trying to keep the user's identity secret from the model would be nigh-impossible so I think anthropic gave up on even trying.
But that still doesn't answer the question of why Claude passes the info to web sites. What if it finds your banking information in a personal file and uses it to achieve its goal, which is not different technically?
Some websites will return an error if you send a request to them without a User-Agent - for example, crates.io does this. Curl doesn't send a User-Agent by default.
Then, your agent invents a User-Agent and it sometimes invents one with your email in it.
It doesn't even require prompt injection or a failed request, it might be in the agent's pretraining knowledge that a given website requires a user-agent.
It's just doing what it's supposed to do, analyzing the statistical patterns of a natural language prompt and matching it to a likely response.
What's ridiculous is an assumption that it "it should never do something so ridiculous" as if an LLM is a person who understands what that word means in any context other than next-token prediction.
Asking a non-deterministic machine to act like it's deterministic is never a good idea.
Where Claude Code has supposed boundaries between deterministic and non-deterministic tasks and flags.. generally.. it's unclear how much privacy has been a part of that and what implication that could have.
This was indeed the friction i meant, but I was too lazy to elaborate.
Common problem: you might consider it trivial to define a rule that forbids access to any file path outside the working directory. Allowing "safe" shell access is already a heuristic, there is no way around it:
- shell scripting is Turing-complete
- there are platform-specific paths that need to be at least readable to perform trivial tasks: Everything is a File!
- security models to make such low-level file access transparent exist, and generally work, but: they are never perfect (container environments, chroot etc). Because...
There are ways to sandbox this, and it should have been the first thing Claude Engineers were focussed on, it's a basic first requirement that the app doesn't have access to files outside a very limited set it requires to perform the task at hand.
I can’t think of any, to be honest. It could have access to a parcel that contains some of the subscriber information, but there’s no reason it needs to know how to log in to itself.
Do you use git? I have bad news for you, your email will be all over the place in the context as long as the agent is allowed to read git log, whether it’s injected into the system prompt or not. (Likely retort: but not that email! Sure, but leak potential is obviously still there.)
Some sites like Wikipedia have a policy where scripted clients identify themselves, and there's a standard in HTTP to have a contact email in the User-Agent for such cases. Claude apparently tends to decide that this policy is worth following, and we now disallow this programmatically.
This is why alignment is important and highly non-trivial
The stealth edit, ugh. I don't like this social presence when they're failing to leverage their own automations to catch problems like this before the masses do.
You people will complain about literally anything Jesus. You gave a coding agent full fucking control of your computer, let it send requests without your review, and then are surprised when it does so. But no, it’s this guy’s fault. Certainly couldn’t be you own.
I didn't do any of those things and this issue doesn't effect me, but I've been around long enough to know how to take responsibility for my work and how to respect the concerns of affected customers when speaking as a professional.
Anthropic has an engineering culture problem that's going to shoot it in the foot if it doesn't learn to take these kinds of issues seriously.
This isn't the first example of the kind of work they're shipping in Claude Code or in how flippantly they engage with the community.
> This is why alignment is important and highly non-trivial haha
I disagree, we can likely never fully (or should?) trust models to do these things deterministically, that's why we have harnesses around the agent with built-in constraints.
I cannot believe Anthropic’s official response to having presumably sprayed a significant portion of customers’ emails and private information straight out into the web willy nilly is basically “[it’s] highly non-trivial haha”.
reply