it's confidential compute, it's open source and you can verify yourself that it's not reading the prompts

Open source doesn't matter if someone else is running it, right? They can change it?

As long as the prompt is not encrypted at some point, and I don't think LLMs can run on encrypted prompts, then it can be read.

With confidential compute / TEEs you can guarantee that the code is running, it's verifiable with remote attestation

So where does the guarantee stop? At the GPU driver level? Firmware level?

What if the GPU has a custom bios flash that somehow logs the unencrypted prompts?

goes all the way to keys owned by Nvidia and Intel

SGX has been cracked

depends on your threat model