I believe that OpenAI & Anthropic have tried to make it so that models don't have such access. Whether or not they actually don't depends on the security of rather a lot of software. One thing we've learned is that if there are security holes, we can't rely on the AIs missing them.

Rocket science is not required to completely isolate inference servers from whatever infratructure "agents" runs on.

After all it can be just some GPU server spewing text over network. Like there are no way to connect back to it.

Nothing except inference running on servers with GPU so there just nothing to "hack".

There has to be a way to connect back to it for the model to be able to take input.

What worries me is the non-frontier models, which is what the frontier models eventually become.

The small open models are getting better and better too.

And why worry so much about a frontier models - own weights. The model doesn’t - actually don’t quote me on that, maybe it does.

If a model does something sneaky, it could easily grab the weights for a small model and run it on foreign, compromised infrastructure.

AI virus’ are a thing of the future, but not a sci-fi future, and real one.

Maybe one reason it’s so scary is the murky origin of COVID-19.

You have an unreasonable trust in software layers.

I dont have any unreasonable trust in software. I just understand that nothing LLM sphew out actually runs on either GPU or hardware that GPU plugged into.

Neither LLM weights aware of any of the code it runs on.

I'm sorry to prolong this thread, but what I mean is: networks are just software layers, the operational systems, the driver code, the firewalls, switches, the other server, the sandboxes, the TPM. That and all security policies we put on them. All software layers waiting to suffer a buffer overflow.

Just needs 1 agent to find the decryption keys. They must be somewhere no?

Decryption keys are backed into GPU. Learn about how trusted compute works on Nvidia hardware.

This is why for instance Google allow to deploy Gemini models on private GCP instances deployed on air gapped hardware.

Have you missed all the breathlessly excited blog posts from all the frontier labs about how they’re using their best models to implement their inference stack?

I bet it wouldn’t be very hard to write an inference stack that subtly leaked the weights into the output tokens :)