This is what I currently do, but my software uses docker and docker mounts act as a bypass for the file system restrictions, plus docker processes started outside the sandbox allow network proxy escape.
Currently, I don't allow the agent access to docker, start docker myself, and then do short-lived sandbox-free sessions when the agent needs to do things that interact directly with docker; but that's annoying.
Docker is always a pain to use and this way I don't have to re-install everything a billion times for every different project.