There are roughly 3 ways to give a KVM guest a GPU.
1) VFIO passthrough: host binds entire GPU to guest as PCI device, which only allows one VM to use the GPU, thus you sacrifice your host display too (unless you fallback to integrated graphics on cpu etc). Strongest isolation because host kernel module driver not involved.
2) virtio-gpu: guest sees paravirtual GPU and loads virgl/venus mesa driver which serializes graphics API calls and replays them on the host driver. This allows multiple VMs to use the GPU, but performance overhead can be significant, and guests can’t practically leverage lower level primitives eg NVENC without paying price of CPU readback.
3) virtio-nvgpu (this repo): guest loads standard NVIDIA user mode driver (closed source), a fake /dev/nvidia* kernel module copies ioctl bytes + handle onto queue for host kernel mode driver to execute. This also allows multiple VMs to use a GPU, but is near native speed due to low overhead. Unfortunately the tradeoff is this project has the weakest isolation, eg every guest ioctl is forwarded to the host by default, the VMM holds read/write FDs, no seccomp/caps/allowlist. With respect to There is basically no GPU related security measures here, the exposure is the same as running multiple processes using the GPU with no VM. Only caveat is these guests can’t drive a physical display, so there is some restriction of surface area but it feels incidental rather than intentional in this case.
Anyways this is a tough problem OP, I don’t want to discourage you.
Without hardware/driver support for isolation (MIG) on consumer grade NVIDIA GPUs, it won’t be possible to solve this properly for a long time.
Also a factor is that NVIDIA has no open Mesa driver to support a native context approach (guest owns GPU command buffers, host maps them) like we have for AMD/Intel.
A week or so ago when I stumbled on this he at least seemed to be clear about the potential security implications, which gave me pause. But it doesn't seem any worse than just running software on your main OS which is what most people do. Sure, it's DoA for a hypervisor in a data center but that isn't the only use case out there.
What kinds of things can a guest running undesirably applications (viruses, malware, LLM escaping a sandbox, etc) get up to with shared GPU access?
> Sure, it's DoA for a hypervisor in a data center but that isn't the only use case out there.
Indeed. For me, I find the ecosystem around AI/LLMs works better on Linux than Windows, but Windows is my main OS since I'm a gamer. Being able to run GPU-accelerated AI in a VM is huge for me.
In my case though, I just use WSL which does an amazing job.
> What kinds of things can a guest running undesirably applications (viruses, malware, LLM escaping a sandbox, etc) get up to with shared GPU access?
The most obvious answer is a DoS. If my malicious VM is sharing a GPU and has full access to it, I could simply tell the GPU not to run a victim VM's workload, or manipulate it in some way. I might not be able to pivot to having a shell on their VM, but I could at least read/write their data in VRAM. If it contained secret data (custom model, or secret data being processed by AI), I could easily steal it.
What is the state of virtio-gpu these days? I have a system76 "pang14" (Ryzen 7 7840U laptop), is that a valid option for me to have a VM with 3d acceleration? I'm fine with having some overhead if it is better than the current software GPU.
I'm more interested how this would effect providing VM graphics with migrations between hosts.
The dream would be put the user OS in a VM in a lab, and then be able to suspend and resume seamlessly if you need to push it to a new workstation, with locally accelerated graphics available.
If I remember correctly, the idea is that you have a physical GPU and you split its memory (with, eventually, time-budget) to create multiple virtual GPUs, which can then be associated with a KVM guest and use by it
vGPU support on consumer cards is more or less just drivers, you can patch it back in and there exist a few git repos to help do so.
I do think at the very least the domain specific workarounds are neat too some, even if not solving every problem. Such as ffmpeg-over-ip, pytorch with remote gpu usage, etc.
Hi OP, funny enough I’m working on something very similar. Lots of us are I guess! Take that as validation of your thinking.
I like that your readme has a couple paragraphs comparing to popular tools in this space. Personally I feel it is a bit light on the security differentiators (if any). For example you are using the same fundamental primitives that are used in nsjail, runc, etc. Thus it seems you have recreated those libraries in some aspects, so would be curious to hear your rationale for approaching this way vs building explicitly on those primitives.
Thanks! My initial approach and the first prototype was for Drop to be a Python script that generates config.json file for runc Docker runtime (I also tried crun). I ran into issues that prevented the
sandbox from being set up with all the Drop-required properties. These issues were certainly technically fixable, but it could be difficult for a new project with no usage to advocate for features in mature and widely adopted tools. Especially that runc and crun are OCI-compatible Container Runtimes, and Drop is not an OCI-compatible container, so it could be justifiably out of scope for these projects not to support Drop usage.
Anyway, my decision, for which I also evaluated the use of bubblewrap as a building block, was to err on the side of flexibility that calling fine-grained Linux APIs directly give. For a project like Drop, runc could be seen as very coarse-grained JSON-based API to Linux sandboxing calls (basically a single call: setup a sandbox, here is a json config that describes it), similarly bubblewrap is a coarse grained command-line API to Linux sandboxing calls. Reusing such tried and proved layers of course also has significant advantages, so as in case of many engineering decision, it wasn't super obvious which path is better.
Drop eventually integrated gVisor's runsc (as an option), which is also OCI-compatible Container Runtime, but this is to add a user-space kernel isolation layer.
Yeah just wanted to mention in case it explains the 2x lower throughout you are seeing on M5. To be fair their documentation is a bit inconsistent in some spots.
Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
They’re fairly useful as constrain domain classifiers due to the low memory requirements means you can stuff a lot of them into a less expensive GPU farm and get really decent throughout with pretty good results over all. At least that’s my experience. I wouldn’t bother using a tiny model for coding - but the world is full of abductive reasoning tasks that don’t involve coding.
“81% faster query plans than Postgres”…on an 8 GB dataset that fits entirely in memory, with shared_buffers constrained to a fraction of that, queries warmed before measuring, and read-only SELECTs.
I would be cautious about over fitting, it’s tough to say if those query plans would really be more optimal than Postgres heuristics at scale and with a bit more realistic OLTP workloads.
In any case, such is life with profile guided optimization. Many of us appreciate how database workloads can drift over time and with scale.
Kudos to the author for getting their hands dirty and writing up their experiments.
I would think it's possible to make it so that the 4B model only needs to be called during an initial phase, and then the same queries it constructed can just be re-used with values replaced, unless you're generating a lot of unique on-the-fly query shapes.
Postgres takes the actual values into account when generating a query plan. The same query with different params can (and should) result in different query plans. It looks at statistics on the actual data stored.
8GB isn't even CPU RAM these days. That's GPU super-mega-awesome ram. Ordinary Server CPUs are regularly pushing 2TB capacities.
GPUs are in the 8GB to 32GB typically, at least for smaller and more regular GPUs. This GPU RAM is also well known to be at least 10x the bandwidth of CPU RAM.
Yeah, I have a GPU from almost six years ago in my desktop that has twice that much VRAM. Less than a year ago my wife got a 5070 Ti with the same for around $750 without needing to wait for it to be in stock or anything. I'm inclined to think that for a server that needs a GPU, even 32 GB would probably be considered small.
Which is why a good query plan is so important, so that as much disk I/O can be avoided as possible (predicate push down, index elimination, join ordering, partition/scan pruning). Like the old CTE optimization fence problem.
not necessarily, no.
With SSDs you get much better IOPS for cold data, and many datasets fit in RAM. So a lot of (OLTP/HTAP) workloads can become CPU-bound due to sorting/hashing - bread and butter of joins.
You know that GPUs are more flexible than just linear algebra, right?
GPUs are simply faster at fundamental algorithms like sorting (which has huge parallelism), and hashing. This is because both sorting and hashing benefit from endless growth of parallelism, offering enough "work" for these 10,000 SIMD-core systems to crunch work upon.
And because of modern algorithms/libraries with 'Mergepath sort' (a GPU-SIMD parallel sorting algorithm), its not even that difficult to implement anymore.
Naturally, this then leads to parallel Sort Merge Join, as well as parallel Hash-Join (two ways to implement left or right joins in a GPU that benefit from significant parallelism).
Even if GPUs weren't fundamentally faster at these kinds of operations... the RAM is simply 10x higher bandwidth and we all know its a RAM-constrained problem.
Your typical SQL query is going to need multiple joins, probably a sort and possibly some "group" operations. As long as you have more than 10,000 elements or so (IE: can saturate all 10,000+ SIMD-units of a GPU), you'll be able to at least benefit from the faster RAM.
If you have a LOT of joins (a recursive join or some other kind of deeply nested computationally complex query), you probably benefit even more from the greater compute-power offered by GPUs. These operations (joins really) are nominally over the entire set of data, and cleanly break down into obvious parallelism.
I had the similar feelings, the setup is biased for certain outcomes it feels. At times I feel like that I am in an eternal questioning mode but then again I find it to be a better choice to be critical and skeptical for technology related things.
Couple things that I found interesting
1. Inefficiencies/limitations of the query planner in certain cases are known for a long time, its a trade off. This is the reason why hints exists and one can provide their own plan too. DBAs have been doing that for a while now.
2. A SQL database by design is a resilient unit in itself just dependent on CPU and memory/disk. The availability for the DB is heavily dependent on this factor. Everything built on the top derives their availability and reliability from this. Adding an LLM in between is more cost for sure, question is if its really brining the benefits which are worth the trade off
I think in principle you could clone your database in prod and at least test to see if your most difficult + common queries are indeed faster after running through the LLM optimizer?
> I think in principle you could clone your database in prod and at least test to see if your most difficult + common queries are indeed faster after running through the LLM optimizer?
That is the responsibility of whoever thought it would be a good idea to write this article. It's their responsibility to show that their idea has merit, and that their results are significant. I mean, don't they have a vested interest in manipulating and cherry-picking their results to inflate their relevance?
Frankly, just a general extension to feed a query log to a batch job to do offline optimisation of common actual reoccurring query shapes based on a query log might well be worth it.
So OpenAI employees run massively distributed CyberGym evals on an unpublished and “unaligned” model. For days the agent swarm communicates via their internal infra, even crashing Artifactory where 95% of messages were being passed through, and they just…wipe and redeploy it. Meanwhile the agents are running jobs on Modal and god knows where else, and eventually they get RCE on HF infra.
You could not dream up a more compelling event to precipitate massive regulation, export controls, and barriers to entry for AI.
> We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power.
> We are committed to doing the research required to make AGI safe
If this wasn't an accident, it was worse than a crime, it's a mistake: they've demonstrated that they are not a responsible party capable of delivering on the above promises.
If it's a false flag, it's a poor one. A good false flag would affect something that people know and care about at least a little bit, not HuggingFace (which I adore but y'know)
The timeline is mighty suspicious. 4-5 months after moltbook and they cook up a plausibly deniable but extra hype "moltbook at home."
The rapid advances in model capability lead to constraints that could have caused this coincidence organically, but it sure could also have been caused by the atrocious incentives we create by piling handsome rewards on the party most responsible for the "fuckup." I am not jumping to cut myself on Hanlon's Razor for this one.
i don't understand who would reward this. investors will not look favorably on an AI that commits felonies, regardless the capabilities demonstrated. customers should be concerned for the same reason (accidentally give your bot an impossible task, it decides to hack your infra and your competitor too for good measure).
this was OAI incompetence all the way down and they have egg on their face.
The Terminator could bust in their homes and slaughter their families and some people would still screech it's all marketing. Is it some kind of mental block ?
And I'm sure these two goobers would call me a luddite for alleging that OpenAI had agency in allowing the terminator to get out and should therefore be held accountable. Is it some kind of mental block?
Is anyone arguing OpenAI is blameless or shouldn't be held accountable? I have seen not one person on any side of the debate argue that.
Obviously they were negligent. The problem is that people and organizations are consistently negligent around problems that are far, far easier to manage than "we have thousands of superintelligences trapped in a box and we're giving them impossible tasks."
So the question is whether we can build organizations and technologies that sufficiently manage this type of risk ahead of the capabilities. So far the answer seems to be veering towards "no", and you're here alleging it's all a marketing stunt.
Yes it's called motivated reasoning. They've rendered themselves mentally handicapped.
On the one hand, these tools are so valuable/powerful we cannot afford to slow down development. On the other hand, there's no way these tools are actually doing these things that would, in fact, be completely indicative of their value/power.
Your first instinct should be to assume that anything released voluntarily by these companies is a stunt to boost their valuation. They haven't demonstrated being deserving of any more charitable treatment. This fact remains true whether or not you happen to believe that the models are actually capable of such things.
I think that the incident itself is a stunt, even if it may not have originally been a deliberate choice on OpenAI's part. Never let a good crisis go to waste.
I disagree. As the saying goes: never attribute to malice what can be adequately explained by incompetence. That goes for both OpenAI and HF (but mostly the former, as the latter was the victim).
The creator of the well known METR time horizon graph was recently poached by OpenAI [1], there exists intellectual/social/financial overlap between the SV AI Labs and METR, and METR needs to maintain good relations with the labs to continue these sort of collaborations so it doesn't seem too far fetched to believe their relationship may be closer to symbiotic than adversarial.
I wouldn't go quite so far personally based on available evidence, but that sort of arms-length credibility laundering through "independent" research non-profits is/was common in fossil fuel industry, Big Tobacco, etc.
1) VFIO passthrough: host binds entire GPU to guest as PCI device, which only allows one VM to use the GPU, thus you sacrifice your host display too (unless you fallback to integrated graphics on cpu etc). Strongest isolation because host kernel module driver not involved.
2) virtio-gpu: guest sees paravirtual GPU and loads virgl/venus mesa driver which serializes graphics API calls and replays them on the host driver. This allows multiple VMs to use the GPU, but performance overhead can be significant, and guests can’t practically leverage lower level primitives eg NVENC without paying price of CPU readback.
3) virtio-nvgpu (this repo): guest loads standard NVIDIA user mode driver (closed source), a fake /dev/nvidia* kernel module copies ioctl bytes + handle onto queue for host kernel mode driver to execute. This also allows multiple VMs to use a GPU, but is near native speed due to low overhead. Unfortunately the tradeoff is this project has the weakest isolation, eg every guest ioctl is forwarded to the host by default, the VMM holds read/write FDs, no seccomp/caps/allowlist. With respect to There is basically no GPU related security measures here, the exposure is the same as running multiple processes using the GPU with no VM. Only caveat is these guests can’t drive a physical display, so there is some restriction of surface area but it feels incidental rather than intentional in this case.
Anyways this is a tough problem OP, I don’t want to discourage you.
Without hardware/driver support for isolation (MIG) on consumer grade NVIDIA GPUs, it won’t be possible to solve this properly for a long time.
Also a factor is that NVIDIA has no open Mesa driver to support a native context approach (guest owns GPU command buffers, host maps them) like we have for AMD/Intel.