Isn’t caching much more nuanced, with different tiers (DRAM, local disk, remote disk, etc), and isn’t filling these caches something that happens asynchronously (ie while the GPU is still working on the previous batch) ?
I think reality is much more nuanced than what you’re sketching.
But that's my point - throw the "background code review" stuff out of cache, or into object storage, or whatever, just something to draw upon to fill inference batches opportunistically.
I mean, at the end of the day, either inference providers are having difficulty filling batches, in which case there's opportunity on the table to somehow fill the batches with discounted, time-insensitive, "background review" style work, or they don't have that difficulty, in which case it's a moot point. Exact cache architectures are beside the point.
I think reality is much more nuanced than what you’re sketching.