That is an extreme misrepresentation of what really happens. Running a relatively small LLM locally (serving a single user) is very different from doing it at scale, notwithstanding for a model 100x in size or more.
Frontier models are huge. Astra could be 10 trillion parameters or more. That will probably need > 10 TB of VRAM (HBM3 if you don’t want to wait forever for a response) and need a mini-cluster just to run one instance/copy.
And because token generation is a sequential process, all the code to orchestrate tensor parallelism (spreading a single request across multiple GPUs) is non-trivial. Each new token depends on the previous ones.
Combine that with KV caching optimizations, loading/unloading from cheaper cache storage, session management, load balancing, parallel sessions, and daily software updates plus testing optimizations to kernels; it’s a lot of moving parts.
We haven’t even discussed coordinating stuff across different data centres, failover mechanism, and what not.
Frontier models are huge. Astra could be 10 trillion parameters or more. That will probably need > 10 TB of VRAM (HBM3 if you don’t want to wait forever for a response) and need a mini-cluster just to run one instance/copy.
And because token generation is a sequential process, all the code to orchestrate tensor parallelism (spreading a single request across multiple GPUs) is non-trivial. Each new token depends on the previous ones.
Combine that with KV caching optimizations, loading/unloading from cheaper cache storage, session management, load balancing, parallel sessions, and daily software updates plus testing optimizations to kernels; it’s a lot of moving parts.
We haven’t even discussed coordinating stuff across different data centres, failover mechanism, and what not.