I’m guessing that there’s a system prompt at the top telling the model about its reasoning budget. So when you switch reasoning effort it busts the cache.
The KV cache is probably offloaded to RAM after a generation is complete. Then it is pulled back into any rack in the data center that has the model you’re using.
Prompt caching only works based on the prefix. Let’s call your output Y and the low reasoning prompt A and a medium reasoning prompt B.
Previously you were at A+Y. Switching to medium reasoning makes it B+Y. There’s no prefix which can be cached, so the entire B+Y needs to be reprocessed.
