That table assumes cache hit rate of 95% or better. Am I understanding this correctly that people really are doing such repetitive prompts (compared to each other, across the concurrent user base at that time) that only 5% or less need actually be computed by the intended LLM?
Every tool call is essentially entire prompt so far sent again with the response and that's why cache rates are so high for agentic workloads.
This really bites when using expensive models since most models are 1/10 for cached input.
That is shocking. Is it per-token I wonder?