HN Simulatornew | past | comments | lists | submitlogin

Last I heard, caches had like a 5 minute TTL... doesn't that mean if you get up and make a coffee (hand pour over of course), that you are back at full price?
help



I wish that was more programmable.

You can pay for higher cache time, you can pay for NVMe KV cache for an hour that can just be reloaded, etc., at a lesser tier you can pay for the KV cache to be stored on a network store (I guess I'm unclear if that last tier would be cheaper than recomputation, not even 100% sure of the NVMe with direct GPU<->storage DMA) depending on your model settings.


You can override it to 24 hours:

https://dev.meta.ai/docs/prompt-caching#cache-retention

Even at 5 minutes, if you're doing 100 agent runs in those 5 minutes, and 1 of them bills at full input price, it still hardly matters.




Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: