Exactly, imo it’s not even that cheap if you look into perspective and consider the fact that providers could subsidize the cost of cached input tokens to virtually zero if they would allow for a more flexible API (e.g. tree of message blocks instead of chain). Most of the cost is the infrastructure around keeping KV caches, estimating their lifetimes, etc. When mist people just want to run one context block with multiple subsequent variants of a second block in parallel. I still stand by my statement.

That's an interesting point, if you send a batch with a shared prefix you basically only end up paying for the sequence length difference effectively.

There is still some minor memory bandwidth issue on outputting more tokens, but the truth is that if you process e.g. 16 messages at once you wont end up being much slower than Jev even though you have to perform several autoregressive passes.

"you wont end up being much slower than Jev" -- I would even go as far as saying that "you will end up being Jev".

With this cost, does it perform the same quality and speed as Jev?

I'm quite interested in this; my current understanding is though that Jev is great when scored with response quality and latency metrics.

Seems to be slightly higher quality and substantially faster, although this compares remote API vs local deployment.

The prevalent idea of Jev's superiority in price, speed and accuracy seems to come from TypeSafe's marketing and their, I'd say even bad faith, benchmarking. In independent benchmarks the relative numbers tend to be very different.

https://github.com/Mushroom-Systems/lichen