One of the main things I've noticed is that several of the techniques are well intentioned for other reasons. For example, when we debug low cache rates with users probably culprit #1 is usually some memory system. Many of those memory systems are trading a stable prefix for the ability to augment the current turns request with some new memory data. For some -- this may be the right trade-off, but the impact is slower responses times due to the extra prefill compute.
So when a memory system promises better LLM performance the other trade-offs aren't necessarily well understood (at the expense of caching).
Features like OMP trying to clean up history to shrink context I think are also well intentioned. But I found these trade-offs interesting hence wanting to share.
I recall this company now for their unintuitive/abstracted pricing model I have no interest in, but it does make clear why they would be helping customers with cache hit rates, that's going to lower bills on a per-request basis
I agree with you, accuracy of answers and the ability to complete task is the most important thing. But once a system can achieve a goal, throughput and cost come into consideration. In this regard, I think cache rates help make things both faster and cheaper.
So when a memory system promises better LLM performance the other trade-offs aren't necessarily well understood (at the expense of caching).
Features like OMP trying to clean up history to shrink context I think are also well intentioned. But I found these trade-offs interesting hence wanting to share.