is kvcache a prematural optimization?

Not having it would be like a text editor that loads your entire document from disk and re-saves it every time you press a key. On a floppy drive.

It's pretty much required to make it work with large contexts, so probably not

Do you even understand what you're suggesting? Do you understand how the data flow would look like without a kv cache?

Java spent decades making pointer-rich identity semantics cheaper, before increasingly recognizing that many things programmers call “objects” are actually values.

We may spend a decade making persistent token-level KV state cheaper, before recognizing that much of what we call “context” should actually be transient, compressed, reconstructed, or represented in an entirely different state space.

KV cache risks turning an optimization of today's representation of history into tomorrow's semantics of memory.

yep, deepseek

`harness` is a term invented/borrowed to describe a vague vision of applying llm in a more controlled manner, suitable for semi organized, complex work. the sad part is then people shift the attention from the real problem but seeking inspiration from the terminology.

the real problem: is there a general framework to allow llm with limited context size to approach problems, just like a capable newbie integrated into a chaotic code base? how does it work?

I think it's an epistemic problem, and my fork of pi: pie (github.com/wangii/pie)

Is Dario Amodei another version of Trump? https://www.youtube.com/watch?v=RDrfE9I8_hs