Enterprise is not freaking out because DeepSeek 4.1 Flash does not actually occupy a spot on the Pareto frontier for non-coding enterprise workflows. We see this at my employer, focused on non-technical knowledge work. Luna 6 and now Haiku 5.5 are both very competitive if not better on all axes that we care about

Letting a model manage its own context is very bitter lesson-pilled

Biggest challenge is you will get a much lower cache hit rate if you frequently edit the agent's context/prefix, so this can not be implemented efficiently via e.g. the Anthropic API.

This ^ can be solved in principle but likely requires modifications to the transformer architecture and definitely to serving infrastructure

See related: "KV Cache Rules Everything Around Me": https://www.completeskeptic.com/p/kv-cache-rules-everything-...

If you read the paper, they had a solution for this.

and if you didn't read the paper, they still have a solution.

this is done on the model side so it is invisible to the caller. all of deepseeks kv cache compression stuff is basically the same thing. it doesn't really put the entire context into kv cache and does various inference side stuff to decide what to actually use as context.

True facts. Cline does this with auto compact and it's so dumb. At the start of the session it reads the plan.md. during compaction, it forgets it. Same with its own output. It'll tell me the list of steps to take. By step 4 it needs to compact. So I prompt to do step 4. It spends 15 minutes searching the codebase and comes back that it couldn't find such a step. So dumb. So so dumb

is kvcache a prematural optimization?

Not having it would be like a text editor that loads your entire document from disk and re-saves it every time you press a key. On a floppy drive.

It's pretty much required to make it work with large contexts, so probably not

Do you even understand what you're suggesting? Do you understand how the data flow would look like without a kv cache?

Java spent decades making pointer-rich identity semantics cheaper, before increasingly recognizing that many things programmers call “objects” are actually values.

We may spend a decade making persistent token-level KV state cheaper, before recognizing that much of what we call “context” should actually be transient, compressed, reconstructed, or represented in an entirely different state space.

KV cache risks turning an optimization of today's representation of history into tomorrow's semantics of memory.

Shocking to me how little interest the general public has in mechinterp given the alien capabilities demonstrated by LLMs - this and the subsequent transformer-circuits.pub publications will be seen as classic, foundational work in a few years

Likely from Palmer Luckey, who coined the term 'Chinese instruction manuals'

Only the ones that beat expectations

formal verifiability e.g. vi Lean

Even Lean has bugs.

> AI "Proves" Collatz Conjecture with Lean 4 Bug

https://news.ycombinator.com/item?id=49101465

of course it does, but it's still the best thing we have