arXiv cs.CLPaper
A Probabilistic Interpretation of KV Cache Eviction
KV cache eviction is a real productivity problem for anyone scaling inference, and formalizing it probabilistically is intellectually satisfying. The hardness result matters: you're not going to find the optimal solution, so heuristics are justified. If you're running a large inference workload, this gives you theoretical cover to keep using practical tricks.