arXiv cs.LGPaper
The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction
KV-cache compression is a real cost lever for long-context inference, and KV-Surgeon's joint optimization beats greedy entry-by-entry methods. If you're deploying long-context models or building RAG systems with large fixed contexts, this is worth benchmarking against your current compression approach. The gains are material across multiple domains.