arXiv cs.AIPaper
vToken: Token-Level Virtualization for Reclaimable KV Caches
KV cache fragmentation is a real cost center for anyone running high-throughput inference, and this closes a gap that block-level eviction schemes like PagedAttention leave open. If your serving stack uses vLLM with token-level eviction policies, this is worth testing directly since the memory reclamation gains translate straight into cost per token served.