The Signal

Everything that matters in AI, with our take.

Updated through the day. Every headline links straight to the source. The two lines underneath are ours.

agents152 research125 infra71 claude63 capital52 labs51 safety41 culture37 security35 anthropic30 regulation30 openai24 evaluation18 enterprise13 reasoning12 robotics12

arXiv cs.AIPaper6d ago

DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models

KV cache compression is a real scaling lever for deployed models. Getting 55x compression with under 2 points loss means longer contexts become economical on constrained hardware. The trick is conditioning compression on prompt tokens and neighboring positions, not treating the cache as generic data. For builders: this directly reduces serving costs for long-context applications. For infrastructure teams: this is a production-ready technique worth implementing.

infra efficiency long-context kv-cache

arXiv cs.LGPaper7d ago

BellowsKV: One Latent for Many Layers, Expanded on Demand

KV-cache memory is the real bottleneck in long-context and agentic workloads. This attacks it at the layer level by noticing that you can compress more aggressively when you exploit structural similarity in inputs. The method is practical for serving teams trying to squeeze longer contexts onto the same hardware. Test this if you're hitting memory walls with multi-turn or tool-use agents.

research infra kv-cache efficiency

arXiv cs.LGPaper14d ago

The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction

KV-cache compression is a real cost lever for long-context inference, and KV-Surgeon's joint optimization beats greedy entry-by-entry methods. If you're deploying long-context models or building RAG systems with large fixed contexts, this is worth benchmarking against your current compression approach. The gains are material across multiple domains.

infra inference kv-cache optimization

arXiv cs.LGPaper18d ago

When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

This is infrastructure troubleshooting with real teeth. A remote KV cache sounds efficient but the paper shows it can wreck p99 latency: 17.3s instead of 3.96s. The cost model (hit pays off only above B* = sR threshold, plus fixed per-hit engine cost) gives operators a way to decide when remote tiers make sense. If you're scaling long-context serving across multiple engines, this reframes the cache strategy conversation.

infra serving llms latency kv-cache