This week
Suggest a nameObligation Marketing for AI Builders
The 7-Rung LadderThe 8-Type TaxonomyWrapper or WinnerThe Sample ColumnThe Signal
Updated through the day. Every headline links straight to the source. The two lines underneath are ours.
KV cache offloading is the frontier for long-context inference cost. This work shows that the I/O scheduling matters as much as raw bandwidth, and py-kvcache's preloading strategy is a genuine improvement. If you're running vLLM for long-context workloads, this is worth benchmarking against LMCache. It's infrastructure, not research.
Speculative decoding is table stakes now; the news is the AMD port. If you're locked into AMD hardware for cost or supply reasons, this gets you much closer to NVIDIA's inference performance per dollar. This is infrastructure work that unblocks entire deployment strategies, but only if AMD GPUs are in your constraint set.
Faster inference backends matter for anyone serving open models in production, and native vLLM integration into transformers cuts a layer of glue code teams usually maintain themselves. Worth a look if you're running self-hosted inference at scale, less relevant if you're API-only.