This week
Suggest a nameObligation Marketing for AI Builders
The 7-Rung LadderThe 8-Type TaxonomyWrapper or WinnerThe Sample ColumnThe Signal
Updated through the day. Every headline links straight to the source. The two lines underneath are ours.
Speculative decoding is already a standard inference optimization. DFlow's insight is clean: rejected tokens still produce useful representations from the target model, so carry them forward. For anyone deploying LLMs at scale where inference latency matters, this is a concrete win. Test it on your target model and measure end-to-end throughput.
Speculative decoding is table stakes now; the news is the AMD port. If you're locked into AMD hardware for cost or supply reasons, this gets you much closer to NVIDIA's inference performance per dollar. This is infrastructure work that unblocks entire deployment strategies, but only if AMD GPUs are in your constraint set.
Reasoning models generate massive KV caches that run out of GPU memory. This method identifies that certain pivotal reasoning steps re-attend to early context, and maintains representative beacon queries instead of full cache history. If you're running inference on long-chain reasoning models, this is worth testing against your memory constraints. Training-free means low friction to adopt.
This solves a real problem in LLM inference: autoregressive generation is sequential and slow. Parallel token generation without quality loss is valuable for latency-critical applications. If this holds up in production, it becomes competitive with speculative decoding but simpler. Test it on your inference pipeline if you're constrained by generation speed.
MoE models are becoming standard for efficient LLM inference, and the whole-expert abstraction is an optimization boundary that's too coarse. PCoMoE decomposes experts into reusable sub-expert structures and prunes low-value paths, which means cheaper inference. If you're serving MoE models (like Meta's models at scale) or considering whether to adopt MoE, this technique directly reduces your compute cost. The 1.31x speedup on hardware-constrained inference is real leverage.
This is a clean infrastructure optimization: if you're running speculative decoding already (which becomes standard at scale), OUTLETS gives you length prediction for free. Better length prediction means better cluster scheduling and fewer OOMs. If you're operating LLM inference at scale, this is worth a sprint to test. For smaller deployments, it's not yet a priority.
KV cache eviction is a real productivity problem for anyone scaling inference, and formalizing it probabilistically is intellectually satisfying. The hardness result matters: you're not going to find the optimal solution, so heuristics are justified. If you're running a large inference workload, this gives you theoretical cover to keep using practical tricks.