The insight here is structural: template-based safety training doesn't generalize to adversarial input spaces because it's supervised by fixed outputs, not by reasoning. Self-distillation forces the model to learn why it should refuse, not just when, by training on the distribution of prefixes the attacker actually explores. If you're shipping a safety-critical model, this is worth testing against your threat model because it handles a known failure mode that existing defenses don't.
Clean and potentially high-impact for teams managing multiple fine-tuned models. If you have separate specialists for math and code, you can now fold them without gradient steps. The catch is that rollout quality matters more than the solver, so garbage specialist behavior spreads to the merged model. Test this on your specialist stack; if it works, you cut serving costs and latency. If it doesn't, the reason is instructive.
Training economics matter, and 30% VRAM savings with near-zero overhead is material for builders scaling large models. Sparse activations are becoming standard with recent architectures, so this timing is right. The custom Triton kernels mean implementation friction, but the payoff justifies it if you're training or fine-tuning at scale. Test on your specific hardware and model.
Games are a meaningful test for agent generalization that's harder than desktop automation. Player-1 shows that synthetic training data can close the gap where real demonstrations run out, which matters because real game footage is expensive and the state space is vast. If you're training agents for complex visual navigation or multi-step planning, this synthetic-experience pipeline is worth studying.