arXiv cs.CLPaper
Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs
The core insight is sharp: a compressed model's bfloat16 checkpoint is itself an approximation, so healing against it compounds error, while distilling straight from the original full-precision model avoids that. Anyone running structural compression plus quantization pipelines for cost reasons should look at this before defaulting to standard QAT, since the reported gains, matching bfloat16 on 7 of 9 benchmarks at a quarter of the memory, are the kind of number that changes a serving cost model.