arXiv cs.LGPaper
Length Generalization for Transformers via Compression
This is genuine theoretical progress on a hard problem. The practical question it settles is whether transformers learning length generalization require impractically large datasets or not. The answer, based on this, is more optimistic than before. If you're training models to extrapolate sequence length, this tells you the sample complexity you're facing is actually tractable.