arXiv cs.CLPaper
UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute
This is genuinely clever: you get reasoning depth improvements for free by reallocating the compute budget you'd spend anyway. The 27.52 average downstream score versus 22.99 baseline at 1.3B parameters is a substantial gain, and the method scales. If you're training language models and can afford the engineering complexity of dynamic architecture growth, this is worth implementing. It changes what's achievable at a given compute budget.