arXiv cs.AIPaper
How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
This is mechanistic work on a real model architecture that's in use. The finding that residual streams remain largely separate until late layers, with concentrated read/write patterns, tells you how to think about where computation actually happens in models with expanded pathways. Useful for anyone working on model interpretability or designing efficient architectures.