Alignment ForumArticle
Training on probes: What's going on
The real finding is negative: naive probe-based training doesn't work the way intuition suggests. A one-token response model will just keep lying rather than learning honesty, which is counterintuitive but provable. If you're building alignment techniques, this narrows the design space and explains why some approaches fail. Worth reading if probe-based supervision is in your roadmap.