Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence
The core finding is uncomfortable: the factors LLMs say drove their decisions often don't match what actually changed the output under intervention. This matters if you're using LLM explanations to monitor or debug agent systems. An operator trusting the named factors to diagnose errors will often be fooled. The paper suggests dedicated models are better, but for now, treat LLM explanations as decorative rather than diagnostic. If explainability is part of your compliance or safety story, you need to read this.