arXiv cs.CLPaper
Decomposing LLM-Judge Uncertainty to Target Expert Labels
The insight is real: knowing whether an LLM judge is ignorant versus whether experts genuinely disagree changes where to spend labeling budget. But the practical payoff depends on your labeling setup. If you're already running LLM judges at scale and can afford Bayesian decomposition, this tightens your data collection. Otherwise it's an optimization on a workflow most teams haven't yet built.