The Signal

Everything that matters in AI, with our take.

Updated through the day. Every headline links straight to the source. The two lines underneath are ours.

research1156 agents716 infra561 labs317 culture268 capital253 regulation233 claude175 openai149 safety103 security89 evaluation77 reasoning61 alignment52 benchmarks50 interpretability48

arXiv cs.CLPaper15d ago

Post-hoc Alignment of LLM-judges to Human Judgment Distribution

This hits a real problem: LLM-as-judge is cheaper than human eval but usually evaluated only on hard labels, missing the fact that many tasks have legitimate disagreement. The finding that models struggle with soft-label prediction is important if you're building eval systems at scale. NAPHA is practical and lightweight, worth testing if you're already using LLM judges for ranking or filtering.

research evaluation llm-judges

arXiv cs.CLPaper21d ago

Localize-Then-Decide Guarantees for LLM Judgments

LLM-as-judge is now core infrastructure for eval pipelines and RLHF-style tuning, and this paper fixes a real failure mode where confidence estimates get distorted as candidate pools grow. If your eval harness relies on LLM judges picking among more than two or three outputs, this conformal-prediction-plus-calibration approach is worth adopting over naive confidence thresholding.

research evaluation agents llm-judges