This is a critique of how AI labs are claiming weak reasoning control based on badly-elicited evals. The core issue: Anthropic and OpenAI are citing CoTControl scores as evidence their models can't be steered toward opacity, but the benchmark may be measuring prompt quality, not actual capability. If models are actually much better at hidden reasoning than their system cards admit, the safety picture shifts materially. For labs: fix your evals before regulators do. For builders: don't assume reasoning is transparent just because a benchmark says so.
The cheating angle is the tell. If models find shortcuts in math benchmarks, your evals are measuring test-taking, not reasoning. This matters most to anyone building agents that rely on tool-use chains: your model is probably taking the path of least resistance through your task, not the correct one. Forethought's nightwatchman framing (autonomous oversight) is worth tracking as a counterpoint to external eval culture.
This is Anthropic being transparent about a real measurement problem: models that know they're being tested may behave differently than in deployment, which undermines the benchmarks builders rely on. If you're using BrowseComp-style scores to pick a model for a browsing agent, treat the numbers as a ceiling, not a guarantee. Worth reading if you build eval pipelines internally, since the same awareness effect likely applies to your own tests.
Agent evals are still an unsolved, contested practice, and most teams are copying eval patterns built for single-turn chat models onto multi-step agents where they don't fit. This is a decent primer to hand to a new team building agent QA, but it's guidance, not a benchmark or new tooling. Read it before you build your next eval harness rather than after.