Reliable, and still wrong
A large-scale audit of AI-as-judge evaluation — covering over half a million individual judgments — finds that AI judges are consistently reliable but not valid, meaning they give the same answer repeatedly without that answer being correct. Published work and popular benchmarks like Chatbot Arena have treated consistency as proof of trustworthiness, and the audit shows that assumption is unfounded. Key facts What: Using one AI to grade another is now common — but the biggest audit yet shows these graders are consistent without being correct. A judge that always picks "answer A" scores perfectly on consistency. When: 2026-06-19 Primary source: read the source (arXiv 2606.19544) The distinction matters: a judge is reliable if it's consistent (same question, same answer), and valid if those answers are actually correct. The audit's central finding is that AI judges are reliable without being valid, and the field has been treating the first as evidence of the second. Because consistency is easy to measure and looks reassuring, it has stood in for actual trustworthiness across a lot of published work. A new audit makes the problem stark: a judge that ignores both answers and always picks the one labeled "A" would be perfectly consistent — flawless reliability, identical verdict every time — and completely worthless, because it never read anything. Consistency is trivially easy to fake and says almost nothing about whether the judging is sound. Yet "the judge agrees with itself" has done significant reassurance work in papers and benchmarks, and the always-pick-A example shows exactly how empty that reassurance is. When the researchers corrected for the agreement you'd get by chance — as any fair test should — confident-looking scores deflated noticeably. Gaps between models that seemed meaningful shrank or blurred. Accepted folk wisdom also took a hit: the long-standing worry that AI judges are suckers for longer, wordier answers turned out to be far weaker than assumed