Which drifted, the system or the judge? We measured our jury’s strictness

An evaluation of 21 AI judges shows rankings shifting by up to 14 places depending on the benchmark. Our average score dropped six points between campaigns: we had to prove whether our engine or our jury had changed.

An evaluation of 21 AI judges shows agent rankings shifting by up to 14 places depending on the benchmark used (arxiv.org/html/2606.19544). We experienced it: an average fell from 93.3 to 87.0 between campaigns, and nothing showed whether our engine or our jury had changed.

The symptom

We had already explained in quality measured over six days of duels that a six-point fall made us suspect a regression. This article goes further: how to prove methodically that the jury had become stricter rather than our engine having gone backwards.

The answer: paired duels

An absolute average proves nothing on its own, because a jury is never perfectly stable between sessions: more material to examine, more code that compiles, runs and can be tested, pushes a serious judge to be more demanding, not less. The answer is a paired duel: the same judge, two anonymised submissions for the same task side by side, in random order. Under this protocol, the latest version won six duels out of eight against its predecessor despite a lower absolute average.

What the duels revealed

The duels' execution probes showed why earlier averages misled us: several good scores rewarded untested houses of cards. A project rated 92.5 out of 100 did not compile and wiped its database at every launch. Another, rated 91.6, silently overwrote files. Recent submissions deliver more genuinely executable code, and a jury that runs, compiles and probes this code logically becomes stricter when there is more material to judge. Strictness drifted with what it was shown, not in the abstract.

Calibrated scoring

To make the two campaigns comparable anyway, a margin scale was fixed before the verdicts: tie, slight margin, clear margin and decisive margin, each worth a fixed number of points anchored to the previous version's sealed score. On this common scale, the latest version was worth around 96 out of 100, against 93.3 for its predecessor at its original strictness: a gain of around 3.5 points, in the opposite direction from what the absolute average suggested.

The rule we finally adopted

Absolute averages are now compared only within a single campaign. Between two versions, only paired duels count, and the jury now receives two versions of the target, the initial brief and the brief as it evolved before development, to stop confusing a deliberate pivot with a propagation defect. This is the same standard we apply when publishing our defeats: a measurement is only worthwhile if its measurement method is itself tested.

Why this extends beyond our case

Every team using a language model to judge its own work faces the same trap: a falling average is not automatically a regression, and a rising average is not automatically progress. Without paired duels or cross-rejudging, there is no way to decide.