Swap-Consistency Does Not Certify a Code Judge: Separating Calibrated Abstention from Position-Locking with an Execution Oracle
Ching-Yu Lin ⋅ Yifan Liu
Abstract
Pairwise LLM judges grade code at scale, and the field certifies them with a single scalar, swap-consistency --- whether a verdict survives swapping the candidates' order. That scalar conflates two fundamentally different behaviors into the same recorded “tie,” a failure we call \emph{tie conflation}. One is calibrated abstention, where the judge recognizes two programs as behaviorally equivalent and appropriately declines to choose a winner. The other is position-locking, where the judge favors the same slot regardless of content. We separate the two with an oracle the judge never sees: each program runs its problem's unit tests, and pairs with identical per-test results form an execution-indistinguishable arm, all others an unequal-baseline arm; the split comes from execution alone, and no LLM grades anything. On it we define and analyze the \emph{tie-calibration gap}, $\Delta_{\text{tie}}$: a judge's excess both-order abstention rate on execution-indistinguishable pairs over unequal-baseline pairs --- high when ties track genuine equivalence, near zero when ``ties'' are one slot winning every time. Across six judges and 20,432 judgments, $\Delta_{\text{tie}}$ spans +66.05 pp to $-$0.08 pp between judges with similar overall tie rates, a spread no consistency scalar resolves. Two checks separate calibration from timidity or inability: abstention falls from 77.8\% at gap 0 to a 9.6--9.7\% floor as the gap in tests passed between the two programs grows, and the five non-degenerate judges pick the program passing more tests on 81.3--96.2\% of the pairs they decide. Nor is forcing a verdict a fix: re-judged under a no-tie rubric, the same slot still wins both orders on 36.5--40.0\% of pairs. We distil a four-part audit protocol --- report all four swap outcomes, test for first-shown bias, report $\Delta_{\text{tie}}$ wherever outputs execute, floor inconsistency claims at the judge's own test--retest flip rate --- deployable wherever a mechanical execution oracle exists.
Chat is not available.
Successful Page Load