Below the Reliability Floor: Recovering True Success from Judge-Gated Loops

LLM judges are increasingly placed inside an agent's loop, scoring the agent's own attempts and re-prompting until one passes. We show this quietly corrupts measurement: retry-until-PASS is optional stopping against a noisy classifier—it keeps drawing until the judge slips—so the reported pass rate is a biased estimator of true success, upward in the pass-prone retry regimes of interest (and downward under conservative rules such as strict rubrics or unanimous juries). We make this exact. The cap-K gate is a binary classifier with closed-form sensitivity/specificity, and its bias is governed by one coefficient, the gate's Youden index J: as J → 0 the gated rate becomes uninformative about π, so recovery must fall back to gold labels and no estimator beats the gold-only mean. Across 44 capable-agent loops on GSM8K, MATH, and code with objective ground truth (no authored weakness; a separate terse-agent stress set is excluded here) the inflation is systematic (median slip +0.16; worst on code, where the judge cannot run the candidate, a true 0.74 inflated to a reported 0.98) and obeys a closed-form law predicting the slip from per-attempt statistics (pooled r = 0.95; errors-in-variables slope 0.765 [0.70, 0.84], excluding 1). To recover true success we benchmark Rogan–Gladen against prediction-powered inference: the label-efficient estimators (PPI/PPI++) recover π far more accurately than naive reporting and Rogan–Gladen (mean recovery MAE 0.050 vs. 0.149, and 0.081 vs. 0.241 as gold becomes scarce), because they escape the 1/J² variance that makes the classical correction fragile; on balanced low-bias gates they match the gold-only mean (per-gate differences within bootstrap noise, none uniformly best), so our contribution is not a uniquely-best estimator but the identification of loop-induced bias and the J diagnostic that says when—and whether—to correct at all. Beyond verifiable gold, we measure recovery on public, human-labeled non-verifiable gates—response safety and summary quality—where PPI++ recovers the true rate to mean MAE 0.043 versus naive 0.165 (~4× in aggregate, up to >10× on the most-biased gates); and—in the motivating regime, a non-verifiable safety judge inside a real retry loop (n = 400, against a pre-registered 3-model strong-LLM panel—a disclosed proxy, human-anchored at raw 0.90 agreement)—a lenient gate ships 6.8% panel-labeled-unsafe responses (95% CI [0.047, 0.096]) while the calibrated correction recovers the panel safe-rate ~3.5× more accurately than naive. The deliverable is a recipe: report a PPI++ estimate alongside J as a reliability/identifiability diagnostic, measure at K = 1, and—via a label-free drift detector (ROC-AUC 0.80)—de-bias only when calibration transfers. We release all code and content-free data.