An Exact Conditional Null for Correlated Failure in Open-Model Ecosystems, and a Route to Anytime-Valid Monitoring

Measuring correlated failure between language models requires a null hypothesis for "independent enough," and prior work shows the conclusion is highly sensitive to which null is chosen, leaving the choice apparently subjective. We show one null is not a choice: the row and column margins of a model-by-item outcome matrix are the jointly sufficient statistics of the Rasch family, so the uniform distribution over margin-matched matrices is the unique fit-free conditional null for that family. At this null, on 1,228–1,362 open models across five benchmarks, the commonly reported mean-level "excess" co-failure is a deterministic function of item difficulty and vanishes identically, while a second-order residual correlation of 2.9–11.9× its null survives, is robust to removing 21–64% of models as near-duplicates, and concentrates in 4–12 latent dimensions rather than one shared factor. Our test is a fixed-sample exact randomization test, and because open leaderboards grow continuously the natural downstream use is a standing monitor. We build the fixed-set-of-looks version on the real arrival stream: twelve monthly looks over a population growing from 241 to 3,762 models, a Monte Carlo $p$-value against each look's own exact null, the standard calibrator, and the arithmetic mean of the calibrated e-values, which is valid under the severe dependence that nested populations induce. Against a control stream in which the null holds by construction the merged e-value stays at 0.90 (maximum 1.17 over all looks), as a valid e-value must; on real data every look saturates the Monte Carlo resolution ceiling, which we report as a ceiling rather than as unbounded evidence. What the exact null does not yet supply is an e-process permitting optional stopping, and we state precisely why: conditioning on the accumulated margins does not deliver a null for the increment when a row is added.