Unexplained Variance Is Not Evidence of Discovery: Validating Interpretability-Derived Claims in ECG Age Models

Interpretability is increasingly used to argue that a model has discovered new knowledge: a network predicts a clinical variable, its residual error is "unexplained" by standard measurements, attribution localises that residual to an interpretable structure, and a discovery is claimed. We show this inference is unsafe, and give a protocol that makes it testable. Training a 12-lead ECG age regressor from scratch on PTB-XL, we find an apparently significant association between its age-gap residual and myocardial infarction ($+0.015$ AUC, significant after correction) that fails to replicate across data splits and vanishes when the set of already-measurable quantities is expanded from 5 to 15, with no change to the model. That expansion quadrupled the variance attributable to existing knowledge, from 3.5% to 14.6%: the apparent magnitude of a discovery is partly a property of how thoroughly the analyst enumerated prior knowledge, not of the data. Applying the same discipline to attribution, we identify one claim that survives: the model causally depends on atrial information classical P-wave measurement does not capture. It replicates across three independently trained models in each of two cohorts on different continents and survives model transfer between them. Turning the protocol on this claim narrows it without closing, leaving three quarters of the age gap unexplained. We report it as a falsifiable pointer, not a validated marker.