Right for the Wrong Reason: OOD Prescreening Helps Parkinson’s Detection Without Detecting the Shift

Deep learning models for Parkinson's disease detection can fail without warning on patients who differ from the training population. Out-of-distribution (OOD) prescreening has been proposed to catch such cases before they reach an unreliable model, but whether this is feasible under realistic deployment conditions remains largely untested. In this study, we evaluate six OOD detectors on gait recordings from three clinical studies, under a deliberately imposed demographic shift. None of the detectors separate held-out-site records from training records better than chance, even though predictions on flagged records are consistently less reliable than those retained. This apparent benefit stems from ranking sample difficulty rather than detecting the actual uniqueness in the data distribution. Indeed, at matched rejection rates, the top detector recovers only 14% of the performance headroom available by an oracle rejection rule, while a naive demographic heuristic performs worse than random rejection. These findings suggest that OOD prescreening can still be useful for deciding which predictions to withhold, but not because it detects the shift it is meant to catch, nor because obvious heuristics are safe substitutes.