Architecture-Induced Rank Reversal in Out-of-Distribution Detection

Out-of-distribution (OOD) detection aims to identify inputs that differ from the data used to train a model. OOD detectors are typically compared on fixed benchmarks, but their performance also depends on the architecture used to represent the inputs. While recent work shows that OOD performance varies across architectures, whether such architectural changes alter the relative ranking of detectors remains underexplored. We study this question by evaluating six post-hoc OOD detectors across four architecture families and seven ImageNet-scale OOD benchmarks. We find that the architecture can change detector rankings even when the datasets, metrics, and evaluation protocol remain fixed, a phenomenon we call $\textit{Architecture-Induced Rank Reversal}$. Activation-based methods can lose their advantage on LayerNorm-based architectures, while distance- and density-based methods such as relative Mahalanobis distance and $k$-nearest neighbors become more competitive, and the gradient-based perturbation signal also weakens as the number of classes increases. In our experiments, one detector moves from first to last across architectures, while another moves from fifth to first, and Kendall's rank correlation reaches $-0.60$ between two supervised architectures. Within each architecture, by contrast, the ranking is recovered exactly under four evaluation seeds and three training configurations ($\tau=1.00$ in all twenty-seven comparisons), so these reversals track architecture rather than the choice of checkpoint. The same collapse also reproduces after fine-tuning on an entirely different dataset and label space, ruling out an explanation specific to ImageNet pretraining. The perturbation-robustness drop ratio also falls from $3.40\times$ at 10 classes to $0.89\times$ at 1000 classes. These results indicate that OOD detector rankings are architecture-dependent and may not generalize from a single-architecture evaluation.