Shape, Not Location: A Diagnosis of Calibration Failure in Open-Set Recognition

Open-set recognition requires a model to distinguish among classes with abundant training data and classes for which no training images are available. A common approach is calibrated stacking, which reconciles supervised and zero-shot predictions by applying an additive calibration constant. We argue that this formulation addresses the wrong failure mode: the central problem is not score location, but score shape. We investigate this hypothesis on a 17,393-class fine-grained fish recognition task containing 5,795 seen classes and 11,598 unseen classes. Although the isolated zero-shot branch achieves 0.1198 accuracy, naive additive calibration produces only 0.0113 accuracy on unseen classes and 0.4402 overall accuracy. Using the same underlying prediction heads, we instead score seen and unseen candidates homogeneously in a shared embedding space, increasing overall accuracy to 0.4737 and unseen accuracy to 0.1265. This result shows that changing the scoring geometry can recover substantially more zero-shot signal than a constant score shift. We further introduce a nine-signal routing classifier to determine when seen- or unseen-oriented evidence should be trusted. Under proper five-fold cross-validation, the router achieves an AUROC of 0.8749 ± 0.0035, compared with 0.8506 for a single margin-based signal. On the real held-out evaluation, routing improves accuracy from 0.4461 to 0.4506, corresponding to 163 additional correct predictions. Importantly, an alternative routing configuration that appeared promising offline underperforms the no-routing baseline after deployment, demonstrating the importance of deployment-level validation. Together, these results identify score-shape mismatch as a fundamental failure mode of additive calibration in open-set recognition and motivate homogeneous scoring and multi-signal routing as more effective alternatives.