The Calibration Channel Determines the Bayes-Error Proxy: An Exact Law for Temperature-Induced Distortion

An evaluation conclusion is only as trustworthy as the instrument that produces 2 it. This paper examines one such instrument: the soft-label Bayes-error estimator 3 β(z) = E[min(z, 1 − z)] of Ishida et al. [1], which is used to judge whether a 4 classifier’s residual error is irreducible and to flag test errors that are implausibly 5 low. Because the estimator consumes probabilities rather than data, whatever 6 produced those probabilities becomes part of the measuring apparatus. Recent 7 work by Ushio et al. [2] showed that this apparatus is fragile: even perfectly 8 calibrated soft labels can yield a substantially inaccurate estimate, and they propose 9 isotonic calibration as a consistent remedy. We complement that line of work 10 by characterizing exactly how the most widely used post-hoc calibration map— 11 temperature scaling—distorts the metric. We prove an exact, model-free identity 12 reducing the temperature-scaled proxy to the classifier’s margin distribution, from 13 which we obtain (i) strict monotonicity in the temperature and (ii) a continuous bijection from the temperature axis onto the open interval (0, 1 2 14 ): a fixed classifier— 15 with fixed decisions and fixed 0–1 error—can be made to report any value of the 16 metric whatsoever, purely by a post-processing choice that changes nothing about 17 the system being evaluated. Under a Gaussian model of the logits we further derive 18 a two-parameter closed form for the entire metric-versus-temperature curve. Across 19 CIFAR-10, Fashion-MNIST, and SVHN (eight binary tasks), the proxy varies by 20 56× to 980× at constant test error, the closed form reproduces the empirical curve 21 to within 0.019, and the temperature that minimizes the expected calibration error 22 does not coincide with any stable proxy value—so calibrating well does not by 23 itself restore the conclusion. Our results give a precise, predictive account of the 24 distortion whose existence motivates calibration-based remedies, and they support 25 a concrete reporting norm: a proxy value is meaningful only together with the 26 mechanism that produced its probabilities.