Better Prediction Does Not Mean Better Decisions: Evaluating Learned Measurement Surrogates by Decision Value

Predictive models are often evaluated based on their ability to accurately match or reconstruct a reference measurement, even when their results are used to guide real-world decisions. However, does higher predictive accuracy necessarily translate to better decisions? We examine when predictive fidelity is informative about decision-relevant outcomes. Our framework considers a latent state relevant to the decision, a primary target measurement, a surrogate derived from richer input data, and a downstream policy, all within a unified information-theoretic structure. We demonstrate, via a simple counterexample, that two surrogates may both achieve very low squared reconstruction error, yet the surrogate with the lower error can yield inferior decision performance. Blackwell’s classical theory provides a sufficient condition: if the surrogate signal is obtained solely by state-independent post-processing of the target measurement, its decision value cannot exceed that of the target for any policy class closed under such post-processing. However, a learned surrogate leveraging richer inputs may not satisfy this property. To illustrate, we analyze a public biological dataset comprising 16 matched events from 11 healthy participants, each with four regional measurements plus a downstream stool measurement. We observed that rankings of reconstruction accuracy varied depending on the metric (Aitchison or total-variation geometry). For three prespecified proxy decision tasks, only the distal-local task exhibited a clear discrepancy between the fitted surrogate and stool measurement, and omitting the state-defining region reduced this discrepancy. Our findings indicate that predictive loss alone does not guarantee task-specific decision-curve fidelity. When predictions inform actions, evaluating both predictive accuracy and decision-aware performance is essential, as they address distinct aspects of model utility.