Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction

Decision-making pipelines are generally characterized by tradeoffs among various risk functions. Itis often desirable to manage such tradeoffs in a data-adaptive manner. As we demonstrate, if this isdone naively, state-of-the art uncertainty quantification methods can lead to significant violations ofputative risk guarantees. To address this issue, we develop methods that permit valid control of riskwhen threshold and tradeoff parameters are chosen adaptively. Our methodology supports monotoneand nearly-monotone risks, but otherwise makes no distributional assumptions. To illustrate the benefitsof our approach, we carry out numerical experiments on synthetic data and the large-scale vision datasetMS-COCO.