`

Timezone: »

 
Poster
Active Comparison of Prediction Models
Christoph Sawade · Niels Landwehr · Tobias Scheffer

Tue Dec 04 07:00 PM -- 12:00 AM (PST) @ Harrah’s Special Events Center 2nd Floor
in Poster Session »

We address the problem of comparing the risks of two given predictive models - for instance, a baseline model and a challenger - as confidently as possible on a fixed labeling budget. This problem occurs whenever models cannot be compared on held-out training data, possibly because the training data are unavailable or do not reflect the desired test distribution. In this case, new test instances have to be drawn and labeled at a cost. We devise an active comparison method that selects instances according to an instrumental sampling distribution. We derive the sampling distribution that maximizes the power of a statistical test applied to the observed empirical risks, and thereby minimizes the likelihood of choosing the inferior model. Empirically, we investigate model selection problems on several classification and regression tasks and study the accuracy of the resulting p-values.

[ PDF] 

Author Information

Christoph Sawade (University of Potsdam)
Niels Landwehr (University of Potsdam)
Tobias Scheffer (Universität Potsdam)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors