Timezone: »
We address the problem of comparing the risks of two given predictive models - for instance, a baseline model and a challenger - as confidently as possible on a fixed labeling budget. This problem occurs whenever models cannot be compared on held-out training data, possibly because the training data are unavailable or do not reflect the desired test distribution. In this case, new test instances have to be drawn and labeled at a cost. We devise an active comparison method that selects instances according to an instrumental sampling distribution. We derive the sampling distribution that maximizes the power of a statistical test applied to the observed empirical risks, and thereby minimizes the likelihood of choosing the inferior model. Empirically, we investigate model selection problems on several classification and regression tasks and study the accuracy of the resulting p-values.
Author Information
Christoph Sawade (University of Potsdam)
Niels Landwehr (University of Potsdam)
Tobias Scheffer (Universität Potsdam)
Related Events (a corresponding poster, oral, or spotlight)
-
2012 Spotlight: Active Comparison of Prediction Models »
Tue. Dec 4th 07:44 -- 07:48 PM Room Harveys Convention Center Floor, CC
More from the Same Authors
-
2010 Spotlight: Throttling Poisson Processes »
Uwe Dick · Peter Haider · Tobias Scheffer -
2010 Poster: Throttling Poisson Processes »
Uwe Dick · Peter Haider · Thomas Vanck · Michael Brückner · Tobias Scheffer -
2010 Poster: Active Estimation of F-Measures »
Christoph Sawade · Niels Landwehr · Tobias Scheffer -
2009 Poster: Localizing Bugs in Program Executions with Graphical Models »
Laura Dietz · Valentin Dallmeier · Andreas Zeller · Tobias Scheffer -
2009 Poster: Nash Equilibria of Static Prediction Games »
Michael Brückner · Tobias Scheffer -
2008 Poster: Transfer Learning by Distribution Matching for Targeted Advertising »
Steffen Bickel · Christoph Sawade · Tobias Scheffer -
2006 Poster: Dirichlet-Enhanced Spam Filtering based on Biased Samples »
Steffen Bickel · Tobias Scheffer -
2006 Spotlight: Dirichlet-Enhanced Spam Filtering based on Biased Samples »
Steffen Bickel · Tobias Scheffer