Timezone: »
We investigate the relationship between three fundamental problems in machine learning: binary classification, bipartite ranking, and binary class probability estimation (CPE). It is known that a good binary CPE model can be used to obtain a good binary classification model (by thresholding at 0.5), and also to obtain a good bipartite ranking model (by using the CPE model directly as a ranking model); it is also known that a binary classification model does not necessarily yield a CPE model. However, not much is known about other directions. Formally, these relationships involve regret transfer bounds. In this paper, we introduce the notion of weak regret transfer bounds, where the mapping needed to transform a model from one problem to another depends on the underlying probability distribution (and in practice, must be estimated from data). We then show that, in this weaker sense, a good bipartite ranking model can be used to construct a good classification model (by thresholding at a suitable point), and more surprisingly, also to construct a good binary CPE model (by calibrating the scores of the ranking model).
Author Information
Harikrishna Narasimhan (Google AI)
Shivani Agarwal (University of Pennsylvania)
Related Events (a corresponding poster, oral, or spotlight)
-
2013 Poster: On the Relationship Between Binary Classification, Bipartite Ranking, and Binary Class Probability Estimation »
Sun. Dec 8th 03:00 -- 07:59 AM Room Harrah's Special Events Center, 2nd Floor
More from the Same Authors
-
2016 Poster: Dueling Bandits: Beyond Condorcet Winners to General Tournament Solutions »
Siddartha Ramamohan · Arun Rajkumar · Shivani Agarwal · Shivani Agarwal -
2014 Workshop: Analysis of Rank Data: Confluence of Social Choice, Operations Research, and Machine Learning »
Shivani Agarwal · Hossein Azari Soufiani · Guy Bresler · Sewoong Oh · David Parkes · Arun Rajkumar · Devavrat Shah -
2014 Poster: On the Statistical Consistency of Plug-in Classifiers for Non-decomposable Performance Measures »
Harikrishna Narasimhan · Rohit Vaish · Shivani Agarwal -
2014 Poster: Online and Stochastic Gradient Methods for Non-decomposable Loss Functions »
Purushottam Kar · Harikrishna Narasimhan · Prateek Jain -
2014 Poster: Online Decision-Making in General Combinatorial Spaces »
Arun Rajkumar · Shivani Agarwal -
2013 Poster: Convex Calibrated Surrogates for Low-Rank Loss Matrices with Applications to Subset Ranking Losses »
Harish G Ramaswamy · Shivani Agarwal · Ambuj Tewari -
2013 Spotlight: Convex Calibrated Surrogates for Low-Rank Loss Matrices with Applications to Subset Ranking Losses »
Harish G Ramaswamy · Shivani Agarwal · Ambuj Tewari -
2012 Poster: Classification Calibration Dimension for General Multiclass Losses »
Harish G Ramaswamy · Shivani Agarwal -
2012 Spotlight: Classification Calibration Dimension for General Multiclass Losses »
Harish G Ramaswamy · Shivani Agarwal -
2009 Workshop: Advances in Ranking »
Shivani Agarwal · Chris J Burges · Yacov Crammer