Timezone: »

 
Differentially Private Gradient Boosting on Linear Learners for Tabular Data
Saeyoung Rho · Shuai Tang · Sergul Aydore · Michael Kearns · Aaron Roth · Yu-Xiang Wang · Steven Wu · Cedric Archambeau

@
in Trustworthy and Socially Responsible Machine Learning »
Event URL: https://openreview.net/forum?id=uPF2bs14E3p »

Gradient boosting takes \emph{linear} combinations of weak base learners. Therefore, absent privacy constraints (when we can exactly optimize over the base models) it is not effective when run over base learner classes that are closed under linear combinations (e.g. linear models). As a result, gradient boosting is typically implemented with tree base learners (e.g., XGBoost), and this has become the state of the art approach in tabular data analysis. Prior work on private gradient boosting focused on taking the state of the art algorithm in the non-private regime---boosting on trees---and making it differentially private. Surprisingly, we find that when we use differentially private learners, gradient boosting over trees is not as effective as gradient boosting over linear learners. In this paper, we propose differentially private gradient-boosted linear models as a private classification method for tabular data. We empirically demonstrate that, under strict privacy constraints, it yields higher F1 scores than the private versions of gradient-boosted trees on five real-world binary classification problems. This work adds to the growing picture that the most effective learning methods under differential privacy may be quite different from the most effective learning methods without privacy.

[ Poster]  [ OpenReview]  [ Topia] 

Author Information

Saeyoung Rho (Columbia University)
Shuai Tang (Amazon Web Services)
Sergul Aydore (AWS AI)
Michael Kearns (University of Pennsylvania)

Michael Kearns is Professor and National Center Chair in the Computer and Information Science department at the University of Pennsylvania. His research interests include topics in machine learning, algorithmic game theory, social networks, and computational finance. Prior to joining the Penn faculty, he spent a decade at AT&T/Bell Labs, where he was head of AI Research. He is co-director of Penn’s Warren Center for Network and Data Sciences (warrencenter.upenn.edu), and founder of Penn’s Networked and Social Systems Engineering (NETS) undergraduate program (www.nets.upenn.edu). Kearns consults extensively in technology and finance, and is a Fellow of the Association for the Advancement of Artificial Intelligence and the American Academy of Arts and Sciences.

Aaron Roth (University of Pennsylvania)
Yu-Xiang Wang (UC Santa Barbara)
Steven Wu (Carnegie Mellon University)
Cedric Archambeau (Amazon Web Services)

More from the Same Authors