Timezone: »
A common pain point in differentially private machine learning is the significant runtime overhead incurred when executing Differentially Private Stochastic Gradient Descent (DPSGD), which may be as large as two orders of magnitude. We thoroughly demonstrate that by exploiting powerful language primitives, including vectorization, just-in-time compilation, and static graph optimization, one can dramatically reduce these overheads, in many cases nearly matching the best non-private running times. These gains are realized in two frameworks: one is JAX, which provides rich support for these primitives through the XLA compiler. We also rebuild core parts of TensorFlow Privacy, integrating more effective vectorization as well as XLA compilation, granting significant memory and runtime improvements over previous release versions. Our proposed approaches allow us to achieve up to 50x speedups compared to the best alternatives. Our code is available at https://github.com/TheSalon/fast-dpsgd.
Author Information
Pranav Subramani (University Of Waterloo)
Nicholas Vadivelu (University of Waterloo)
I'm a student interested in **machine learning** and **programming languages**. Currently, I'm making machine learning inference faster at **NVIDIA** ([TensorRT](https://developer.nvidia.com/tensorrt)) and studying Computer Science & Statistics at the **University of Waterloo**. Previously, I did deep learning research on collaborative self-driving at **Uber ATG**, helped improve the neural network optimization algorithm [K-FAC](https://github.com/tensorflow/kfac) at **Google Brain**, and developed data-driven models to identify fraud at **John Hancock Financial**. I've also worked as a Research Assistant for **Prof. Lin Tan** in deep learning-driven software analysis and **Prof. Pascal Poupart** on neural network parameter learning. If you love tidbits of technical info, follow me on [twitter](https://twitter.com/nicvadivelu). Feel free to reach out through email: nicholas.vadivelu@gmail.com or book some time with me through [calendly](https://calendly.com/nicv)--I love to chat!
Gautam Kamath (University of Waterloo)
More from the Same Authors
-
2020 : Enabling Fast Differentially Private SGD via Static Graph Compilation and Batch-Level Parallelism »
Pranav Subramani -
2022 : Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance »
Xin Gu · Gautam Kamath · Steven Wu -
2022 : Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attacks »
Jimmy Di · Jack Douglas · Jayadev Acharya · Gautam Kamath · Ayush Sekhari -
2022 : Indiscriminate Data Poisoning Attacks on Neural Networks »
Yiwei Lu · Gautam Kamath · Yaoliang Yu -
2022 : Indiscriminate Data Poisoning Attacks on Neural Networks »
Yiwei Lu · Gautam Kamath · Yaoliang Yu -
2022 : Hidden Poison: Machine unlearning enables camouflaged poisoning attacks »
Jimmy Di · Jack Douglas · Jayadev Acharya · Gautam Kamath · Ayush Sekhari -
2022 : Private GANs, Revisited »
Alex Bie · Gautam Kamath · Guojun Zhang -
2022 Poster: New Lower Bounds for Private Estimation and a Generalized Fingerprinting Lemma »
Gautam Kamath · Argyris Mouzakis · Vikrant Singhal -
2022 Poster: Private Estimation with Public Data »
Alex Bie · Gautam Kamath · Vikrant Singhal -
2021 Poster: Remember What You Want to Forget: Algorithms for Machine Unlearning »
Ayush Sekhari · Jayadev Acharya · Gautam Kamath · Ananda Theertha Suresh -
2020 Poster: The Discrete Gaussian for Differential Privacy »
Clément L Canonne · Gautam Kamath · Thomas Steinke -
2020 Social: Data Privacy: Academia, Industry, Policy, and Society »
Gautam Kamath -
2020 Poster: CoinPress: Practical Private Mean and Covariance Estimation »
Sourav Biswas · Yihe Dong · Gautam Kamath · Jonathan Ullman -
2020 Poster: Private Identity Testing for High-Dimensional Distributions »
Clément L Canonne · Gautam Kamath · Audra McMillan · Jonathan Ullman · Lydia Zakynthinou -
2020 Spotlight: Private Identity Testing for High-Dimensional Distributions »
Clément L Canonne · Gautam Kamath · Audra McMillan · Jonathan Ullman · Lydia Zakynthinou -
2019 Poster: Private Hypothesis Selection »
Mark Bun · Gautam Kamath · Thomas Steinke · Steven Wu -
2019 Poster: Differentially Private Algorithms for Learning Mixtures of Separated Gaussians »
Gautam Kamath · Or Sheffet · Vikrant Singhal · Jonathan Ullman -
2017 Poster: Concentration of Multilinear Functions of the Ising Model with Applications to Network Data »
Constantinos Daskalakis · Nishanth Dikkala · Gautam Kamath -
2015 Poster: Optimal Testing for Properties of Distributions »
Jayadev Acharya · Constantinos Daskalakis · Gautam Kamath -
2015 Spotlight: Optimal Testing for Properties of Distributions »
Jayadev Acharya · Constantinos Daskalakis · Gautam Kamath