Timezone: »
Deep learning has driven dramatic performance advances on numerous difficult machine learning tasks in a wide range of applications. Yet, its theoretical foundations remain poorly understood, with many more questions than answers. For example: What are the modeling assumptions underlying deep networks? How well can we expect deep networks to perform? When a certain network succeeds or fails, can we determine why and how? How can we adapt deep learning to new domains in a principled way?
While some progress has been made recently towards a foundational understanding of deep learning, most theory work has been disjointed, and a coherent picture has yet to emerge. Indeed, the current state of deep learning theory is like the fable “The Blind Men and the Elephant”.
The goal of this workshop is to provide a forum where theoretical researchers of all stripes can come together not only to share reports on their individual progress but also to find new ways to join forces towards the goal of a coherent theory of deep learning. Topics to be discussed include:
- Statistical guarantees for deep learning models
- Expressive power and capacity of neural networks
- New probabilistic models from which various deep architectures can be derived
- Optimization landscapes of deep networks
- Deep representations and invariance to latent factors
- Tensor analysis of deep learning
- Deep learning from an approximation theory perspective
- Sparse coding and deep learning
- Mixture models, the EM algorithm, and deep learning
In addition to invited and contributed talks by leading researchers from diverse backgrounds, the workshop will feature an extended poster/discussion session and panel discussion on which combinations of ideas are most likely to move theory of deep learning forward and which might lead to blind alleys.
Accepted Papers and Authors
1. A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks. Sanjeev Arora, Nadav Cohen, Noah Golowich and Wei Hu.
2. On the convergence of SGD on neural nets and other over-parameterized problems. Karthik Abinav Sankararaman, Soham De, Zheng Xu, W. Ronny Huang and Tom Goldstein.
3. Optimal SGD Hyperparameters for Fully Connected Networks. Daniel Park, Samuel Smith, Jascha Sohl-Dickstein and Quoc Le.
4. Invariant representation learning for robust deep networks. Julian Salazar, Davis Liang, Zhiheng Huang and Zachary Lipton.
5. Characterizing & Exploring Deep CNN Representations Using Factorization. Uday Singh Saini and Evangelos Papalexakis.
6. On the Weak Neural Dependence Phenomenon in Deep Learning. Jiayao Zhang, Ruoxi Jia, Bo Li and Dawn Song.
7. DNN or k-NN: That is the Generalize vs. Memorize Question. Gilad Cohen, Guillermo Sapiro and Raja Giryes.
8. On the Margin Theory of Feedforward Neural Networks. Colin Wei, Jason Lee, Qiang Liu and Tengyu Ma.
9. A Differential Topological View of Challenges in Learning with Deep Neural Networks. Hao Shen.
10. Theoretical Analysis of Auto Rate-tuning by Batch Normalization. Sanjeev Arora, Zhiyuan Li and Kaifeng Lyu.
11. Topological Constraints onHomeomorphic Auto-Encoding. Pim de Haan and Luca Falorsi.
12. Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience. Vaishnavh Nagarajan and J. Zico Kolter.
13. Directional Analysis of Stochastic Gradient Descent via von Mises-Fisher Distributions in Deep Learning. Cheolhyoung Lee, Kyunghyun Cho and Wanmo Kang.
14. Multi-dimensional Count Sketch: Dimension Reduction That Retains Efficient Tensor Operations. Yang Shi and Anima Anandkumar.
15. Gradient Descent Provably Optimizes Over-parameterized Neural Networks. Simon Du, Xiyu Zhai, Aarti Singh and Barnabas Poczos.
16. The Dynamic Distance Between Learning Tasks. Alessandro Achille, Glen Bigan Mbeng and Stefano Soatto.
17. Stochastic Gradient/Mirror Descent: Minimax Optimality and Implicit Regularization. Navid Azizan and Babak Hassibi.
18. Shared Representation Across Neural Networks. Qihong Lu, Po-Hsuan Chen, Jonathan Pillow, Peter Ramadge, Kenneth Norman and Uri Hasson.
19. Learning in gated neural networks. Ashok Makkuva, Sewoong Oh, Sreeram Kannan and Pramod Viswanath.
20. Gradient descent aligns the layers of deep linear networks. Ziwei Ji and Matus Telgarsky.
21. Fluctuation-dissipation relation for stochastic gradient descent. Sho Yaida.
22. Identifying Generalization Properties in Neural Networks. Huan Wang, Nitish Shirish Keskar, Caiming Xiong and Richard Socher.
23. A Theoretical Framework for Deep and Locally Connected ReLU Network. Yuandong Tian.
24. Minimum norm solutions do not always generalize well for over-parameterized problems. Vatsal Shah, Anastasios Kyrillidis and Sujay Sanghavi.
25. An Empirical Exploration of Gradient Correlations in Deep Learning. Daniel Rothchild, Roy Fox, Noah Golmant, Joseph Gonzalez, Michael Mahoney, Kai Rothauge, Ion Stoica and Zhewei Yao.
26. Geometric Scattering on Manifolds. Michael Perlmutter, Guy Wolf and Matthew Hirn.
27. Theoretical Insights into Memorization in GANs. Vaishnavh Nagarajan, Colin Raffel and Ian Goodfellow.
28. A jamming transition from under- to over-parametrization affects loss landscape and generalization. Stefano Spigler, Mario Geiger, Stéphane d'Ascoli, Levent Sagun, Giulio Biroli and Matthieu Wyart.
29. A Mean Field Theory of Multi-Layer RNNs. David Anderson, Jeffrey Pennington and Satyen Kale.
30. Generalization and regularization in deep learning for nonlinear inverse problems. Christopher Wong, Maarten de Hoop and Matti Lassas.
31. On the Spectral Bias of Neural Networks. Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio and Aaron Courville.
32. On Generalization Bounds for a Family of Recurrent Neural Networks. Minshuo Chen, Xingguo Li and Tuo Zhao.
33. SGD Implicitly Regularizes Generalization Error. Dan Roberts.
34. Iteratively Learning from the Best. Yanyao Shen and Sujay Sanghavi.
35. Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks. Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun and Nathan Srebro.
36. An Escape-Time Analysis of SGD. Philippe Casgrain, Mufan Li, Gintare Karolina Dziugaite and Daniel Roy.
37. Information Regularized Neural Networks. Tianchen Zhao, Dejiao Zhang, Zeyu Sun and Honglak Lee.
38. Generalization Bounds for Unsupervised Cross-Domain Mapping with WGANs. Tomer Galanti, Sagie Benaim and Lior Wolf.
39. Degeneracy, Trainability, and Generalization in Deep Neural Networks. Emin Orhan and Xaq Pitkow.
40. A Max-Affine Spline View of Deep Network Nonlinearities. Randall Balestriero and Richard Baraniuk.
Schedule
Sat 5:30 a.m. - 5:40 a.m.
|
Opening Remarks
(
Remarks
)
|
🔗 |
Sat 5:40 a.m. - 6:00 a.m.
|
Contributed Talk 1
(
Contributed Talk
)
|
Jason Lee 🔗 |
Sat 6:00 a.m. - 6:20 a.m.
|
Contributed Talk 2
(
Contributed Talk
)
|
Michael Perlmutter 🔗 |
Sat 6:20 a.m. - 7:00 a.m.
|
Plenary Talk 1
(
Plenary Talk
)
|
Sanjeev Arora 🔗 |
Sat 7:00 a.m. - 7:30 a.m.
|
Invited Talk 1
(
Invited Talk
)
|
Zachary Lipton 🔗 |
Sat 7:30 a.m. - 7:50 a.m.
|
Coffee Break
|
🔗 |
Sat 7:50 a.m. - 8:30 a.m.
|
Plenary Talk 2
(
Plenary Talk
)
|
Kamalika Chaudhuri 🔗 |
Sat 8:30 a.m. - 9:00 a.m.
|
Invited Talk 2
(
Invited Talk
)
|
Judy Hoffman 🔗 |
Sat 9:00 a.m. - 10:30 a.m.
|
Lunch Break
|
🔗 |
Sat 10:30 a.m. - 11:10 a.m.
|
Plenary Talk 3
(
Plenary Talk
)
|
Stefano Soatto 🔗 |
Sat 11:10 a.m. - 11:40 a.m.
|
Invited Talk 3
(
Invited Talk
)
|
Irina Higgins 🔗 |
Sat 11:40 a.m. - 12:00 p.m.
|
Contributed Talk 3
(
Contributed Talk
)
|
Tan Nguyen 🔗 |
Sat 12:00 p.m. - 12:50 p.m.
|
Poster Session
|
Sujay Sanghavi · Vatsal Shah · Yanyao Shen · Tianchen Zhao · Yuandong Tian · Tomer Galanti · Mufan Li · Gilad Cohen · Daniel Rothchild · Aristide Baratin · Devansh Arpit · Vagelis Papalexakis · Michael Perlmutter · Ashok Vardhan Makkuva · Pim de Haan · Yingyan Lin · Wanmo Kang · Cheolhyoung Lee · Hao Shen · Sho Yaida · Dan Roberts · Nadav Cohen · Philippe Casgrain · Dejiao Zhang · Tengyu Ma · Avinash Ravichandran · Julian Emilio Salazar · Bo Li · Davis Liang · Christopher Wong · Glen Bigan Mbeng · Animesh Garg
|
Sat 12:50 p.m. - 1:30 p.m.
|
Plenary Talk 4
(
Plenary Talk
)
|
Emily Fox 🔗 |
Sat 1:30 p.m. - 2:00 p.m.
|
Invited Talk 4
(
Invited Talk
)
|
Jeremias Sulam 🔗 |
Sat 2:00 p.m. - 2:55 p.m.
|
Panel Discussion
|
🔗 |
Sat 2:55 p.m. - 3:00 p.m.
|
Closing Remarks
(
Remarks
)
|
🔗 |
Author Information
Richard Baraniuk (Rice University)
Anima Anandkumar (Caltech/Amazon)
Stephane Mallat (Ecole normale superieure)
Ankit Patel (Rice University)
nhật Hồ (University of California, Berkeley)
More from the Same Authors
-
2022 : Investigating Reproducibility from the Decision Boundary Perspective. »
Gowthami Somepalli · Arpit Bansal · Liam Fowl · Ping-yeh Chiang · Yehuda Dar · Richard Baraniuk · Micah Goldblum · Tom Goldstein -
2022 : Retrieval-based Controllable Molecule Generation »
Jack Wang · Weili Nie · Zhuoran Qiao · Chaowei Xiao · Richard Baraniuk · Anima Anandkumar -
2022 : Exact Visualization of Deep Neural Network Geometry and Decision Boundary »
Ahmed Imtiaz Humayun · Randall Balestriero · Richard Baraniuk -
2022 : StyleGAN2-based Out-of-Distribution Detection for Medical Imaging »
McKell Woodland · John Wood · Caleb O'Connor · Ankit Patel · Kristy Brock -
2022 : Using Deep Learning and Macroscopic Imaging of Porcine Heart Valve Leaflets to Predict Uniaxial Stress-Strain Responses »
Luis Victor · CJ Barberan · Richard Baraniuk · Jane Grande-Allen -
2023 Poster: Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals »
Tam Nguyen · Tan Nguyen · Richard Baraniuk -
2023 Workshop: Learning-Based Solutions for Inverse Problems »
Shirin Jalali · christopher metzler · Ajil Jalal · Jon Tamir · Reinhard Heckel · Paul Hand · Arian Maleki · Richard Baraniuk -
2022 Poster: Wavelet Score-Based Generative Modeling »
Florentin Guth · Simon Coste · Valentin De Bortoli · Stephane Mallat -
2022 Poster: Parameters or Privacy: A Provable Tradeoff Between Overparameterization and Membership Inference »
Jasper Tan · Blake Mason · Hamid Javadi · Richard Baraniuk -
2021 Poster: The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization »
Daniel LeJeune · Hamid Javadi · Richard Baraniuk -
2020 : Opening Remarks »
Reinhard Heckel · Paul Hand · Soheil Feizi · Lenka Zdeborová · Richard Baraniuk -
2020 Workshop: Workshop on Deep Learning and Inverse Problems »
Reinhard Heckel · Paul Hand · Richard Baraniuk · Lenka Zdeborová · Soheil Feizi -
2020 Poster: Analytical Probability Distributions and Exact Expectation-Maximization for Deep Generative Networks »
Randall Balestriero · Sebastien PARIS · Richard Baraniuk -
2020 Poster: MomentumRNN: Integrating Momentum into Recurrent Neural Networks »
Tan Nguyen · Richard Baraniuk · Andrea Bertozzi · Stanley Osher · Bao Wang -
2020 Poster: Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning »
Weili Nie · Zhiding Yu · Lei Mao · Ankit Patel · Yuke Zhu · Anima Anandkumar -
2020 Spotlight: Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning »
Weili Nie · Zhiding Yu · Lei Mao · Ankit Patel · Yuke Zhu · Anima Anandkumar -
2019 : Poster Session »
Pravish Sainath · Mohamed Akrout · Charles Delahunt · Nathan Kutz · Guangyu Robert Yang · Joseph Marino · L F Abbott · Nicolas Vecoven · Damien Ernst · andrew warrington · Michael Kagan · Kyunghyun Cho · Kameron Harris · Leopold Grinberg · John J. Hopfield · Dmitry Krotov · Taliah Muhammad · Erick Cobos · Edgar Walker · Jacob Reimer · Andreas Tolias · Alexander Ecker · Janaki Sheth · Yu Zhang · Maciej Wołczyk · Jacek Tabor · Szymon Maszke · Roman Pogodin · Dane Corneil · Wulfram Gerstner · Baihan Lin · Guillermo Cecchi · Jenna M Reinen · Irina Rish · Guillaume Bellec · Darjan Salaj · Anand Subramoney · Wolfgang Maass · Yueqi Wang · Ari Pakman · Jin Hyung Lee · Liam Paninski · Bryan Tripp · Colin Graber · Alex Schwing · Luke Prince · Gabriel Ocker · Michael Buice · Benjamin Lansdell · Konrad Kording · Jack Lindsey · Terrence Sejnowski · Matthew Farrell · Eric Shea-Brown · Nicolas Farrugia · Victor Nepveu · Jiwoong Im · Kristin Branson · Brian Hu · Ramakrishnan Iyer · Stefan Mihalas · Sneha Aenugu · Hananel Hazan · Sihui Dai · Tan Nguyen · Doris Tsao · Richard Baraniuk · Anima Anandkumar · Hidenori Tanaka · Aran Nayebi · Stephen Baccus · Surya Ganguli · Dean Pospisil · Eilif Muller · Jeffrey S Cheng · Gaël Varoquaux · Kamalaker Dadi · Dimitrios C Gklezakos · Rajesh PN Rao · Anand Louis · Christos Papadimitriou · Santosh Vempala · Naganand Yadati · Daniel Zdeblick · Daniela M Witten · Nicholas Roberts · Vinay Prabhu · Pierre Bellec · Poornima Ramesh · Jakob H Macke · Santiago Cadena · Guillaume Bellec · Franz Scherr · Owen Marschall · Robert Kim · Hannes Rapp · Marcio Fonseca · Oliver Armitage · Jiwoong Im · Thomas Hardcastle · Abhishek Sharma · Wyeth Bair · Adrian Valente · Shane Shang · Merav Stern · Rutuja Patil · Peter Wang · Sruthi Gorantla · Peter Stratton · Tristan Edwards · Jialin Lu · Martin Ester · Yurii Vlasov · Siavash Golkar -
2019 : Opening Remarks »
Reinhard Heckel · Paul Hand · Alex Dimakis · Joan Bruna · Deanna Needell · Richard Baraniuk -
2019 Workshop: Solving inverse problems with deep networks: New architectures, theoretical foundations, and applications »
Reinhard Heckel · Paul Hand · Richard Baraniuk · Joan Bruna · Alex Dimakis · Deanna Needell -
2019 Poster: The Geometry of Deep Networks: Power Diagram Subdivision »
Randall Balestriero · Romain Cosentino · Behnaam Aazhang · Richard Baraniuk -
2018 : Panel Discussion »
Richard Baraniuk · Maarten V. de Hoop · Paul A Johnson -
2018 : Introduction »
Laura Pyrak-Nolte · James Rustad · Richard Baraniuk -
2018 Workshop: Machine Learning for Geophysical & Geochemical Signals »
Laura Pyrak-Nolte · James Rustad · Richard Baraniuk -
2018 Poster: Theoretical guarantees for EM under misspecified Gaussian mixture models »
Raaz Dwivedi · nhật Hồ · Koulik Khamaru · Martin Wainwright · Michael Jordan -
2017 Workshop: Advances in Modeling and Learning Interactions from Complex Data »
Gautam Dasarathy · Mladen Kolar · Richard Baraniuk -
2017 Poster: Solid Harmonic Wavelet Scattering: Predicting Quantum Molecular Energy from Invariant Descriptors of 3D Electronic Densities »
Michael Eickenberg · Georgios Exarchakis · Matthew Hirn · Stephane Mallat -
2017 Poster: Learned D-AMP: Principled Neural Network based Compressive Image Recovery »
Chris Metzler · Ali Mousavi · Richard Baraniuk -
2016 Workshop: Machine Learning for Education »
Richard Baraniuk · Jiquan Ngiam · Christoph Studer · Phillip Grimaldi · Andrew Lan -
2016 Poster: A Probabilistic Framework for Deep Learning »
Ankit Patel · Tan Nguyen · Richard Baraniuk -
2015 : Low-dimensional inference with high-dimensional data »
Richard Baraniuk -
2015 : Probabilistic Theory of Deep Learning »
Richard Baraniuk -
2014 Workshop: Human Propelled Machine Learning »
Richard Baraniuk · Michael Mozer · Divyanshu Vats · Christoph Studer · Andrew E Waters · Andrew Lan -
2013 Poster: When in Doubt, SWAP: High-Dimensional Sparse Recovery from Correlated Measurements »
Divyanshu Vats · Richard Baraniuk -
2011 Poster: SpaRCS: Recovering low-rank and sparse matrices from compressive measurements »
Andrew E Waters · Aswin C Sankaranarayanan · Richard Baraniuk -
2009 Workshop: Manifolds, sparsity, and structured models: When can low-dimensional geometry really help? »
Richard Baraniuk · Volkan Cevher · Mark A Davenport · Piotr Indyk · Bruno Olshausen · Michael B Wakin -
2008 Poster: Sparse Signal Recovery Using Markov Random Fields »
Volkan Cevher · Marco F Duarte · Chinmay Hegde · Richard Baraniuk -
2008 Spotlight: Sparse Signal Recovery Using Markov Random Fields »
Volkan Cevher · Marco F Duarte · Chinmay Hegde · Richard Baraniuk -
2007 Poster: Random Projections for Manifold Learning »
Chinmay Hegde · Richard Baraniuk