Timezone: »
Dropout is a relatively new algorithm for training neural networks which relies on stochastically "dropping out'' neurons during training in order to avoid the co-adaptation of feature detectors. We introduce a general formalism for studying dropout on either units or connections, with arbitrary probability values, and use it to analyze the averaging and regularizing properties of dropout in both linear and non-linear networks. For deep neural networks, the averaging properties of dropout are characterized by three recursive equations, including the approximation of expectations by normalized weighted geometric means. We provide estimates and bounds for these approximations and corroborate the results with simulations. We also show in simple cases how dropout performs stochastic gradient descent on a regularized error function.
Author Information
Pierre Baldi (UC Irvine)
Peter Sadowski (University of Hawai‘i)
Related Events (a corresponding poster, oral, or spotlight)
-
2013 Oral: Understanding Dropout »
Sat. Dec 7th 12:20 -- 12:40 AM Room Harvey's Convention Center Floor, CC
More from the Same Authors
-
2020 : Hip Fracture Risk Modeling Using DXA and Deep Learning »
Peter Sadowski -
2021 : Deep learning reconstruction of the neutrino energy with a shallow Askaryan detector »
Stephen McAleer · Christian Glaser · Pierre Baldi -
2021 : G-SpaNet: Generalized Permutationless Set Assignment for Particle Physics using Symmetry Preserving Attention »
Alexander Shmakov · Shih-chieh Hsu · Pierre Baldi -
2022 : Geometry-aware Autoregressive Models for Calorimeter Shower Simulations »
Junze Liu · Aishik Ghosh · Dylan Smith · Pierre Baldi · Daniel Whiteson -
2022 : Self-supervised detection of atmospheric phenomena from remotely sensed synthetic aperture radar imagery »
Yannik Glaser · Peter Sadowski · Justin Stopa -
2022 : Foundations of Attention Mechanisms in Deep Neural Network Architectures »
Pierre Baldi · Roman Vershynin -
2022 : Feasible Adversarial Robust Reinforcement Learning for Underspecified Environments »
JB Lanier · Stephen McAleer · Pierre Baldi · Roy Fox -
2023 Poster: End-To-End Latent Variational Diffusion Models for Inverse Problems in High Energy Physics »
Alexander Shmakov · Kevin Greif · Michael Fenton · Aishik Ghosh · Pierre Baldi · Daniel Whiteson -
2023 Poster: Language Models can Solve Computer Tasks »
Geunwoo Kim · Pierre Baldi · Stephen McAleer -
2023 Poster: AI for Interpretable Chemistry: Predicting Radical Mechanistic Pathways via Contrastive Learning »
Mohammadamin Tavakoli · Pierre Baldi · Ann Marie Carlton · Yin Ting Chiu · Alexander Shmakov · David Van Vranken -
2022 : Foundations of Attention Mechanisms in Deep Neural Network Architectures »
Pierre Baldi · Roman Vershynin -
2021 Poster: XDO: A Double Oracle Algorithm for Extensive-Form Games »
Stephen McAleer · JB Lanier · Kevin A Wang · Pierre Baldi · Roy Fox -
2020 : Nowcasting Solar Irradiance Over Oahu »
Peter Sadowski -
2020 Poster: Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games »
Stephen McAleer · JB Lanier · Roy Fox · Pierre Baldi -
2019 Poster: Modeling Dynamic Functional Connectivity with Latent Factor Gaussian Processes »
Lingge Li · Dustin Pluta · Babak Shahbaba · Norbert Fortin · Hernando Ombao · Pierre Baldi -
2018 : Sherpa: Hyperparameter Optimization for Machine Learning Models »
Peter Sadowski -
2018 Poster: On Neuronal Capacity »
Pierre Baldi · Roman Vershynin -
2018 Oral: On Neuronal Capacity »
Pierre Baldi · Roman Vershynin -
2017 : Poster session »
Abbas Zaidi · Christoph Kurz · David Heckerman · YiJyun Lin · Stefan Riezler · Ilya Shpitser · Songbai Yan · Olivier Goudet · Yash Deshpande · Judea Pearl · Jovana Mitrovic · Brian Vegetabile · Tae Hwy Lee · Karen Sachs · Karthika Mohan · Reagan Rose · Julius Ramakers · Negar Hassanpour · Pierre Baldi · Razieh Nabi · Noah Hammarlund · Eli Sherman · Carolin Lawrence · Fattaneh Jabbari · Vira Semenova · Maria Dimakopoulou · Pratik Gajane · Russell Greiner · Ilias Zadik · Alexander Blocker · Hao Xu · Tal EL HAY · Tony Jebara · Benoit Rostykus -
2014 Workshop: High-energy particle physics, machine learning, and the HiggsML data challenge (HEPML) »
Glen Cowan · Balázs Kégl · Kyle Cranmer · Gábor Melis · Tim Salimans · Vladimir Vava Gligorov · Daniel Whiteson · Lester Mackey · Wojciech Kotlowski · Roberto Díaz Morales · Pierre Baldi · Cecile Germain · David Rousseau · Isabelle Guyon · Tianqi Chen -
2014 Poster: Searching for Higgs Boson Decay Modes with Deep Learning »
Peter Sadowski · Daniel Whiteson · Pierre Baldi -
2014 Spotlight: Searching for Higgs Boson Decay Modes with Deep Learning »
Peter Sadowski · Daniel Whiteson · Pierre Baldi -
2012 Poster: Deep Spatio-Temporal Architectures and Learning for Protein Structure Prediction »
Pietro Di Lena · Pierre Baldi · Ken Nagata -
2012 Spotlight: Deep Spatio-Temporal Architectures and Learning for Protein Structure Prediction »
Pietro Di Lena · Pierre Baldi · Ken Nagata -
2011 Poster: A Machine Learning Approach to Predict Chemical Reactions »
Matthew A Kayala · Pierre Baldi -
2010 Workshop: Charting Chemical Space: Challenges and Opportunities for AI and Machine Learning »
Pierre Baldi · Klaus-Robert Müller · Gisbert Schneider -
2007 Poster: Mining Internet-Scale Software Repositories »
Erik Linstead · Paul Rigor · sushil bajracharya · cristina lopes · Pierre Baldi -
2006 Poster: A Scalable Machine Learning Approach to Go »
Lin Wu · Pierre Baldi -
2006 Talk: A Scalable Machine Learning Approach to Go »
Lin Wu · Pierre Baldi