Timezone: »

 
Poster
Exact Solutions of a Deep Linear Network
Liu Ziyin · Botao Li · Xiangming Meng

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #515
This work finds the analytical expression of the global minima of a deep linear network with weight decay and stochastic neurons, a fundamental model for understanding the landscape of neural networks. Our result implies that zero is a special point in deep neural network architecture. We show that weight decay strongly interacts with the model architecture and can create bad minima at zero in a network with more than $1$ hidden layer, qualitatively different from a network with only $1$ hidden layer. Practically, our result implies that common deep learning initialization methods are insufficient to ease the optimization of neural networks in general.

Author Information

Liu Ziyin (University of Tokyo)
Botao Li (École Normale Supérieure)
Xiangming Meng (The University of Tokyo)

More from the Same Authors

  • 2022 : What shapes the loss landscape of self-supervised learning? »
    Liu Ziyin · Ekdeep S Lubana · Masahito Ueda · Hidenori Tanaka
  • 2022 Panel: Panel 1A-2: Posterior Collapse of… & Understanding and Extending… »
    Fabrizio Frasca · Liu Ziyin
  • 2022 Poster: Posterior Collapse of a Linear Latent Variable Model »
    Zihao Wang · Liu Ziyin
  • 2020 Poster: Neural Networks Fail to Learn Periodic Functions and How to Fix It »
    Liu Ziyin · Tilman Hartwig · Masahito Ueda
  • 2019 Poster: Deep Gamblers: Learning to Abstain with Portfolio Theory »
    Liu Ziyin · Zhikang Wang · Paul Pu Liang · Russ Salakhutdinov · Louis-Philippe Morency · Masahito Ueda
  • 2018 : Poster Session 1 »
    Stefan Gadatsch · Danil Kuzin · Navneet Kumar · Patrick Dallaire · Tom Ryder · Remus-Petru Pop · Nathan Hunt · Adam Kortylewski · Sophie Burkhardt · Mahmoud Elnaggar · Dieterich Lawson · Yifeng Li · Jongha (Jon) Ryu · Juhan Bae · Micha Livne · Tim Pearce · Mariia Vladimirova · Jason Ramapuram · Jiaming Zeng · Xinyu Hu · Jiawei He · Danielle Maddix · Arunesh Mittal · Albert Shaw · Tuan Anh Le · Alexander Sagel · Lisha Chen · Victor Gallego · Mahdi Karami · Zihao Zhang · Tal Kachman · Noah Weber · Matt Benatan · Kumar K Sricharan · Vincent Cartillier · Ivan Ovinnikov · Buu Phan · Mahmoud Hossam · Liu Ziyin · Valerii Kharitonov · Eugene Golikov · Qiang Zhang · Jae Myung Kim · Sebastian Farquhar · Jishnu Mukhoti · Xu Hu · Gregory Gundersen · Lavanya Sita Tekumalla · Paris Perdikaris · Ershad Banijamali · Siddhartha Jain · Ge Liu · Martin Gottwald · Katy Blumer · Sukmin Yun · Ranganath Krishnan · Roman Novak · Yilun Du · Yu Gong · Beliz Gokkaya · Jessica Ai · Daniel Duckworth · Johannes von Oswald · Christian Henning · Louis-Philippe Morency · Ali Ghodsi · Mahesh Subedar · Jean-Pascal Pfister · Rémi Lebret · Chao Ma · Aleksander Wieczorek · Laurence Perreault Levasseur