Timezone: »
What shapes the loss landscape of self-supervised learning?
Liu Ziyin · Ekdeep S Lubana · Masahito Ueda · Hidenori Tanaka
Event URL: https://openreview.net/forum?id=J9EyxEpWYVj »
Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL). However, questions remain in our theoretical understanding: Under what precise condition do these collapses occur? We provide theoretically grounded answers to this question by analyzing SSL loss landscapes for a linear model. We derive an analytically tractable theory of SSL landscape and show that it accurately captures an array of collapse phenomena and identifies their causes.
Author Information
Liu Ziyin (University of Tokyo)
Ekdeep S Lubana (University of Michigan; CBS, Harvard University)
Masahito Ueda (University of Tokyo)
Hidenori Tanaka (Harvard University, Harvard University)
More from the Same Authors
-
2022 : Mechanistic Lens on Mode Connectivity »
Ekdeep S Lubana · Eric Bigelow · Robert Dick · David Krueger · Hidenori Tanaka -
2022 : Geometric Considerations for Normalization Layers in Equivariant Neural Networks »
Max Aalto · Ekdeep S Lubana · Hidenori Tanaka -
2022 : A Mechanistic Lens on Mode Connectivity »
Ekdeep S Lubana · Eric Bigelow · Robert Dick · David Krueger · Hidenori Tanaka -
2022 Panel: Panel 1A-2: Posterior Collapse of… & Understanding and Extending… »
Fabrizio Frasca · Liu Ziyin -
2022 Poster: Exact Solutions of a Deep Linear Network »
Liu Ziyin · Botao Li · Xiangming Meng -
2022 Poster: Analyzing Data-Centric Properties for Graph Contrastive Learning »
Puja Trivedi · Ekdeep S Lubana · Mark Heimann · Danai Koutra · Jayaraman Thiagarajan -
2022 Poster: Posterior Collapse of a Linear Latent Variable Model »
Zihao Wang · Liu Ziyin -
2021 Poster: Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning »
Ekdeep S Lubana · Robert Dick · Hidenori Tanaka -
2020 Poster: Neural Networks Fail to Learn Periodic Functions and How to Fix It »
Liu Ziyin · Tilman Hartwig · Masahito Ueda -
2019 Poster: Deep Gamblers: Learning to Abstain with Portfolio Theory »
Liu Ziyin · Zhikang Wang · Paul Pu Liang · Russ Salakhutdinov · Louis-Philippe Morency · Masahito Ueda -
2018 : Poster Session 1 »
Stefan Gadatsch · Danil Kuzin · Navneet Kumar · Patrick Dallaire · Tom Ryder · Remus-Petru Pop · Nathan Hunt · Adam Kortylewski · Sophie Burkhardt · Mahmoud Elnaggar · Dieterich Lawson · Yifeng Li · Jongha (Jon) Ryu · Juhan Bae · Micha Livne · Tim Pearce · Mariia Vladimirova · Jason Ramapuram · Jiaming Zeng · Xinyu Hu · Jiawei He · Danielle Maddix · Arunesh Mittal · Albert Shaw · Tuan Anh Le · Alexander Sagel · Lisha Chen · Victor Gallego · Mahdi Karami · Zihao Zhang · Tal Kachman · Noah Weber · Matt Benatan · Kumar K Sricharan · Vincent Cartillier · Ivan Ovinnikov · Buu Phan · Mahmoud Hossam · Liu Ziyin · Valerii Kharitonov · Eugene Golikov · Qiang Zhang · Jae Myung Kim · Sebastian Farquhar · Jishnu Mukhoti · Xu Hu · Gregory Gundersen · Lavanya Sita Tekumalla · Paris Perdikaris · Ershad Banijamali · Siddhartha Jain · Ge Liu · Martin Gottwald · Katy Blumer · Sukmin Yun · Ranganath Krishnan · Roman Novak · Yilun Du · Yu Gong · Beliz Gokkaya · Jessica Ai · Daniel Duckworth · Johannes von Oswald · Christian Henning · Louis-Philippe Morency · Ali Ghodsi · Mahesh Subedar · Jean-Pascal Pfister · Rémi Lebret · Chao Ma · Aleksander Wieczorek · Laurence Perreault Levasseur