Timezone: »
What shapes the loss landscape of self-supervised learning?
Liu Ziyin · Ekdeep S Lubana · Masahito Ueda · Hidenori Tanaka
Event URL: https://openreview.net/forum?id=J9EyxEpWYVj »
Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL). However, questions remain in our theoretical understanding: Under what precise condition do these collapses occur? We provide theoretically grounded answers to this question by analyzing SSL loss landscapes for a linear model. We derive an analytically tractable theory of SSL landscape and show that it accurately captures an array of collapse phenomena and identifies their causes.
Author Information
Liu Ziyin (University of Tokyo)
Ekdeep S Lubana (University of Michigan; CBS, Harvard University)
Masahito Ueda (University of Tokyo)
Hidenori Tanaka (Harvard University, Harvard University)
More from the Same Authors
-
2022 : Mechanistic Lens on Mode Connectivity »
Ekdeep S Lubana · Eric Bigelow · Robert Dick · David Krueger · Hidenori Tanaka -
2022 : Geometric Considerations for Normalization Layers in Equivariant Neural Networks »
Max Aalto · Ekdeep S Lubana · Hidenori Tanaka -
2022 : A Mechanistic Lens on Mode Connectivity »
Ekdeep S Lubana · Eric Bigelow · Robert Dick · David Krueger · Hidenori Tanaka -
2023 : How does fine-tuning affect your model? Mechanistic analysis on procedural tasks »
Samyak Jain · Robert Kirk · Ekdeep S Lubana · Robert Dick · Hidenori Tanaka · Tim Rocktäschel · Edward Grefenstette · David Krueger -
2023 : Stepwise Inference in Transformers: Exploring a Synthetic Graph Navigation Task »
Mikail Khona · Maya Okawa · Rahul Ramesh · Kento Nishi · Robert Dick · Ekdeep S Lubana · Hidenori Tanaka -
2023 : How Capable Can a Transformer Become? A Study on Synthetic, Interpretable Tasks »
Rahul Ramesh · Mikail Khona · Robert Dick · Hidenori Tanaka · Ekdeep S Lubana -
2023 : Enhanced cue associated memory in temporally consistent recurrent neural networks »
Udith Haputhanthri · Liam Storan · Adam Shai · Surya Ganguli · Mark Schnitzer · Hidenori Tanaka · Fatih Dinc -
2023 : What Does Knowledge Distillation Distill? »
Cindy Wu · Ekdeep S Lubana · Bruno Mlodozeniec · Robert Kirk · David Krueger -
2023 : Subjective Randomness and In-Context Learning »
Eric Bigelow · Ekdeep S Lubana · Robert Dick · Hidenori Tanaka · Tomer Ullman -
2023 : How does fine-tuning affect your model? Mechanistic analysis on procedural tasks »
Samyak Jain · Robert Kirk · Ekdeep S Lubana · Robert Dick · Hidenori Tanaka · Tim Rocktäschel · Edward Grefenstette · David Krueger -
2023 : FoMo rewards: Casting foundation models as generic reward functions »
Ekdeep S Lubana · Pim de Haan · Taco Cohen · Johann Brehmer -
2023 : FoMo rewards: Casting foundation models as generic reward functions »
Ekdeep S Lubana · Pim de Haan · Taco Cohen · Johann Brehmer -
2023 : What Does Knowledge Distillation Distill? »
Cindy Wu · Ekdeep S Lubana · Bruno Mlodozeniec · Robert Kirk · David Krueger -
2023 : Subjective Randomness and In-Context Learning »
Eric Bigelow · Ekdeep S Lubana · Robert Dick · Hidenori Tanaka · Tomer Ullman -
2023 : How does fine-tuning affect your model? Mechanistic analysis on procedural tasks »
Samyak Jain · Robert Kirk · Ekdeep S Lubana · Robert Dick · Hidenori Tanaka · Tim Rocktäschel · Edward Grefenstette · David Krueger -
2023 Poster: Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task »
Maya Okawa · Ekdeep S Lubana · Robert Dick · Hidenori Tanaka -
2023 Poster: CORNN: Convex optimization of recurrent neural networks for rapid inference of neural dynamics »
Fatih Dinc · Adam Shai · Mark Schnitzer · Hidenori Tanaka -
2022 Panel: Panel 1A-2: Posterior Collapse of… & Understanding and Extending… »
Fabrizio Frasca · Liu Ziyin -
2022 Poster: Exact Solutions of a Deep Linear Network »
Liu Ziyin · Botao Li · Xiangming Meng -
2022 Poster: Analyzing Data-Centric Properties for Graph Contrastive Learning »
Puja Trivedi · Ekdeep S Lubana · Mark Heimann · Danai Koutra · Jayaraman Thiagarajan -
2022 Poster: Posterior Collapse of a Linear Latent Variable Model »
Zihao Wang · Liu Ziyin -
2021 Poster: Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning »
Ekdeep S Lubana · Robert Dick · Hidenori Tanaka -
2020 Poster: Neural Networks Fail to Learn Periodic Functions and How to Fix It »
Liu Ziyin · Tilman Hartwig · Masahito Ueda -
2019 Poster: Deep Gamblers: Learning to Abstain with Portfolio Theory »
Liu Ziyin · Zhikang Wang · Paul Pu Liang · Russ Salakhutdinov · Louis-Philippe Morency · Masahito Ueda -
2018 : Poster Session 1 »
Stefan Gadatsch · Danil Kuzin · Navneet Kumar · Patrick Dallaire · Tom Ryder · Remus-Petru Pop · Nathan Hunt · Adam Kortylewski · Sophie Burkhardt · Mahmoud Elnaggar · Dieterich Lawson · Yifeng Li · Jongha (Jon) Ryu · Juhan Bae · Micha Livne · Tim Pearce · Mariia Vladimirova · Jason Ramapuram · Jiaming Zeng · Xinyu Hu · Jiawei He · Danielle Maddix · Arunesh Mittal · Albert Shaw · Tuan Anh Le · Alexander Sagel · Lisha Chen · Victor Gallego · Mahdi Karami · Zihao Zhang · Tal Kachman · Noah Weber · Matt Benatan · Kumar K Sricharan · Vincent Cartillier · Ivan Ovinnikov · Buu Phan · Mahmoud Hossam · Liu Ziyin · Valerii Kharitonov · Eugene Golikov · Qiang Zhang · Jae Myung Kim · Sebastian Farquhar · Jishnu Mukhoti · Xu Hu · Gregory Gundersen · Lavanya Sita Tekumalla · Paris Perdikaris · Ershad Banijamali · Siddhartha Jain · Ge Liu · Martin Gottwald · Katy Blumer · Sukmin Yun · Ranganath Krishnan · Roman Novak · Yilun Du · Yu Gong · Beliz Gokkaya · Jessica Ai · Daniel Duckworth · Johannes von Oswald · Christian Henning · Louis-Philippe Morency · Ali Ghodsi · Mahesh Subedar · Jean-Pascal Pfister · Rémi Lebret · Chao Ma · Aleksander Wieczorek · Laurence Perreault Levasseur