Timezone: »

 
What shapes the loss landscape of self-supervised learning?
Liu Ziyin · Ekdeep S Lubana · Masahito Ueda · Hidenori Tanaka

@
in Symmetry and Geometry in Neural Representations (NeurReps) »
Event URL: https://openreview.net/forum?id=J9EyxEpWYVj »

Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL). However, questions remain in our theoretical understanding: Under what precise condition do these collapses occur? We provide theoretically grounded answers to this question by analyzing SSL loss landscapes for a linear model. We derive an analytically tractable theory of SSL landscape and show that it accurately captures an array of collapse phenomena and identifies their causes.

Keywords: Self-supervised learning · symmetry breaking · Collapse · phase transition

Author Information

Liu Ziyin (University of Tokyo)
Ekdeep S Lubana (University of Michigan; CBS, Harvard University)
Masahito Ueda (University of Tokyo)
Hidenori Tanaka (Harvard University, Harvard University)

More from the Same Authors