Timezone: »

What shapes the loss landscape of self-supervised learning?
Liu Ziyin · Ekdeep S Lubana · Masahito Ueda · Hidenori Tanaka
Event URL: https://openreview.net/forum?id=J9EyxEpWYVj »

Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL). However, questions remain in our theoretical understanding: Under what precise condition do these collapses occur? We provide theoretically grounded answers to this question by analyzing SSL loss landscapes for a linear model. We derive an analytically tractable theory of SSL landscape and show that it accurately captures an array of collapse phenomena and identifies their causes.

Author Information

Liu Ziyin (University of Tokyo)
Ekdeep S Lubana (University of Michigan; CBS, Harvard University)
Masahito Ueda (University of Tokyo)
Hidenori Tanaka (Harvard University, Harvard University)

More from the Same Authors