Timezone: »
Poster
$\alpha$-ReQ : Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay
Kumar K Agrawal · Arnab Kumar Mondal · Arna Ghosh · Blake Richards
Self-Supervised Learning (SSL) with large-scale unlabelled datasets enables learning useful representations for multiple downstream tasks. However, assessing the quality of such representations efficiently poses nontrivial challenges. Existing approaches train linear probes (with frozen features) to evaluate performance on a given task. This is expensive both computationally, since it requires retraining a new prediction head for each downstream task, and statistically, requires task-specific labels for multiple tasks. This poses a natural question, how do we efficiently determine the "goodness" of representations learned with SSL across a wide range of potential downstream tasks? In particular, a task-agnostic statistical measure of representation quality, that predicts generalization without explicit downstream task evaluation, would be highly desirable. In this work, we analyze characteristics of learned representations $\mathbf{f_\theta}$, in well-trained neural networks with canonical architectures \& across SSL objectives. We observe that the eigenspectrum of the empirical feature covariance $\mathrm{Cov}(\mathbf{f_\theta}$) can be well approximated with the family of power-law distribution. We analytically and empirically (using multiple datasets, e.g. CIFAR, STL10, MIT67, ImageNet) demonstrate that the decay coefficient $\alpha$ serves as a measure of representation quality for tasks that are solvable with a linear readout, i.e. there exist well-defined intervals for $\alpha$ where models exhibit excellent downstream generalization. Furthermore, our experiments suggest that key design parameters in SSL algorithms, such as BarlowTwins, implicitly modulate the decay coefficient of the eigenspectrum ($\alpha$). As $\alpha$ depends only on the features themselves, this measure for model selection with hyperparameter tuning for BarlowTwins enables search with less compute.
Author Information
Kumar K Agrawal (Indian Institute of Technology, Kharagpur)
Arnab Kumar Mondal (Mila - Quebec AI Institute McGill University)

I am a fourth-year Ph.D. candidate in Computer Science at Mcgill University and Mila - Quebec Artificial Intelligence Institute, supervised by Prof. Siamak Ravanbakhsh and Prof. Kaleem Siddiqi. My primary areas of interest include representation learning, deep reinforcement learning, self-supervised learning, equivariance and geometric deep learning. I am also interested in efficient long-range sequence modeling, uncertainty estimation, generative modeling and computer vision. Before moving to Montreal, I did my undergraduate studies in Electronics and Electrical Engineering at the Indian Institute of Technology, Kharagpur.
Arna Ghosh (McGill University/ Mila/ Meta)
Blake Richards (Mila/McGill)
More from the Same Authors
-
2021 Spotlight: The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning »
Shahab Bakhtiari · Patrick Mineault · Timothy Lillicrap · Christopher Pack · Blake Richards -
2021 Spotlight: Your head is there to move you around: Goal-driven models of the primate dorsal pathway »
Patrick Mineault · Shahab Bakhtiari · Blake Richards · Christopher Pack -
2023 Poster: A Unified, Scalable Framework for Neural Population Decoding »
Mehdi Azabou · Vinam Arora · Venkataramana Ganesh · Ximeng Mao · Santosh Nachimuthu · Michael Mendelson · Blake Richards · Matthew Perich · Guillaume Lajoie · Eva Dyer -
2023 Poster: Formalizing locality for normative synaptic plasticity models »
Colin Bredenberg · Ezekiel Williams · Cristina Savin · Blake Richards · Guillaume Lajoie -
2023 Poster: Learning better with Dale’s Law: A Spectral Perspective »
Pingsheng Li · Jonathan Cornford · Arna Ghosh · Blake Richards -
2023 Poster: Equivariant Adaptation of Large Pre-Trained Models »
Arnab Kumar Mondal · Siba Smarak Panigrahi · Oumar Kaba · Sai Rajeswar Mudumba · Siamak Ravanbakhsh -
2023 Poster: ConSpec: honing in on critical steps for rapid learning and generalization in RL »
Chen Sun · Wannan Yang · Thomas Jiralerspong · Dane Malenfant · Benjamin Alsbury-Nealy · Yoshua Bengio · Blake Richards -
2022 Poster: Structuring Representations Using Group Invariants »
Mehran Shakerinava · Arnab Kumar Mondal · Siamak Ravanbakhsh -
2022 Poster: Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules »
Yuhan Helena Liu · Arna Ghosh · Blake Richards · Eric Shea-Brown · Guillaume Lajoie -
2021 Poster: Adversarial Feature Desensitization »
Pouya Bashivan · Reza Bayat · Adam Ibrahim · Kartik Ahuja · Mojtaba Faramarzi · Touraj Laleh · Blake Richards · Irina Rish -
2021 Poster: The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning »
Shahab Bakhtiari · Patrick Mineault · Timothy Lillicrap · Christopher Pack · Blake Richards -
2021 Poster: Your head is there to move you around: Goal-driven models of the primate dorsal pathway »
Patrick Mineault · Shahab Bakhtiari · Blake Richards · Christopher Pack -
2020 : Closing remarks »
Raymond Chua · Feryal Behbahani · Julie J Lee · Rui Ponte Costa · Doina Precup · Blake Richards · Ida Momennejad -
2020 Workshop: Biological and Artificial Reinforcement Learning »
Raymond Chua · Feryal Behbahani · Julie J Lee · Sara Zannone · Rui Ponte Costa · Blake Richards · Ida Momennejad · Doina Precup -
2020 : Organizers Opening Remarks »
Raymond Chua · Feryal Behbahani · Julie J Lee · Ida Momennejad · Rui Ponte Costa · Blake Richards · Doina Precup