Timezone: »

Functional Regularization for Representation Learning: A Unified Theoretical Perspective
Siddhant Garg · Yingyu Liang

Thu Dec 10 09:00 AM -- 11:00 AM (PST) @ Poster Session 5 #1450

Unsupervised and self-supervised learning approaches have become a crucial tool to learn representations for downstream prediction tasks. While these approaches are widely used in practice and achieve impressive empirical gains, their theoretical understanding largely lags behind. Towards bridging this gap, we present a unifying perspective where several such approaches can be viewed as imposing a regularization on the representation via a learnable function using unlabeled data. We propose a discriminative theoretical framework for analyzing the sample complexity of these approaches, which generalizes the framework of (Balcan and Blum, 2010) to allow learnable regularization functions. Our sample complexity bounds show that, with carefully chosen hypothesis classes to exploit the structure in the data, these learnable regularization functions can prune the hypothesis space, and help reduce the amount of labeled data needed. We then provide two concrete examples of functional regularization, one using auto-encoders and the other using masked self-supervision, and apply our framework to quantify the reduction in the sample complexity bound of labeled data. We also provide complementary empirical results to support our analysis.

Author Information

Siddhant Garg (Amazon Alexa AI Search)

I am an Applied Scientist at Amazon working in the Alexa AI Search team on Question Answering systems. I graduated from the University of Wisconsin-Madison with a Masters in Computer Science in May 2020. I completed my Masters Thesis under the guidance of Prof. Yingyu Liang on representation learning paradigms and their application to text classification. Prior to this, I completed my Bachelors in Computer Science and Engineering from IIT Bombay in August 2018 where I was advised by Prof. Sunita Sarawagi.

Yingyu Liang (University of Wisconsin Madison)

More from the Same Authors