Timezone: »

 
Poster
Masked Prediction: A Parameter Identifiability View
Bingbin Liu · Daniel Hsu · Pradeep Ravikumar · Andrej Risteski

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #508
in Poster Session 3 »

The vast majority of work in self-supervised learning have focused on assessing recovered features by a chosen set of downstream tasks. While there are several commonly used benchmark datasets, this lens of feature learning requires assumptions on the downstream tasks which are not inherent to the data distribution itself. In this paper, we present an alternative lens, one of parameter identifiability: assuming data comes from a parametric probabilistic model, we train a self-supervised learning predictor with a suitable parametric form, and ask whether the parameters of the optimal predictor can be used to extract the parameters of the ground truth generative model.Specifically, we focus on latent-variable models capturing sequential structures, namely Hidden Markov Models with both discrete and conditionally Gaussian observations. We focus on masked prediction as the self-supervised learning task and study the optimal masked predictor. We show that parameter identifiability is governed by the task difficulty, which is determined by the choice of data model and the amount of tokens to predict. Technique-wise, we uncover close connections with the uniqueness of tensor rank decompositions, a widely used tool in studying identifiability through the lens of the method of moments.

Keywords: Self-supervised learning · Tensor Decomposition · parameter identifiability · masked prediction
[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Bingbin Liu (Carnegie Mellon University)

Bingbin is a PhD student at the Machine Learning Department of Carnegie Mellon University co-advised by Prof. Pradeep Ravikumar and Prof. Andrej Risteski. Her current research focus is the theoretical understanding of self-supervised learning and representation learning, often motivated by findings in vision and language.

Daniel Hsu (Columbia University)

See <https://www.cs.columbia.edu/~djhsu/>

Pradeep Ravikumar (Carnegie Mellon University)
Andrej Risteski (CMU)

Assistant Professor in the ML department at CMU. Prior to that I was a Wiener Fellow at MIT, and prior to that finished my PhD at Princeton University.

More from the Same Authors