Timezone: »

 
Poster
Inverse Problems Leveraging Pre-trained Contrastive Representations
Sriram Ravula · Georgios Smyrnis · Matt Jordan · Alex Dimakis

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @
in Poster Session 6 »

We study a new family of inverse problems for recovering representations of corrupted data. We assume access to a pre-trained representation learning network R(x) that operates on clean images, like CLIP. The problem is to recover the representation of an image R(x), if we are only given a corrupted version A(x), for some known forward operator A. We propose a supervised inversion method that uses a contrastive objective to obtain excellent representations for highly corrupted images. Using a linear probe on our robust representations, we achieve a higher accuracy than end-to-end supervised baselines when classifying images with various types of distortions, including blurring, additive noise, and random pixel masking. We evaluate on a subset of ImageNet and observe that our method is robust to varying levels of distortion. Our method outperforms end-to-end baselines even with a fraction of the labeled data in a wide range of forward operators.

Keywords: Robustness · Contrastive Learning · Representation Learning

Author Information

Sriram Ravula (University of Texas, Austin)
Georgios Smyrnis (University of Texas at Austin)
Matt Jordan (UT Austin)
Alex Dimakis (University of Texas, Austin)

More from the Same Authors