Timezone: »

 
Tutorial
Self-Supervised Learning: Self-Prediction and Contrastive Learning
Lilian Weng · Jong Wook Kim

Mon Dec 06 05:00 PM -- 07:21 PM (PST) @ Virtual

Self-supervised learning is a great way to extract training signals from massive amounts of unlabelled data and to learn good representation to facilitate downstream tasks where it is expensive to collect task-specific labels. This tutorial will focus on two major approaches for self-supervised learning, self-prediction and contrastive learning. Self-prediction refers to self-supervised training tasks where the model learns to predict a portion of the available data from the rest. Contrastive learning is to learn a representation space in which similar data samples stay close to each other while dissimilar ones are far apart, by constructing similar and dissimilar pairs from the dataset. This tutorial will cover methods on both topics and across various applications including vision, language, video, multimodal, and reinforcement learning.

[ Slides] 

Author Information

Lilian Weng (OpenAI)

Lilian Weng is working at OpenAI over a variety of research and applied projects. In the Robotics team, she worked on several challenging robotic manipulation tasks, including solving a fully scrambled Rubik's cube with a single robot hand, via deep reinforcement learning and sim2real transfer techniques. Currently she leads the Applied AI Research team to use powerful language models to solve real-world applications. Her research interests are quite broad, as she writes about various topics in deep learning in her highly viewed ML blog https://lilianweng.github.io/lil-log/.

Jong Wook Kim (OpenAI)

Jong Wook Kim is a member of technical staff at OpenAI, where he worked on GPT-2 output detection, Jukebox, and CLIP. His research interests include representation learning and generative modeling of audio and music, as well as its applications to multimodal deep learning. Prior to OpenAI, he completed a Ph.D. in music technology from NYU, which focused on automatic music transcription. He also worked as a research scientist intern at Pandora and Spotify, and as a software engineer at Kakao and NCSOFT.

More from the Same Authors