Timezone: »

 
Contrastive Learning on Synthetic Videos for GAN Latent Disentangling
Kevin Duarte · Wei-An Lin · Ratheesh Kalarot · Jingwan (Cynthia) Lu · Eli Shechtman · Shabnam Ghadar · Mubarak Shah

Fri Dec 02 08:54 AM -- 08:56 AM (PST) @
in Synthetic Data for Empowering ML Research »
Event URL: https://openreview.net/forum?id=B9W7BV6fRC »

In this paper, we present a method to disentangle appearance and structural information in the latent space of StyleGAN. We train an autoencoder whose encoder extracts appearance and structural features from an input latent code and then reconstructs the original input using the decoder. To train this network, We propose a video-based latent contrastive learning framework. With the observation that the appearance of a face does not change within a short video, the encoder learns to pull appearance representations of various video frames together while pushing appearance representations of different faces apart. Similarly, the structural representations of augmented versions of the same frame are pulled together, while the representation across different frames are pushed apart. As face video datasets lack sufficient number of unique identities, we propose a method to synthetically generate videos. This allows our disentangling network to observe a larger variation of appearances, expressions, and poses during training. We evaluate our approach on the tasks of expression transfer in images and motion transfer in videos.

Author Information

Kevin Duarte (University of Central Florida)
Wei-An Lin (Adobe Systems)
Ratheesh Kalarot (Adobe Systems)
Jingwan (Cynthia) Lu (Adobe Research)

Jingwan has a passion for data-driven content creation. Her primary research focus is to apply deep generative models for photography applications. Her vision is to harness the power of machine learning in the age of data explosion to invent the next generation image and video editing tools. She also worked on brush models, stylization, guided texture synthesis, voice synthesis, etc. using various data-driven approaches.

Eli Shechtman (Adobe)
Shabnam Ghadar (Adobe Systems)
Mubarak Shah (University of Central Florida)

More from the Same Authors