`

Timezone: »

 
Poster
Multimodal Generative Models for Scalable Weakly-Supervised Learning
Mike Wu · Noah Goodman

Tue Dec 04 07:45 AM -- 09:45 AM (PST) @ Room 210 #12
in Tue Poster Session A »

Multiple modalities often co-occur when describing natural phenomena. Learning a joint representation of these modalities should yield deeper and more useful representations.Previous generative approaches to multi-modal input either do not learn a joint distribution or require additional computation to handle missing data. Here, we introduce a multimodal variational autoencoder (MVAE) that uses a product-of-experts inference network and a sub-sampled training paradigm to solve the multi-modal inference problem. Notably, our model shares parameters to efficiently learn under any combination of missing modalities. We apply the MVAE on four datasets and match state-of-the-art performance using many fewer parameters. In addition, we show that the MVAE is directly applicable to weakly-supervised learning, and is robust to incomplete supervision. We then consider two case studies, one of learning image transformations---edge detection, colorization, segmentation---as a set of modalities, followed by one of machine translation between two languages. We find appealing results across this range of tasks.

Keywords: Semi-Supervised Learning · Generative Models
[ Paper] 

Author Information

Mike Wu (Stanford University)
Noah Goodman (Stanford University)

More from the Same Authors