Timezone: »

 
Poster
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
Basil Mustafa · Carlos Riquelme · Joan Puigcerver · Rodolphe Jenatton · Neil Houlsby

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #439
in Poster Session 2 »

Large sparsely-activated models have obtained excellent performance in multiple domains.However, such models are typically trained on a single modality at a time.We present the Language-Image MoE, LIMoE, a sparse mixture of experts model capable of multimodal learning.LIMoE accepts both images and text simultaneously, while being trained using a contrastive loss.MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities.However, new challenges arise; in particular, training stability and balanced expert utilization, for which we propose an entropy-based regularization scheme.Across multiple scales, we demonstrate performance improvement over dense models of equivalent computational cost.LIMoE-L/16 trained comparably to CLIP-L/14 achieves 77.9% zero-shot ImageNet accuracy (vs. 76.2%), and when further scaled to H/14 (with additional data) it achieves 83.8%, approaching state-of-the-art methods which use custom per-modality backbones and pre-training schemes.We analyse the quantitative and qualitative behavior of LIMoE, and demonstrate phenomena such as differing treatment of the modalities and the emergence of modality-specific experts.

Keywords: Computer Vision · Natural Language Processing · machine learning · contrastive learning · mixture of experts · sparse computation · multimodal · conditional computation · zeroshot classification

Author Information

Basil Mustafa (Google)
Carlos Riquelme (Google Brain)
Joan Puigcerver (Google Research)
Rodolphe Jenatton (Amazon Research)
Neil Houlsby (Google)

More from the Same Authors