Timezone: »

 
Poster
Self-Supervised MultiModal Versatile Networks
Jean-Baptiste Alayrac · Adria Recasens · Rosalia Schneider · Relja Arandjelović · Jason Ramapuram · Jeffrey De Fauw · Lucas Smaira · Sander Dieleman · Andrew Zisserman

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #555
in Poster Session 2 »

Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -- a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51, Kinetics600, AudioSet and ESC-50 when compared to previous self-supervised work. Our models are publicly available.

Author Information

Jean-Baptiste Alayrac (DeepMind)
Adria Recasens (DeepMind)
Rosalia Schneider (DeepMind)
Relja Arandjelović (DeepMind)
Jason Ramapuram (University of Geneva)
Jeffrey De Fauw (DeepMind)
Lucas Smaira (DeepMind)
Sander Dieleman (DeepMind)
Andrew Zisserman (DeepMind & University of Oxford)

More from the Same Authors