Timezone: »
Contrastive learning has delivered impressive results for various tasks in the self-supervised regime. However, existing approaches optimize for learning representations specific to downstream scenarios, i.e., global representations suitable for tasks such as classification or local representations for tasks such as detection and localization. While they produce satisfactory results in the intended downstream scenarios, they often fail to generalize to tasks that they were not originally designed for. In this work, we propose to learn video representations that generalize to both the tasks which require global semantic information (e.g., classification) and the tasks that require local fine-grained spatio-temporal information (e.g., localization). We achieve this by optimizing two contrastive objectives that together encourage our model to learn global-local visual information given audio signals. We show that the two objectives mutually improve the generalizability of the learned global-local representations, significantly outperforming their disjointly learned counterparts. We demonstrate our approach on various tasks including action/sound classification, lipreading, deepfake detection, event and sound localization.
Author Information
shuang ma (Microsoft)
Zhaoyang Zeng (SUN YAT-SEN UNIVERSITY)
Daniel McDuff (Microsoft Research)
Yale Song (Microsoft Research)
More from the Same Authors
-
2020 : Paper 64: Modeling Affect-based Intrinsic Rewards for Exploration and Learning »
Daniel McDuff · Ashish Kapoor -
2022 : PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pretraining »
Rogerio Bonatti · Sai Vemprala · shuang ma · Felipe Vieira Frujeri · Shuhang Chen · Ashish Kapoor -
2022 : SMART: Self-supervised Multi-task pretrAining with contRol Transformers »
Yanchao Sun · shuang ma · Ratnesh Madaan · Rogerio Bonatti · Furong Huang · Ashish Kapoor -
2022 : LATTE: LAnguage Trajectory TransformEr »
A Bucker · Luis Figueredo · Sami Haddadin · Ashish Kapoor · shuang ma · Sai Vemprala · Rogerio Bonatti -
2023 Poster: Ego4D Goal-Step: Toward Hierarchical Understanding of Procedural Activities »
Yale Song · Eugene Byrne · Tushar Nagarajan · Huiyu Wang · Miguel Martin · Lorenzo Torresani -
2022 Workshop: Self-Supervised Learning: Theory and Practice »
Ishan Misra · Pengtao Xie · Gul Varol · Yale Song · Yuki Asano · Xiaolong Wang · Pauline Luc -
2020 Poster: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement »
Xin Liu · Josh Fromm · Shwetak Patel · Daniel McDuff -
2020 Oral: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement »
Xin Liu · Josh Fromm · Shwetak Patel · Daniel McDuff -
2019 Poster: Characterizing Bias in Classifiers using Generative Models »
Daniel McDuff · Shuang Ma · Yale Song · Ashish Kapoor