Timezone: »

 
Poster
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Antoine Yang · Antoine Miech · Josef Sivic · Ivan Laptev · Cordelia Schmid

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #238
in Poster Session 2 »

Video question answering (VideoQA) is a complex task that requires diverse multi-modal data for training. Manual annotation of question and answers for videos, however, is tedious and prohibits scalability. To tackle this problem, recent methods consider zero-shot settings with no manual annotation of visual question-answer. In particular, a promising approach adapts frozen autoregressive language models pretrained on Web-scale text-only data to multi-modal inputs. In contrast, we here build on frozen bidirectional language models (BiLM) and show that such an approach provides a stronger and cheaper alternative for zero-shot VideoQA. In particular, (i) we combine visual inputs with the frozen BiLM using light trainable modules, (ii) we train such modules using Web-scraped multi-modal data, and finally (iii) we perform zero-shot VideoQA inference through masked language modeling, where the masked text is the answer to a given question. Our proposed approach, FrozenBiLM, outperforms the state of the art in zero-shot VideoQA by a significant margin on a variety of datasets, including LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA, TGIF-FrameQA, How2QA and TVQA. It also demonstrates competitive performance in the few-shot and fully-supervised setting. Our code and models are publicly available at https://github.com/antoyang/FrozenBiLM.

Keywords: Computer Vision · Zero-Shot · vision and language · Video Question Answering
[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Antoine Yang (Inria)
Antoine Miech (DeepMind)
Josef Sivic (Czech Technical University in Prague)
Ivan Laptev (INRIA)
Cordelia Schmid (INRIA)

More from the Same Authors