Timezone: »

 
Spotlight
LGDN: Language-Guided Denoising Network for Video-Language Modeling
Haoyu Lu · Mingyu Ding · Nanyi Fei · Yuqi Huo · Zhiwu Lu

Thu Dec 08 05:00 PM -- 07:00 PM (PST) @
in Lightning Talks 6A-2 »

Video-language modeling has attracted much attention with the rapid growth of web videos. Most existing methods assume that the video frames and text description are semantically correlated, and focus on video-language modeling at video level. However, this hypothesis often fails for two reasons: (1) With the rich semantics of video contents, it is difficult to cover all frames with a single video-level description; (2) A raw video typically has noisy/meaningless information (e.g., scenery shot, transition or teaser). Although a number of recent works deploy attention mechanism to alleviate this problem, the irrelevant/noisy information still makes it very difficult to address. To overcome such challenge, we thus propose an efficient and effective model, termed Language-Guided Denoising Network (LGDN), for video-language modeling. Different from most existing methods that utilize all extracted video frames, LGDN dynamically filters out the misaligned or redundant frames under the language supervision and obtains only 2--4 salient frames per video for cross-modal token-level alignment. Extensive experiments on five public datasets show that our LGDN outperforms the state-of-the-arts by large margins. We also provide detailed ablation study to reveal the critical importance of solving the noise issue, in hope of inspiring future video-language work.

Keywords: Video-Text Retrieval · Video-language modeling · language supervision · cross-modal alignment

Author Information

Haoyu Lu (Renmin University of China)
Mingyu Ding (The University of Hong Kong)
Nanyi Fei (Renmin University of China)
Yuqi Huo (Renmin University of China)
Zhiwu Lu (Renmin University of China)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors