Timezone: »

 
Poster
MOMA-LRG: Language-Refined Graphs for Multi-Object Multi-Actor Activity Parsing
Zelun Luo · Zane Durante · Linden Li · Wanze Xie · Ruochen Liu · Emily Jin · Zhuoyi Huang · Lun Yu Li · Jiajun Wu · Juan Carlos Niebles · Ehsan Adeli · Fei-Fei Li

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #1025
in Poster Session 4 »

Video-language models (VLMs), large models pre-trained on numerous but noisy video-text pairs from the internet, have revolutionized activity recognition through their remarkable generalization and open-vocabulary capabilities. While complex human activities are often hierarchical and compositional, most existing tasks for evaluating VLMs focus only on high-level video understanding, making it difficult to accurately assess and interpret the ability of VLMs to understand complex and fine-grained human activities. Inspired by the recently proposed MOMA framework, we define activity graphs as a single universal representation of human activities that encompasses video understanding at the activity, sub-activity, and atomic action level. We redefine activity parsing as the overarching task of activity graph generation, requiring understanding human activities across all three levels. To facilitate the evaluation of models on activity parsing, we introduce MOMA-LRG (Multi-Object Multi-Actor Language-Refined Graphs), a large dataset of complex human activities with activity graph annotations that can be readily transformed into natural language sentences. Lastly, we present a model-agnostic and lightweight approach to adapting and evaluating VLMs by incorporating structured knowledge from activity graphs into VLMs, addressing the individual limitations of language and graphical models. We demonstrate strong performance on few-shot activity parsing, and our framework is intended to foster future research in the joint modeling of videos, graphs, and language.

Keywords: Video Understanding · Scene graph generation · activity recognition · temporal action detection · video-language model · fine-grained activity recognition
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Zelun Luo (Stanford University)
Zane Durante (Stanford University)
Linden Li (Computer Science Department, Stanford University)
Wanze Xie (Stanford University)
Ruochen Liu
Emily Jin (Stanford University)
Zhuoyi Huang (Stanford University)
Zhuoyi Huang

I am a second year master student at Stanford in computer science, my research and industrial interest lay in computer vision, reinforcement learning, full-stack development, and massive data analysis relating to health care or other social goods.

Lun Yu Li (Stanford University)
Jiajun Wu (Stanford University)
Juan Carlos Niebles (Stanford University)
Ehsan Adeli (Stanford University)
Fei-Fei Li (Princeton University)

More from the Same Authors