Timezone: »

 
Spotlight
Audio-Driven Co-Speech Gesture Video Generation
Xian Liu · Qianyi Wu · Hang Zhou · Yuanqi Du · Wayne Wu · Dahua Lin · Ziwei Liu

Wed Dec 07 05:00 PM -- 07:00 PM (PST) @
in Lightning Talks 4B-4 »

Co-speech gesture is crucial for human-machine interaction and digital entertainment. While previous works mostly map speech audio to human skeletons (e.g., 2D keypoints), directly generating speakers' gestures in the image domain remains unsolved. In this work, we formally define and study this challenging problem of audio-driven co-speech gesture video generation, i.e., using a unified framework to generate speaker image sequence driven by speech audio. Our key insight is that the co-speech gestures can be decomposed into common motion patterns and subtle rhythmic dynamics. To this end, we propose a novel framework, Audio-driveN Gesture vIdeo gEneration (ANGIE), to effectively capture the reusable co-speech gesture patterns as well as fine-grained rhythmic movements. To achieve high-fidelity image sequence generation, we leverage an unsupervised motion representation instead of a structural human body prior (e.g., 2D skeletons). Specifically, 1) we propose a vector quantized motion extractor (VQ-Motion Extractor) to summarize common co-speech gesture patterns from implicit motion representation to codebooks. 2) Moreover, a co-speech gesture GPT with motion refinement (Co-Speech GPT) is devised to complement the subtle prosodic motion details. Extensive experiments demonstrate that our framework renders realistic and vivid co-speech gesture video. Demo video and more resources can be found in: https://alvinliu0.github.io/projects/ANGIE

Keywords: Co-Speech Gesture · Audio-Visual

Author Information

Xian Liu (The Chinese University of Hong Kong)
Qianyi Wu (Monash University)
Hang Zhou (The Chinese University of Hong Kong)
Yuanqi Du (Cornell University)
Wayne Wu (Tsinghua University)
Dahua Lin (The Chinese University of Hong Kong)
Ziwei Liu (Nanyang Technological University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors