Timezone: »
Co-speech gesture is crucial for human-machine interaction and digital entertainment. While previous works mostly map speech audio to human skeletons (e.g., 2D keypoints), directly generating speakers' gestures in the image domain remains unsolved. In this work, we formally define and study this challenging problem of audio-driven co-speech gesture video generation, i.e., using a unified framework to generate speaker image sequence driven by speech audio. Our key insight is that the co-speech gestures can be decomposed into common motion patterns and subtle rhythmic dynamics. To this end, we propose a novel framework, Audio-driveN Gesture vIdeo gEneration (ANGIE), to effectively capture the reusable co-speech gesture patterns as well as fine-grained rhythmic movements. To achieve high-fidelity image sequence generation, we leverage an unsupervised motion representation instead of a structural human body prior (e.g., 2D skeletons). Specifically, 1) we propose a vector quantized motion extractor (VQ-Motion Extractor) to summarize common co-speech gesture patterns from implicit motion representation to codebooks. 2) Moreover, a co-speech gesture GPT with motion refinement (Co-Speech GPT) is devised to complement the subtle prosodic motion details. Extensive experiments demonstrate that our framework renders realistic and vivid co-speech gesture video. Demo video and more resources can be found in: https://alvinliu0.github.io/projects/ANGIE
Author Information
Xian Liu (The Chinese University of Hong Kong)
Qianyi Wu (Monash University)
Hang Zhou (The Chinese University of Hong Kong)
Yuanqi Du (Cornell University)
Wayne Wu (Tsinghua University)
Dahua Lin (The Chinese University of Hong Kong)
Ziwei Liu (Nanyang Technological University)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: Audio-Driven Co-Speech Gesture Video Generation »
Dates n/a. Room
More from the Same Authors
-
2021 : GraphGT: Machine Learning Datasets for Graph Generation and Transformation »
Yuanqi Du · Shiyu Wang · Xiaojie Guo · Hengning Cao · Shujie Hu · Junji Jiang · Aishwarya Varala · Abhinav Angirekula · Liang Zhao -
2021 : Learning Disentangled Representation for Spatiotemporal Graph Generation »
Yuanqi Du · Xiaojie Guo · Hengning Cao · Yanfang (Fa Ye · Liang Zhao -
2021 : Semi-Supervised Domain Generalization with Stochastic StyleMatch »
Kaiyang Zhou · Chen Change Loy · Ziwei Liu -
2021 : GraphGT: Machine Learning Datasets for Graph Generation and Transformation »
Yuanqi Du · Shiyu Wang · Xiaojie Guo · Hengning Cao · Shujie Hu · Junji Jiang · Aishwarya Varala · Abhinav Angirekula · Liang Zhao -
2021 : Physics-Augmented Learning: A New Paradigm Beyond Physics-Informed Learning »
Ziming Liu · Yuanqi Du · Yunyue Chen · Max Tegmark -
2021 : Learning Disentangled Representation for Spatiotemporal Graph Generation »
Yuanqi Du · Xiaojie Guo · Hengning Cao · Yanfang (Fa Ye · Liang Zhao -
2022 Poster: Delving into Sequential Patches for Deepfake Detection »
Jiazhi Guan · Hang Zhou · Zhibin Hong · Errui Ding · Jingdong Wang · Chengbin Quan · Youjian Zhao -
2022 Poster: Benchmarking and Analyzing 3D Human Pose and Shape Estimation Beyond Algorithms »
Hui En Pang · Zhongang Cai · Lei Yang · Tianwei Zhang · Ziwei Liu -
2022 : GAUCHE: A Library for Gaussian Processes in Chemistry »
Ryan-Rhys Griffiths · Leo Klarner · Henry Moss · Aditya Ravuri · Sang Truong · Bojana Rankovic · Yuanqi Du · Arian Jamasb · Julius Schwartz · Austin Tripp · Gregory Kell · Anthony Bourached · Alex Chan · Jacob Moss · Chengzhi Guo · Alpha Lee · Philippe Schwaller · Jian Tang -
2022 : PIPS: Path Integral Stochastic Optimal Control for Path Sampling in Molecular Dynamics »
Lars Holdijk · Yuanqi Du · Ferry Hooft · Priyank Jaini · Berend Ensing · Max Welling -
2022 : Structural Causal Model for Molecular Dynamics Simulation »
Qi Liu · Yuanqi Du · Fan Feng · Qiwei Ye · Jie Fu -
2022 : Xtal2DoS: Attention-based Crystal to Sequence Learning for Density of States Prediction »
Junwen Bai · Yuanqi Du · Yingheng Wang · Shufeng Kong · John Gregoire · Carla Gomes -
2022 : ChemSpacE: Interpretable and Interactive Chemical Space Exploration »
Yuanqi Du · Xian Liu · Nilay Shah · Shengchao Liu · Jieyu Zhang · Bolei Zhou -
2022 : Structure-based Drug Design with Equivariant Diffusion Models »
Arne Schneuing · Yuanqi Du · Charles Harris · Arian Jamasb · Ilia Igashov · weitao Du · Tom Blundell · Pietro Lió · Carla Gomes · Max Welling · Michael Bronstein · Bruno Correia -
2022 : Improving Molecular Pretraining with Complementary Featurizations »
Yanqiao Zhu · Dingshuo Chen · Yuanqi Du · Yingze Wang · Qiang Liu · Shu Wu -
2022 : Sparse Mixture-of-Experts are Domain Generalizable Learners »
Bo Li · Yifei Shen · Jingkang Yang · Yezhen Wang · Jiawei Ren · Tong Che · Jun Zhang · Ziwei Liu -
2023 Poster: GAUCHE: A Library for Gaussian Processes in Chemistry »
Ryan-Rhys Griffiths · Leo Klarner · Henry Moss · Aditya Ravuri · Sang Truong · Yuanqi Du · Samuel Stanton · Gary Tom · Bojana Rankovic · Arian Jamasb · Aryan Deshwal · Julius Schwartz · Austin Tripp · Gregory Kell · Simon Frieder · Anthony Bourached · Alex Chan · Jacob Moss · Chengzhi Guo · Johannes Peter Dürholt · Saudamini Chaurasia · Ji Won Park · Felix Strieth-Kalthoff · Alpha Lee · Bingqing Cheng · Alan Aspuru-Guzik · Philippe Schwaller · Jian Tang -
2023 Poster: What Makes Good Examples for Visual In-Context Learning? »
Yuanhan Zhang · Kaiyang Zhou · Ziwei Liu -
2023 Poster: Segment Any Point Cloud Sequences by Distilling Vision Foundation Models »
Youquan Liu · Lingdong Kong · Jun CEN · Runnan Chen · Wenwei Zhang · Liang Pan · Kai Chen · Ziwei Liu -
2023 Poster: InsActor: Instruction-driven Physics-based Characters »
Jiawei Ren · Mingyuan Zhang · Cunjun Yu · Xiao Ma · Liang Pan · Ziwei Liu -
2023 Poster: A new perspective on building efficient and expressive 3D equivariant graph neural networks »
weitao Du · Yuanqi Du · Limei Wang · Dieqiao Feng · Guifeng Wang · Shuiwang Ji · Carla Gomes · Zhi-Ming Ma -
2023 Poster: On Separate Normalization in Self-supervised Transformers »
Xiaohui Chen · Yinkai Wang · Yuanqi Du · Soha Hassoun · Liping Liu -
2023 Poster: Towards Robust and Expressive Whole-body Human Pose and Shape Estimation »
Hui En Pang · Zhongang Cai · Lei Yang · Tianwei Zhang · Qingyi Tao · Zhonghua Wu · Ziwei Liu -
2023 Poster: Stochastic Optimal Control for Collective Variable Free Sampling of Molecular Transition Paths »
Lars Holdijk · Yuanqi Du · Ferry Hooft · Priyank Jaini · Berend Ensing · Max Welling -
2023 Poster: FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing »
Mingyuan Zhang · Huirong Li · Zhongang Cai · Jiawei Ren · Lei Yang · Ziwei Liu -
2023 Poster: 4D Panoptic Scene Graph Generation »
Jingkang Yang · Jun CEN · WENXUAN PENG · Shuai Liu · Fangzhou Hong · Xiangtai Li · Kaiyang Zhou · Qifeng Chen · Ziwei Liu -
2023 Poster: PrimDiffusion: Volumetric Primitives Diffusion for 3D Human Generation »
Zhaoxi Chen · Fangzhou Hong · Haiyi Mei · Guangcong Wang · Lei Yang · Ziwei Liu -
2023 Poster: Language Models are Visual Reasoning Coordinators »
Liangyu Chen · Bo Li · Sheng Shen · Jingkang Yang · Chunyuan Li · Kurt Keutzer · Trevor Darrell · Ziwei Liu -
2023 Poster: SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation »
Zhongang Cai · Wanqi Yin · Ailing Zeng · CHEN WEI · Qingping SUN · Yanjun Wang · Hui En Pang · Haiyi Mei · Mingyuan Zhang · Lei Zhang · Chen Change Loy · Lei Yang · Ziwei Liu -
2023 Poster: M$^2$Hub: Unlocking the Potential of Machine Learning for Materials Discovery »
Yuanqi Du · Yingheng Wang · Yining Huang · Jianan Canal Li · Yanqiao Zhu · Tian Xie · Chenru Duan · John Gregoire · Carla Gomes -
2023 Poster: Uncovering Neural Scaling Law in Molecular Representation Learning »
Dingshuo Chen · Yanqiao Zhu · Jieyu Zhang · Yuanqi Du · Zhixun Li · Qiang Liu · Shu Wu · Liang Wang -
2023 Poster: RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars »
Dongwei Pan · Long Zhuo · Jingtan Piao · Huiwen Luo · Wei Cheng · Yuxin WANG · Siming Fan · Shengqi Liu · Lei Yang · Bo Dai · Ziwei Liu · Chen Change Loy · Chen Qian · Wayne Wu · Dahua Lin · Kwan-Yee Lin -
2023 Workshop: AI for Science: from Theory to Practice »
Yuanqi Du · Max Welling · Yoshua Bengio · Marinka Zitnik · Carla Gomes · Jure Leskovec · Maria Brbic · Wenhao Gao · Kexin Huang · Ziming Liu · Rocío Mercado · Miles Cranmer · Shengchao Liu · Lijing Wang -
2023 Workshop: Computational Sustainability: Promises and Pitfalls from Theory to Deployment »
Suzanne Stathatos · Christopher Yeh · Laura Greenstreet · Tarun Sharma · Katelyn Morrison · Yuanqi Du · Chenlin Meng · Sherrie Wang · Fei Fang · Pietro Perona · Yoshua Bengio -
2022 : Factor Investing with a Deep Multi-Factor Model »
Zikai Wei · Bo Dai · Dahua Lin -
2022 Spotlight: Delving into Sequential Patches for Deepfake Detection »
Jiazhi Guan · Hang Zhou · Zhibin Hong · Errui Ding · Jingdong Wang · Chengbin Quan · Youjian Zhao -
2022 Spotlight: Lightning Talks 4B-4 »
Ziyue Jiang · Zeeshan Khan · Yuxiang Yang · Chenze Shao · Yichong Leng · Zehao Yu · Wenguan Wang · Xian Liu · Zehua Chen · Yang Feng · Qianyi Wu · James Liang · C.V. Jawahar · Junjie Yang · Zhe Su · Songyou Peng · Yufei Xu · Junliang Guo · Michael Niemeyer · Hang Zhou · Zhou Zhao · Makarand Tapaswi · Dongfang Liu · Qian Yang · Torsten Sattler · Yuanqi Du · Haohe Liu · Jing Zhang · Andreas Geiger · Yi Ren · Long Lan · Jiawei Chen · Wayne Wu · Dahua Lin · Dacheng Tao · Xu Tan · Jinglin Liu · Ziwei Liu · 振辉 叶 · Danilo Mandic · Lei He · Xiangyang Li · Tao Qin · sheng zhao · Tie-Yan Liu -
2022 Spotlight: Benchmarking and Analyzing 3D Human Pose and Shape Estimation Beyond Algorithms »
Hui En Pang · Zhongang Cai · Lei Yang · Tianwei Zhang · Ziwei Liu -
2022 Workshop: AI for Science: Progress and Promises »
Yi Ding · Yuanqi Du · Tianfan Fu · Hanchen Wang · Anima Anandkumar · Yoshua Bengio · Anthony Gitter · Carla Gomes · Aviv Regev · Max Welling · Marinka Zitnik -
2022 Poster: Semi-Supervised Semantic Segmentation via Gentle Teaching Assistant »
Ying Jin · Jiaqi Wang · Dahua Lin -
2022 Poster: Multi-objective Deep Data Generation with Correlated Property Control »
Shiyu Wang · Xiaojie Guo · Xuanyang Lin · Bo Pan · Yuanqi Du · Yinkai Wang · Yanfang Ye · Ashley Petersen · Austin Leitgeb · Saleh Alkhalifa · Kevin Minbiole · William M. Wuest · Amarda Shehu · Liang Zhao -
2022 Poster: AnimeRun: 2D Animation Visual Correspondence from Open Source 3D Movies »
Li Siyao · Yuhang Li · Bo Li · Chao Dong · Ziwei Liu · Chen Change Loy -
2022 Poster: Graphein - a Python Library for Geometric Deep Learning and Network Analysis on Biomolecular Structures and Interaction Networks »
Arian Jamasb · Ramon Viñas Torné · Eric Ma · Yuanqi Du · Charles Harris · Kexin Huang · Dominic Hall · Pietro Lió · Tom Blundell -
2022 Poster: OpenOOD: Benchmarking Generalized Out-of-Distribution Detection »
Jingkang Yang · Pengyun Wang · Dejian Zou · Zitang Zhou · Kunyuan Ding · WENXUAN PENG · Haoqi Wang · Guangyao Chen · Bo Li · Yiyou Sun · Xuefeng Du · Kaiyang Zhou · Wayne Zhang · Dan Hendrycks · Yixuan Li · Ziwei Liu -
2021 Workshop: AI for Science: Mind the Gaps »
Payal Chandak · Yuanqi Du · Tianfan Fu · Wenhao Gao · Kexin Huang · Shengchao Liu · Ziming Liu · Gabriel Spadon · Max Tegmark · Hanchen Wang · Adrian Weller · Max Welling · Marinka Zitnik -
2021 Poster: Generative Occupancy Fields for 3D Surface-Aware Image Synthesis »
Xudong XU · Xingang Pan · Dahua Lin · Bo Dai -
2021 Poster: Balanced Chamfer Distance as a Comprehensive Metric for Point Cloud Completion »
Tong Wu · Liang Pan · Junzhe Zhang · Tai WANG · Ziwei Liu · Dahua Lin -
2021 Poster: Garment4D: Garment Reconstruction from Point Cloud Sequences »
Fangzhou Hong · Liang Pan · Zhongang Cai · Ziwei Liu -
2021 Poster: Few-Shot Object Detection via Association and DIscrimination »
Yuhang Cao · Jiaqi Wang · Ying Jin · Tong Wu · Kai Chen · Ziwei Liu · Dahua Lin -
2021 Poster: Unsupervised Object-Level Representation Learning from Scene Images »
Jiahao Xie · Xiaohang Zhan · Ziwei Liu · Yew Soon Ong · Chen Change Loy -
2019 : Poster and Coffee Break 1 »
Aaron Sidford · Aditya Mahajan · Alejandro Ribeiro · Alex Lewandowski · Ali H Sayed · Ambuj Tewari · Angelika Steger · Anima Anandkumar · Asier Mujika · Hilbert J Kappen · Bolei Zhou · Byron Boots · Chelsea Finn · Chen-Yu Wei · Chi Jin · Ching-An Cheng · Christina Yu · Clement Gehring · Craig Boutilier · Dahua Lin · Daniel McNamee · Daniel Russo · David Brandfonbrener · Denny Zhou · Devesh Jha · Diego Romeres · Doina Precup · Dominik Thalmeier · Eduard Gorbunov · Elad Hazan · Elena Smirnova · Elvis Dohmatob · Emma Brunskill · Enrique Munoz de Cote · Ethan Waldie · Florian Meier · Florian Schaefer · Ge Liu · Gergely Neu · Haim Kaplan · Hao Sun · Hengshuai Yao · Jalaj Bhandari · James A Preiss · Jayakumar Subramanian · Jiajin Li · Jieping Ye · Jimmy Smith · Joan Bas Serrano · Joan Bruna · John Langford · Jonathan Lee · Jose A. Arjona-Medina · Kaiqing Zhang · Karan Singh · Yuping Luo · Zafarali Ahmed · Zaiwei Chen · Zhaoran Wang · Zhizhong Li · Zhuoran Yang · Ziping Xu · Ziyang Tang · Yi Mao · David Brandfonbrener · Shirli Di-Castro · Riashat Islam · Zuyue Fu · Abhishek Naik · Saurabh Kumar · Benjamin Petit · Angeliki Kamoutsi · Simone Totaro · Arvind Raghunathan · Rui Wu · Donghwan Lee · Dongsheng Ding · Alec Koppel · Hao Sun · Christian Tjandraatmadja · Mahdi Karami · Jincheng Mei · Chenjun Xiao · Junfeng Wen · Zichen Zhang · Ross Goroshin · Mohammad Pezeshki · Jiaqi Zhai · Philip Amortila · Shuo Huang · Mariya Vasileva · El houcine Bergou · Adel Ahmadyan · Haoran Sun · Sheng Zhang · Lukas Gruber · Yuanhao Wang · Tetiana Parshakova -
2019 Poster: Policy Continuation with Hindsight Inverse Dynamics »
Hao Sun · Zhizhong Li · Xiaotong Liu · Bolei Zhou · Dahua Lin -
2019 Spotlight: Policy Continuation with Hindsight Inverse Dynamics »
Hao Sun · Zhizhong Li · Xiaotong Liu · Bolei Zhou · Dahua Lin -
2018 Poster: A Neural Compositional Paradigm for Image Captioning »
Bo Dai · Sanja Fidler · Dahua Lin -
2018 Poster: Trajectory Convolution for Action Recognition »
Yue Zhao · Yuanjun Xiong · Dahua Lin -
2017 Poster: Contrastive Learning for Image Captioning »
Bo Dai · Dahua Lin -
2013 Poster: Online Learning of Nonparametric Mixture Models via Sequential Variational Approximation »
Dahua Lin -
2012 Poster: Coupling Nonparametric Mixtures via Latent Dirichlet Processes »
Dahua Lin · John Fisher III -
2010 Oral: Construction of Dependent Dirichlet Processes based on Poisson Processes »
Dahua Lin · Eric Grimson · John Fisher III -
2010 Poster: Construction of Dependent Dirichlet Processes based on Poisson Processes »
Dahua Lin · Eric Grimson · John Fisher III