Timezone: »

 
Poster
Direct Multi-view Multi-person 3D Pose Estimation
tao wang · Jianfeng Zhang · Yujun Cai · Shuicheng Yan · Jiashi Feng

Tue Dec 07 04:30 PM -- 06:00 PM (PST) @
in Poster Session 2 »

We present Multi-view Pose transformer (MvP) for estimating multi-person 3D poses from multi-view images. Instead of estimating 3D joint locations from costly volumetric representation or reconstructing the per-person 3D pose from multiple detected 2D poses as in previous methods, MvP directly regresses the multi-person 3D poses in a clean and efficient way, without relying on intermediate tasks. Specifically, MvP represents skeleton joints as learnable query embeddings and let them progressively attend to and reason over the multi-view information from the input images to directly regress the actual 3D joint locations. To improve the accuracy of such a simple pipeline, MvP presents a hierarchical scheme to concisely represent query embeddings of multi-person skeleton joints and introduces an input-dependent query adaptation approach. Further, MvP designs a novel geometrically guided attention mechanism, called projective attention, to more precisely fuse the cross-view information for each joint. MvP also introduces a RayConv operation to integrate the view-dependent camera geometry into the feature representations for augmenting the projective attention. We show experimentally that our MvP model outperforms the state-of-the-art methods on several benchmarks while being much more efficient. Notably, it achieves 92.3% AP25 on the challenging Panoptic dataset, improving upon the previous best approach [35] by 9.8%. MvP is general and also extendable to recovering human mesh represented by the SMPL model, thus useful for modeling multi-person body shapes. Code and models are available at https://github.com/sail-sg/mvp.

Keywords: Transformers · Vision

Author Information

tao wang (national university of singapore)
Jianfeng Zhang (NUS)
Yujun Cai (Nanyang Technological University)
Shuicheng Yan (National University of Singapore)
Jiashi Feng (UC Berkeley)

More from the Same Authors