Timezone: »
We present MubyNet -- a feed-forward, multitask, bottom up system for the integrated localization, as well as 3d pose and shape estimation, of multiple people in monocular images. The challenge is the formal modeling of the problem that intrinsically requires discrete and continuous computation, e.g. grouping people vs. predicting 3d pose. The model identifies human body structures (joints and limbs) in images, groups them based on 2d and 3d information fused using learned scoring functions, and optimally aggregates such responses into partial or complete 3d human skeleton hypotheses under kinematic tree constraints, but without knowing in advance the number of people in the scene and their visibility relations. We design a multi-task deep neural network with differentiable stages where the person grouping problem is formulated as an integer program based on learned body part scores parameterized by both 2d and 3d information. This avoids suboptimality resulting from separate 2d and 3d reasoning, with grouping performed based on the combined representation. The final stage of 3d pose and shape prediction is based on a learned attention process where information from different human body parts is optimally integrated. State-of-the-art results are obtained in large scale datasets like Human3.6M and Panoptic, and qualitatively by reconstructing the 3d shape and pose of multiple people, under occlusion, in difficult monocular images.
Author Information
Andrei Zanfir (Institute of Mathematics of the Romanian Academy)
Elisabeta Marinoiu (IMAR)
Mihai Zanfir (IMAR)
Alin-Ionut Popa (IMAR)
Cristian Sminchisescu (LTH)
Related Events (a corresponding poster, oral, or spotlight)
-
2018 Spotlight: Deep Network for the Integrated 3D Sensing of Multiple People in Natural Images »
Wed. Dec 5th 02:45 -- 02:50 PM Room Room 220 E
More from the Same Authors
-
2021 Poster: REMIPS: Physically Consistent 3D Reconstruction of Multiple Interacting People under Weak Supervision »
Mihai Fieraru · Mihai Zanfir · Teodor Szente · Eduard Bazavan · Vlad Olaru · Cristian Sminchisescu -
2020 Poster: LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration »
Bharat Lal Bhatnagar · Cristian Sminchisescu · Christian Theobalt · Gerard Pons-Moll -
2020 Oral: LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration »
Bharat Lal Bhatnagar · Cristian Sminchisescu · Christian Theobalt · Gerard Pons-Moll -
2019 : Lunch Break and Posters »
Xingyou Song · Elad Hoffer · Wei-Cheng Chang · Jeremy Cohen · Jyoti Islam · Yaniv Blumenfeld · Andreas Madsen · Jonathan Frankle · Sebastian Goldt · Satrajit Chatterjee · Abhishek Panigrahi · Alex Renda · Brian Bartoldson · Israel Birhane · Aristide Baratin · Niladri Chatterji · Roman Novak · Jessica Forde · YiDing Jiang · Yilun Du · Linara Adilova · Michael Kamp · Berry Weinstein · Itay Hubara · Tal Ben-Nun · Torsten Hoefler · Daniel Soudry · Hsiang-Fu Yu · Kai Zhong · Yiming Yang · Inderjit Dhillon · Jaime Carbonell · Yanqing Zhang · Dar Gilboa · Johannes Brandstetter · Alexander R Johansen · Gintare Karolina Dziugaite · Raghav Somani · Ari Morcos · Freddie Kalaitzis · Hanie Sedghi · Lechao Xiao · John Zech · Muqiao Yang · Simran Kaur · Qianli Ma · Yao-Hung Hubert Tsai · Ruslan Salakhutdinov · Sho Yaida · Zachary Lipton · Daniel Roy · Michael Carbin · Florent Krzakala · Lenka Zdeborová · Guy Gur-Ari · Ethan Dyer · Dilip Krishnan · Hossein Mobahi · Samy Bengio · Behnam Neyshabur · Praneeth Netrapalli · Kris Sankaran · Julien Cornebise · Yoshua Bengio · Vincent Michalski · Samira Ebrahimi Kahou · Md Rifat Arefin · Jiri Hron · Jaehoon Lee · Jascha Sohl-Dickstein · Samuel Schoenholz · David Schwab · Dongyu Li · Sang Choe · Henning Petzka · Ashish Verma · Zhichao Lin · Cristian Sminchisescu -
2019 Poster: Domes to Drones: Self-Supervised Active Triangulation for 3D Human Pose Reconstruction »
Aleksis Pirinen · Erik Gärtner · Cristian Sminchisescu -
2013 Poster: Action from Still Image Dataset and Inverse Optimal Control to Learn Task Specific Visual Scanpaths »
Stefan Mathe · Cristian Sminchisescu