Timezone: »
We address the problem of fitting 3D human models to 3D scans of dressed humans. Classical methods optimize both the data-to-model correspondences and the human model parameters (pose and shape), but are reliable only when initialised close to the solution. Some methods initialize the optimization based on fully supervised correspondence predictors, which is not differentiable end-to-end, and can only process a single scan at a time. Our main contribution is LoopReg, an end-to-end learning framework to register a corpus of scans to a common 3D human model. The key idea is to create a self-supervised loop. A backward map, parameterized by a Neural Network, predicts the correspondence from every scan point to the surface of the human model. A forward map, parameterized by a human model, transforms the corresponding points back to the scan based on the model parameters (pose and shape), thus closing the loop. Formulating this closed loop is not straightforward because it is not trivial to force the output of the NN to be on the surface of the human model -- outside this surface the human model is not even defined. To this end, we propose two key innovations. First, we define the canonical surface implicitly as the zero level set of a distance field in R3, which in contrast to more common UV parameterizations does not require cutting the surface, does not have discontinuities, and does not induce distortion. Second, we diffuse the human model to the 3D domain. This allows to map the NN predictions forward, even when they slightly deviate from the zero level set. Results demonstrate that we can train LoopReg mainly self-supervised -- following a supervised warm-start, the model becomes increasingly more accurate as additional unlabelled raw scans are processed. Our code and pre-trained models can be downloaded for research.
Author Information
Bharat Lal Bhatnagar (MPI-INF)
Cristian Sminchisescu (Google Research)
Christian Theobalt (MPI Informatik)
Gerard Pons-Moll (MPII, Germany)
Related Events (a corresponding poster, oral, or spotlight)
-
2020 Poster: LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration »
Thu. Dec 10th 05:00 -- 07:00 AM Room Poster Session 4 #1229
More from the Same Authors
-
2021 Spotlight: NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction »
Peng Wang · Lingjie Liu · Yuan Liu · Christian Theobalt · Taku Komura · Wenping Wang -
2023 Poster: DELIFFAS: Deformable Light Fields for Fast Avatar Synthesis »
Youngjoong Kwon · Lingjie Liu · Henry Fuchs · Marc Habermann · Christian Theobalt -
2023 Poster: 3D Open-vocabulary Segmentation with Foundation Models »
Kunhao Liu · Fangneng Zhan · Jiahui Zhang · MUYU XU · Yingchen Yu · Abdulmotaleb El Saddik · Christian Theobalt · Eric Xing · Shijian Lu -
2021 Poster: A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis »
Xingang Pan · Xudong XU · Chen Change Loy · Christian Theobalt · Bo Dai -
2021 Poster: NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction »
Peng Wang · Lingjie Liu · Yuan Liu · Christian Theobalt · Taku Komura · Wenping Wang -
2020 Poster: Neural Unsigned Distance Fields for Implicit Function Learning »
Julian Chibane · Mohamad Aymen mir · Gerard Pons-Moll -
2020 Poster: Neural Sparse Voxel Fields »
Lingjie Liu · Jiatao Gu · Kyaw Zaw Lin · Tat-Seng Chua · Christian Theobalt -
2020 Spotlight: Neural Sparse Voxel Fields »
Lingjie Liu · Jiatao Gu · Kyaw Zaw Lin · Tat-Seng Chua · Christian Theobalt -
2019 : Lunch Break and Posters »
Xingyou Song · Elad Hoffer · Wei-Cheng Chang · Jeremy Cohen · Jyoti Islam · Yaniv Blumenfeld · Andreas Madsen · Jonathan Frankle · Sebastian Goldt · Satrajit Chatterjee · Abhishek Panigrahi · Alex Renda · Brian Bartoldson · Israel Birhane · Aristide Baratin · Niladri Chatterji · Roman Novak · Jessica Forde · YiDing Jiang · Yilun Du · Linara Adilova · Michael Kamp · Berry Weinstein · Itay Hubara · Tal Ben-Nun · Torsten Hoefler · Daniel Soudry · Hsiang-Fu Yu · Kai Zhong · Yiming Yang · Inderjit Dhillon · Jaime Carbonell · Yanqing Zhang · Dar Gilboa · Johannes Brandstetter · Alexander R Johansen · Gintare Karolina Dziugaite · Raghav Somani · Ari Morcos · Freddie Kalaitzis · Hanie Sedghi · Lechao Xiao · John Zech · Muqiao Yang · Simran Kaur · Qianli Ma · Yao-Hung Hubert Tsai · Ruslan Salakhutdinov · Sho Yaida · Zachary Lipton · Daniel Roy · Michael Carbin · Florent Krzakala · Lenka Zdeborová · Guy Gur-Ari · Ethan Dyer · Dilip Krishnan · Hossein Mobahi · Samy Bengio · Behnam Neyshabur · Praneeth Netrapalli · Kris Sankaran · Julien Cornebise · Yoshua Bengio · Vincent Michalski · Samira Ebrahimi Kahou · Md Rifat Arefin · Jiri Hron · Jaehoon Lee · Jascha Sohl-Dickstein · Samuel Schoenholz · David Schwab · Dongyu Li · Sang Choe · Henning Petzka · Ashish Verma · Zhichao Lin · Cristian Sminchisescu -
2019 Poster: Domes to Drones: Self-Supervised Active Triangulation for 3D Human Pose Reconstruction »
Aleksis Pirinen · Erik Gärtner · Cristian Sminchisescu -
2018 Poster: Deep Network for the Integrated 3D Sensing of Multiple People in Natural Images »
Andrei Zanfir · Elisabeta Marinoiu · Mihai Zanfir · Alin-Ionut Popa · Cristian Sminchisescu -
2018 Spotlight: Deep Network for the Integrated 3D Sensing of Multiple People in Natural Images »
Andrei Zanfir · Elisabeta Marinoiu · Mihai Zanfir · Alin-Ionut Popa · Cristian Sminchisescu -
2013 Poster: Action from Still Image Dataset and Inverse Optimal Control to Learn Task Specific Visual Scanpaths »
Stefan Mathe · Cristian Sminchisescu