Timezone: »

NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos
Yi-Ling Qiao · Alexander Gao · Ming Lin

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #606

We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input. To decouple the learning of underlying scene geometry from dynamic motion, we represent the scene as a time-invariant signed distance function (SDF) which serves as a reference frame, along with a time-conditioned deformation field. We further bridge this neural geometry representation with a differentiable physics simulator by designing a two-way conversion between the neural field and its corresponding hexahedral mesh, enabling us to estimate physics parameters from the source video by minimizing a cycle consistency loss. Our method also allows a user to interactively edit 3D objects from the source video by modifying the recovered hexahedral mesh, and propagating the operation back to the neural field representation. Experiments show that our method achieves superior mesh and video reconstruction of dynamic scenes compared to competing Neural Field approaches, and we provide extensive examples which demonstrate its ability to extract useful 3D representations from videos captured with consumer-grade cameras.

Author Information

Yi-Ling Qiao (University of Maryland, College Park)
Alexander Gao (University of Maryland, College Park)
Alexander Gao

I am a second-year PhD student in Computer Science at University of Maryland, College Park, advised by Ming C. Lin in the GAMMA Lab. My research interests revolve around Inverse Graphics, Physically-Based Simulation and Scientific Computing.

Ming Lin (University of Maryland - College Park)

Ming C. Lin is currently the Elizabeth Stevinson Iribe Chair of Computer Science at the University of Maryland College Park and John R. & Louise S. Parker Distinguished Professor Emerita of Computer Science at the University of North Carolina (UNC), Chapel Hill. She was also an Honorary Visiting Chair Professor at Tsinghua University in China and at University of Technology Sydney in Australia. She obtained her B.S., M.S., and Ph.D. in Electrical Engineering and Computer Science from the University of California, Berkeley. She received several honors and awards, including the NSF Young Faculty Career Award in 1995, Honda Research Initiation Award in 1997, UNC/IBM Junior Faculty Development Award in 1999, UNC Hettleman Award for Scholarly Achievements in 2003, Beverly W. Long Distinguished Professorship 2007-2010, UNC WOWS Scholar 2009-2011, IEEE VGTC Virtual Reality Technical Achievement Award in 2010, and many best paper awards at international conferences. She is a Fellow of ACM, IEEE, and Eurographics.

More from the Same Authors