Timezone: »

Dynamics-regulated kinematic policy for egocentric pose estimation
Zhengyi Luo · Ryo Hachiuma · Ye Yuan · Kris Kitani

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @ Virtual

We propose a method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information. Unlike prior kinematics or dynamics-based approaches where the two components are used disjointly, we synergize the two approaches via dynamics-regulated training. At each timestep, a kinematic model is used to provide a target pose using video evidence and simulation state. Then, a prelearned dynamics model attempts to mimic the kinematic pose in a physics simulator. By comparing the pose instructed by the kinematic model against the pose generated by the dynamics model, we can use their misalignment to further improve the kinematic model. By factoring in the 6DoF pose of objects (e.g., chairs, boxes) in the scene, we demonstrate for the first time, the ability to estimate physically-plausible 3D human-object interactions using a single wearable camera. We evaluate our egocentric pose estimation method in both controlled laboratory settings and real-world scenarios.

Author Information

Zhengyi Luo (Carnegie Mellon University)

Hi there! My name is Zhengyi Luo (Zen) and I am a first year PhD student at Carnegie Mellon University’s Robotics Institute, School of Computer Science, advised by Prof. Kris Kitani. I earned my bachelor’s degree from University of Pennsylvania in 2019, where I worked with Prof. Kostas Daniilidis. My research interest lies at the intersection of computer vision, machine learning, and robotics. I am working on topics including human pose estimation, human-object interaction estimation, human motion generation, etc. Through my research, I want to create methods that effectively interpret spatial-temporal sensory input and build a compositional representation of the 3D world to reason about the interactions between agents and the physical environment. On the application side, I am excited about building systems that would be useful as assistive robots, autonomous vehicles, and AR/VR virtual assistants.

Ryo Hachiuma (Keio University)
Ye Yuan (Carnegie Mellon University)
Kris Kitani (Carnegie Mellon University)

More from the Same Authors