Timezone: »

 
Poster
Unsupervised learning of object structure and dynamics from videos
Matthias Minderer · Chen Sun · Ruben Villegas · Forrester Cole · Kevin Murphy · Honglak Lee

Tue Dec 10 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #149
in Deep Learning -- Predictive Models »

Extracting and predicting object structure and dynamics from videos without supervision is a major challenge in machine learning. To address this challenge, we adopt a keypoint-based image representation and learn a stochastic dynamics model of the keypoints. Future frames are reconstructed from the keypoints and a reference frame. By modeling dynamics in the keypoint coordinate space, we achieve stable learning and avoid compounding of errors in pixel space. Our method improves upon unstructured representations both for pixel-level video prediction and for downstream tasks requiring object-level understanding of motion dynamics. We evaluate our model on diverse datasets: a multi-agent sports dataset, the Human3.6M dataset, and datasets based on continuous control tasks from the DeepMind Control Suite. The spatially structured representation outperforms unstructured representations on a range of motion-related tasks such as object tracking, action recognition and reward prediction.

Keywords: Deep Learning · Predictive Models · Deep Learning
[ Paper]  [ Poster] 

Author Information

Matthias Minderer (Google Research)
Chen Sun (Google Research)
Ruben Villegas (Adobe Research / U. Michigan)
Forrester Cole (Google Research)
Kevin Murphy (Google)
Honglak Lee (Google Brain)

More from the Same Authors