Timezone: »

 
Poster
SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
Rishabh Kabra · Daniel Zoran · Goker Erdogan · Loic Matthey · Antonia Creswell · Matt Botvinick · Alexander Lerchner · Chris Burgess

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @
in Poster Session 1 »

To help agents reason about scenes in terms of their building blocks, we wish to extract the compositional structure of any given scene (in particular, the configuration and characteristics of objects comprising the scene). This problem is especially difficult when scene structure needs to be inferred while also estimating the agent’s location/viewpoint, as the two variables jointly give rise to the agent’s observations. We present an unsupervised variational approach to this problem. Leveraging the shared structure that exists across different scenes, our model learns to infer two sets of latent representations from RGB video input alone: a set of "object" latents, corresponding to the time-invariant, object-level contents of the scene, as well as a set of "frame" latents, corresponding to global time-varying elements such as viewpoint. This factorization of latents allows our model, SIMONe, to represent object attributes in an allocentric manner which does not depend on viewpoint. Moreover, it allows us to disentangle object dynamics and summarize their trajectories as time-abstracted, view-invariant, per-object properties. We demonstrate these capabilities, as well as the model's performance in terms of view synthesis and instance segmentation, across three procedurally generated video datasets.

Keywords: Self-Supervised Learning

Author Information

Rishabh Kabra (DeepMind)
Daniel Zoran (DeepMind)
Goker Erdogan (DeepMind)
Loic Matthey (DeepMind)
Antonia Creswell (Imperial College London)
Matt Botvinick (Google DeepMind / University College London)
Alexander Lerchner (DeepMind)
Chris Burgess (Wayve)

More from the Same Authors