Skip to yearly menu bar Skip to main content


Unsupervised Learning of 3D Structure from Images

Danilo Jimenez Rezende · S. M. Ali Eslami · Shakir Mohamed · Peter Battaglia · Max Jaderberg · Nicolas Heess

Area 5+6+7+8 #2

Keywords: [ Variational Inference ] [ (Other) Probabilistic Models and Methods ] [ (Other) Unsupervised Learning Methods ] [ Deep Learning or Neural Networks ]


A key goal of computer vision is to recover the underlying 3D structure that gives rise to 2D observations of the world. If endowed with 3D understanding, agents can abstract away from the complexity of the rendering process to form stable, disentangled representations of scene elements. In this paper we learn strong deep generative models of 3D structures, and recover these structures from 2D images via probabilistic inference. We demonstrate high-quality samples and report log-likelihoods on several datasets, including ShapeNet, and establish the first benchmarks in the literature. We also show how these models and their inference networks can be trained jointly, end-to-end, and directly from 2D images without any use of ground-truth 3D labels. This demonstrates for the first time the feasibility of learning to infer 3D representations of the world in a purely unsupervised manner.

Live content is unavailable. Log in and register to view live content