Timezone: »
For embodied agents to infer representations of the underlying 3D physical world they inhabit, they should efficiently combine multisensory cues from numerous trials, e.g., by looking at and touching objects. Despite its importance, multisensory 3D scene representation learning has received less attention compared to the unimodal setting. In this paper, we propose the Generative Multisensory Network (GMN) for learning latent representations of 3D scenes which are partially observable through multiple sensory modalities. We also introduce a novel method, called the Amortized Product-of-Experts, to improve the computational efficiency and the robustness to unseen combinations of modalities at test time. Experimental results demonstrate that the proposed model can efficiently infer robust modality-invariant 3D-scene representations from arbitrary combinations of modalities and perform accurate cross-modal generation. To perform this exploration we have also developed a novel multi-sensory simulation environment for embodied agents.
Author Information
Jae Hyun Lim (Mila, University of Montreal)
Pedro O. Pinheiro (Element AI)
Negar Rostamzadeh (Elemenet AI)
Chris Pal (MILA, Polytechnique Montréal, Element AI)
Sungjin Ahn (Rutgers University)
More from the Same Authors
-
2021 Spotlight: A Variational Perspective on Diffusion-Based Generative Models and Score Matching »
Chin-Wei Huang · Jae Hyun Lim · Aaron Courville -
2021 : Artsheets for Art Datasets »
Ramya Srinivasan · Remi Denton · Jordan Famularo · Negar Rostamzadeh · Fernando Diaz · Beth Coleman -
2021 : Thinking Beyond Distributions in Testing Machine Learned Models »
Negar Rostamzadeh · Ben Hutchinson · Vinodkumar Prabhakaran -
2021 : DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations »
Fei Deng · Ingook Jang · Sungjin Ahn -
2021 : TransDreamer: Reinforcement Learning with Transformer World Models »
· Jaesik Yoon · Yi-Fu Wu · Sungjin Ahn -
2021 : Learning Representations for Zero-Shot Image Generation without Text »
Gautam Singh · Fei Deng · Sungjin Ahn -
2023 Poster: Object-Centric Slot Diffusion »
Jindong Jiang · Fei Deng · Gautam Singh · Sungjin Ahn -
2023 Poster: 3D molecule generation by denoising voxel grids »
Pedro O. Pinheiro · Joshua Rackers · Joseph Kleinhenz · Michael Maser · Omar Mahmood · Andrew Watkins · Stephen Ra · Vishnu Sresht · Saeed Saremi -
2023 Poster: Facing-off World Model Backbones: RNN, Transformer, and S4 »
Fei Deng · Junyeong Park · Sungjin Ahn -
2023 Poster: Imagine the Unseen World: A Systematic Visual Imagination Benchmark »
Yeongbin Kim · Gautam Singh · Junyeong Park · Caglar Gulcehre · Sungjin Ahn -
2022 Poster: Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos »
Gautam Singh · Yi-Fu Wu · Sungjin Ahn -
2021 Poster: A Variational Perspective on Diffusion-Based Generative Models and Score Matching »
Chin-Wei Huang · Jae Hyun Lim · Aaron Courville -
2020 : Invited Talk: Sungjin Ahn »
Sungjin Ahn -
2020 Poster: Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning »
Julien Roy · Paul Barde · Félix Harvey · Derek Nowrouzezahrai · Chris Pal -
2020 Poster: Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization »
Paul Barde · Julien Roy · Wonseok Jeon · Joelle Pineau · Chris Pal · Derek Nowrouzezahrai -
2020 Poster: Unsupervised Learning of Dense Visual Representations »
Pedro O. Pinheiro · Amjad Almahairi · Ryan Benmalek · Florian Golemo · Aaron Courville -
2020 Poster: Generative Neurosymbolic Machines »
Jindong Jiang · Sungjin Ahn -
2020 Spotlight: Generative Neurosymbolic Machines »
Jindong Jiang · Sungjin Ahn -
2020 Spotlight: Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization »
Paul Barde · Julien Roy · Wonseok Jeon · Joelle Pineau · Chris Pal · Derek Nowrouzezahrai -
2019 : Poster Session »
Jonathan Scarlett · Piotr Indyk · Ali Vakilian · Adrian Weller · Partha P Mitra · Benjamin Aubin · Bruno Loureiro · Florent Krzakala · Lenka Zdeborová · Kristina Monakhova · Joshua Yurtsever · Laura Waller · Hendrik Sommerhoff · Michael Moeller · Rushil Anirudh · Shuang Qiu · Xiaohan Wei · Zhuoran Yang · Jayaraman Thiagarajan · Salman Asif · Michael Gillhofer · Johannes Brandstetter · Sepp Hochreiter · Felix Petersen · Dhruv Patel · Assad Oberai · Akshay Kamath · Sushrut Karmalkar · Eric Price · Ali Ahmed · Zahra Kadkhodaie · Sreyas Mohan · Eero Simoncelli · Carlos Fernandez-Granda · Oscar Leong · Wesam Sakla · Rebecca Willett · Stephan Hoyer · Jascha Sohl-Dickstein · Sam Greydanus · Gauri Jagatap · Chinmay Hegde · Michael Kellman · Jonathan Tamir · Nouamane Laanait · Ousmane Dia · Mirco Ravanelli · Jonathan Binas · Negar Rostamzadeh · Shirin Jalali · Tiantian Fang · Alex Schwing · Sébastien Lachapelle · Philippe Brouillard · Tristan Deleu · Simon Lacoste-Julien · Stella Yu · Arya Mazumdar · Ankit Singh Rawat · Yue Zhao · Jianshu Chen · Xiaoyang Li · Hubert Ramsauer · Gabrio Rizzuti · Nikolaos Mitsakos · Dingzhou Cao · Thomas Strohmer · Yang Li · Pei Peng · Gregory Ongie -
2019 Poster: Adaptive Cross-Modal Few-shot Learning »
Chen Xing · Negar Rostamzadeh · Boris Oreshkin · Pedro O. Pinheiro -
2019 Poster: Variational Temporal Abstraction »
Taesup Kim · Sungjin Ahn · Yoshua Bengio -
2019 Poster: On Adversarial Mixup Resynthesis »
Christopher Beckham · Sina Honari · Alex Lamb · Vikas Verma · Farnoosh Ghadiri · R Devon Hjelm · Yoshua Bengio · Chris Pal -
2019 Poster: Sequential Neural Processes »
Gautam Singh · Jaesik Yoon · Youngsung Son · Sungjin Ahn -
2019 Spotlight: Sequential Neural Processes »
Gautam Singh · Jaesik Yoon · Youngsung Son · Sungjin Ahn -
2018 : Coffee Break and Poster Session I »
Pim de Haan · Bin Wang · Dequan Wang · Aadil Hayat · Ibrahim Sobh · Muhammad Asif Rana · Thibault Buhet · Nicholas Rhinehart · Arjun Sharma · Alex Bewley · Michael Kelly · Lionel Blondé · Ozgur S. Oguz · Vaibhav Viswanathan · Jeroen Vanbaar · Konrad Żołna · Negar Rostamzadeh · Rowan McAllister · Sanjay Thakur · Alexandros Kalousis · Chelsea Sidrane · Sujoy Paul · Daphne Chen · Michal Garmulewicz · Henryk Michalewski · Coline Devin · Hongyu Ren · Jiaming Song · Wen Sun · Hanzhang Hu · Wulong Liu · Emilie Wirbel -
2018 Poster: Towards Deep Conversational Recommendations »
Raymond Li · Samira Ebrahimi Kahou · Hannes Schulz · Vincent Michalski · Laurent Charlin · Chris Pal -
2018 Poster: Unsupervised Depth Estimation, 3D Face Rotation and Replacement »
Joel Ruben Antony Moniz · Christopher Beckham · Simon Rajotte · Sina Honari · Chris Pal -
2018 Poster: Bayesian Model-Agnostic Meta-Learning »
Jaesik Yoon · Taesup Kim · Ousmane Dia · Sungwoong Kim · Yoshua Bengio · Sungjin Ahn -
2018 Poster: Sparse Attentive Backtracking: Temporal Credit Assignment Through Reminding »
Nan Rosemary Ke · Anirudh Goyal · Olexa Bilaniuk · Jonathan Binas · Michael Mozer · Chris Pal · Yoshua Bengio -
2018 Spotlight: Sparse Attentive Backtracking: Temporal Credit Assignment Through Reminding »
Nan Rosemary Ke · Anirudh Goyal · Olexa Bilaniuk · Jonathan Binas · Michael Mozer · Chris Pal · Yoshua Bengio -
2018 Spotlight: Bayesian Model-Agnostic Meta-Learning »
Jaesik Yoon · Taesup Kim · Ousmane Dia · Sungwoong Kim · Yoshua Bengio · Sungjin Ahn -
2018 Poster: Towards Text Generation with Adversarially Learned Neural Outlines »
Sandeep Subramanian · Sai Rajeswar Mudumba · Alessandro Sordoni · Adam Trischler · Aaron Courville · Chris Pal -
2017 Poster: ExtremeWeather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events »
Evan Racah · Christopher Beckham · Tegan Maharaj · Samira Ebrahimi Kahou · Mr. Prabhat · Chris Pal -
2015 Poster: Learning to Segment Object Candidates »
Pedro O. Pinheiro · Ronan Collobert · Piotr Dollar -
2015 Spotlight: Learning to Segment Object Candidates »
Pedro O. Pinheiro · Ronan Collobert · Piotr Dollar