Timezone: »
We introduce a variational approach to learning and inference of temporally hierarchical structure and representation for sequential data. We propose the Variational Temporal Abstraction (VTA), a hierarchical recurrent state space model that can infer the latent temporal structure and thus perform the stochastic state transition hierarchically. We also propose to apply this model to implement the jumpy imagination ability in imagination-augmented agent-learning in order to improve the efficiency of the imagination. In experiments, we demonstrate that our proposed method can model 2D and 3D visual sequence datasets with interpretable temporal structure discovery and that its application to jumpy imagination enables more efficient agent-learning in a 3D navigation task.
Author Information
Taesup Kim (Mila, Université de Montréal / Kakao Brain)
Sungjin Ahn (Rutgers University)
Yoshua Bengio (Mila - University of Montreal)
More from the Same Authors
-
2021 : DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations »
Fei Deng · Ingook Jang · Sungjin Ahn -
2021 : TransDreamer: Reinforcement Learning with Transformer World Models »
· Jaesik Yoon · Yi-Fu Wu · Sungjin Ahn -
2021 : Learning Representations for Zero-Shot Image Generation without Text »
Gautam Singh · Fei Deng · Sungjin Ahn -
2023 Poster: Object-Centric Slot Diffusion »
Jindong Jiang · Fei Deng · Gautam Singh · Sungjin Ahn -
2023 Poster: Facing-off World Model Backbones: RNN, Transformer, and S4 »
Fei Deng · Junyeong Park · Sungjin Ahn -
2022 Poster: Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos »
Gautam Singh · Yi-Fu Wu · Sungjin Ahn -
2020 : Invited Talk: Sungjin Ahn »
Sungjin Ahn -
2020 Poster: Generative Neurosymbolic Machines »
Jindong Jiang · Sungjin Ahn -
2020 Spotlight: Generative Neurosymbolic Machines »
Jindong Jiang · Sungjin Ahn -
2019 : Climate Change: A Grand Challenge for ML »
Yoshua Bengio · Carla Gomes · Andrew Ng · Jeff Dean · Lester Mackey -
2019 Poster: How to Initialize your Network? Robust Initialization for WeightNorm & ResNets »
Devansh Arpit · Víctor Campos · Yoshua Bengio -
2019 Poster: Neural Multisensory Scene Inference »
Jae Hyun Lim · Pedro O. Pinheiro · Negar Rostamzadeh · Chris Pal · Sungjin Ahn -
2019 Poster: Fast AutoAugment »
Sungbin Lim · Ildoo Kim · Taesup Kim · Chiheon Kim · Sungwoong Kim -
2019 Poster: Sequential Neural Processes »
Gautam Singh · Jaesik Yoon · Youngsung Son · Sungjin Ahn -
2019 Spotlight: Sequential Neural Processes »
Gautam Singh · Jaesik Yoon · Youngsung Son · Sungjin Ahn -
2018 : Opening remarks »
Yoshua Bengio -
2018 Poster: Image-to-image translation for cross-domain disentanglement »
Abel Gonzalez-Garcia · Joost van de Weijer · Yoshua Bengio -
2018 Poster: MetaGAN: An Adversarial Approach to Few-Shot Learning »
Ruixiang ZHANG · Tong Che · Zoubin Ghahramani · Yoshua Bengio · Yangqiu Song -
2018 Poster: Bayesian Model-Agnostic Meta-Learning »
Jaesik Yoon · Taesup Kim · Ousmane Dia · Sungwoong Kim · Yoshua Bengio · Sungjin Ahn -
2018 Poster: Sparse Attentive Backtracking: Temporal Credit Assignment Through Reminding »
Nan Rosemary Ke · Anirudh Goyal · Olexa Bilaniuk · Jonathan Binas · Michael Mozer · Chris Pal · Yoshua Bengio -
2018 Spotlight: Sparse Attentive Backtracking: Temporal Credit Assignment Through Reminding »
Nan Rosemary Ke · Anirudh Goyal · Olexa Bilaniuk · Jonathan Binas · Michael Mozer · Chris Pal · Yoshua Bengio -
2018 Spotlight: Bayesian Model-Agnostic Meta-Learning »
Jaesik Yoon · Taesup Kim · Ousmane Dia · Sungwoong Kim · Yoshua Bengio · Sungjin Ahn -
2018 Poster: Dendritic cortical microcircuits approximate the backpropagation algorithm »
João Sacramento · Rui Ponte Costa · Yoshua Bengio · Walter Senn -
2018 Oral: Dendritic cortical microcircuits approximate the backpropagation algorithm »
João Sacramento · Rui Ponte Costa · Yoshua Bengio · Walter Senn -
2017 : Yoshua Bengio »
Yoshua Bengio -
2017 : More Steps towards Biologically Plausible Backprop »
Yoshua Bengio -
2017 : A3T: Adversarially Augmented Adversarial Training »
Aristide Baratin · Simon Lacoste-Julien · Yoshua Bengio · Akram Erraqabi -
2017 : Competition III: The Conversational Intelligence Challenge »
Mikhail Burtsev · Ryan Lowe · Iulian Vlad Serban · Yoshua Bengio · Alexander Rudnicky · Alan W Black · Shrimai Prabhumoye · Artem Rodichev · Nikita Smetanin · Denis Fedorenko · CheongAn Lee · EUNMI HONG · Hwaran Lee · Geonmin Kim · Nicolas Gontier · Atsushi Saito · Andrey Gershfeld · Artem Burachenok -
2017 Poster: Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net »
Anirudh Goyal · Nan Rosemary Ke · Surya Ganguli · Yoshua Bengio -
2017 Demonstration: A Deep Reinforcement Learning Chatbot »
Iulian Vlad Serban · Chinnadhurai Sankar · Mathieu Germain · Saizheng Zhang · Zhouhan Lin · Sandeep Subramanian · Taesup Kim · Michael Pieper · Sarath Chandar · Nan Rosemary Ke · Sai Rajeswar Mudumba · Alexandre de Brébisson · Jose Sotelo · Dendi A Suhubdy · Vincent Michalski · Joelle Pineau · Yoshua Bengio -
2017 Poster: GibbsNet: Iterative Adversarial Inference for Deep Graphical Models »
Alex Lamb · R Devon Hjelm · Yaroslav Ganin · Joseph Paul Cohen · Aaron Courville · Yoshua Bengio -
2017 Poster: Plan, Attend, Generate: Planning for Sequence-to-Sequence Models »
Caglar Gulcehre · Francis Dutil · Adam Trischler · Yoshua Bengio -
2017 Poster: Z-Forcing: Training Stochastic Recurrent Networks »
Anirudh Goyal · Alessandro Sordoni · Marc-Alexandre Côté · Nan Rosemary Ke · Yoshua Bengio