Timezone: »
Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial neural networks might benefit from variables that change slower than activities but much faster than the standard weights. These ``fast weights'' can be used to store temporary memories of the recent past and they provide a neurally plausible way of implementing the type of attention to the past that has recently proven helpful in sequence-to-sequence models. By using fast weights we can avoid the need to store copies of neural activity patterns.
Author Information
Jimmy Ba (University of Toronto)
Geoffrey E Hinton (Google)
Geoffrey Hinton received his PhD in Artificial Intelligence from Edinburgh in 1978 and spent five years as a faculty member at Carnegie-Mellon where he pioneered back-propagation, Boltzmann machines and distributed representations of words. In 1987 he became a fellow of the Canadian Institute for Advanced Research and moved to the University of Toronto. In 1998 he founded the Gatsby Computational Neuroscience Unit at University College London, returning to the University of Toronto in 2001. His group at the University of Toronto then used deep learning to change the way speech recognition and object recognition are done. He currently splits his time between the University of Toronto and Google. In 2010 he received the NSERC Herzberg Gold Medal, Canada's top award in Science and Engineering.
Volodymyr Mnih (DeepMind)
Joel Leibo (Google DeepMind)
Catalin Ionescu (Google)
More from the Same Authors
-
2021 Spotlight: Neural Additive Models: Interpretable Machine Learning with Neural Nets »
Rishabh Agarwal · Levi Melnick · Nicholas Frosst · Xuezhou Zhang · Ben Lengerich · Rich Caruana · Geoffrey Hinton -
2021 : Wasserstein Distance Maximizing Intrinsic Control »
Ishan Durugkar · Steven Hansen · Stephen Spencer · Volodymyr Mnih · Ishan Durugkar -
2021 : Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria »
Kavya Kopparapu · Edgar Dueñez-Guzman · Jayd Matyas · Alexander Vezhnevets · John Agapiou · Kevin McKee · Richard Everett · Janusz Marecki · Joel Leibo · Thore Graepel -
2022 : In-context Reinforcement Learning with Algorithm Distillation »
Michael Laskin · Luyu Wang · Junhyuk Oh · Emilio Parisotto · Stephen Spencer · Richie Steigerwald · DJ Strouse · Steven Hansen · Angelos Filos · Ethan Brooks · Maxime Gazeau · Himanshu Sahni · Satinder Singh · Volodymyr Mnih -
2022 : In-context Reinforcement Learning with Algorithm Distillation »
Michael Laskin · Luyu Wang · Junhyuk Oh · Emilio Parisotto · Stephen Spencer · Richie Steigerwald · DJ Strouse · Steven Hansen · Angelos Filos · Ethan Brooks · Maxime Gazeau · Himanshu Sahni · Satinder Singh · Volodymyr Mnih -
2022 Poster: Palm up: Playing in the Latent Manifold for Unsupervised Pretraining »
Hao Liu · Tom Zahavy · Volodymyr Mnih · Satinder Singh -
2022 Invited Talk: The Forward-Forward Algorithm for Training Deep Neural Networks »
Geoffrey Hinton -
2022 Poster: A Unified Sequence Interface for Vision Tasks »
Ting Chen · Saurabh Saxena · Lala Li · Tsung-Yi Lin · David Fleet · Geoffrey Hinton -
2021 Poster: Entropic Desired Dynamics for Intrinsic Control »
Steven Hansen · Guillaume Desjardins · Kate Baumli · David Warde-Farley · Nicolas Heess · Simon Osindero · Volodymyr Mnih -
2021 Poster: Canonical Capsules: Self-Supervised Capsules in Canonical Pose »
Weiwei Sun · Andrea Tagliasacchi · Boyang Deng · Sara Sabour · Soroosh Yazdani · Geoffrey Hinton · Kwang Moo Yi -
2021 Poster: Neural Additive Models: Interpretable Machine Learning with Neural Nets »
Rishabh Agarwal · Levi Melnick · Nicholas Frosst · Xuezhou Zhang · Ben Lengerich · Rich Caruana · Geoffrey Hinton -
2020 Poster: Big Self-Supervised Models are Strong Semi-Supervised Learners »
Ting Chen · Simon Kornblith · Kevin Swersky · Mohammad Norouzi · Geoffrey E Hinton -
2019 Poster: Lookahead Optimizer: k steps forward, 1 step back »
Michael Zhang · James Lucas · Jimmy Ba · Geoffrey E Hinton -
2019 Poster: Generalization of Reinforcement Learners with Working and Episodic Memory »
Meire Fortunato · Melissa Tan · Ryan Faulkner · Steven Hansen · Adrià Puigdomènech Badia · Gavin Buttimore · Charles Deck · Joel Leibo · Charles Blundell -
2019 Poster: Stacked Capsule Autoencoders »
Adam Kosiorek · Sara Sabour · Yee Whye Teh · Geoffrey E Hinton -
2019 Poster: When does label smoothing help? »
Rafael Müller · Simon Kornblith · Geoffrey E Hinton -
2019 Spotlight: When does label smoothing help? »
Rafael Müller · Simon Kornblith · Geoffrey E Hinton -
2019 Poster: Interval timing in deep reinforcement learning agents »
Ben Deverett · Ryan Faulkner · Meire Fortunato · Gregory Wayne · Joel Leibo -
2019 Poster: Unsupervised Learning of Object Keypoints for Perception and Control »
Tejas Kulkarni · Ankush Gupta · Catalin Ionescu · Sebastian Borgeaud · Malcolm Reynolds · Andrew Zisserman · Volodymyr Mnih -
2018 Poster: Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures »
Sergey Bartunov · Adam Santoro · Blake Richards · Luke Marris · Geoffrey E Hinton · Timothy Lillicrap -
2018 Poster: Inequity aversion improves cooperation in intertemporal social dilemmas »
Edward Hughes · Joel Leibo · Matthew Phillips · Karl Tuyls · Edgar Dueñez-Guzman · Antonio García Castañeda · Iain Dunning · Tina Zhu · Kevin McKee · Raphael Koster · Heather Roff · Thore Graepel -
2017 Poster: A multi-agent reinforcement learning model of common-pool resource appropriation »
Julien Pérolat · Joel Leibo · Vinicius Zambaldi · Charles Beattie · Karl Tuyls · Thore Graepel -
2017 Poster: Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation »
Yuhuai Wu · Elman Mansimov · Roger Grosse · Shun Liao · Jimmy Ba -
2017 Spotlight: Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation »
Yuhuai Wu · Elman Mansimov · Roger Grosse · Shun Liao · Jimmy Ba -
2017 Poster: Dynamic Routing Between Capsules »
Sara Sabour · Nicholas Frosst · Geoffrey E Hinton -
2017 Spotlight: Dynamic Routing Between Capsules »
Sara Sabour · Nicholas Frosst · Geoffrey E Hinton -
2016 Poster: Attend, Infer, Repeat: Fast Scene Understanding with Generative Models »
S. M. Ali Eslami · Nicolas Heess · Theophane Weber · Yuval Tassa · David Szepesvari · koray kavukcuoglu · Geoffrey E Hinton -
2016 Poster: Learning values across many orders of magnitude »
Hado van Hasselt · Arthur Guez · Arthur Guez · Matteo Hessel · Volodymyr Mnih · David Silver -
2016 Poster: Using Fast Weights to Attend to the Recent Past »
Jimmy Ba · Geoffrey E Hinton · Volodymyr Mnih · Joel Leibo · Catalin Ionescu -
2016 Poster: Strategic Attentive Writer for Learning Macro-Actions »
Alexander (Sasha) Vezhnevets · Volodymyr Mnih · Simon Osindero · Alex Graves · Oriol Vinyals · John Agapiou · koray kavukcuoglu -
2015 : The Deep Reinforcement Learning Boom »
Volodymyr Mnih -
2015 Poster: Grammar as a Foreign Language »
Oriol Vinyals · Łukasz Kaiser · Terry Koo · Slav Petrov · Ilya Sutskever · Geoffrey Hinton -
2015 Tutorial: Deep Learning »
Geoffrey E Hinton · Yoshua Bengio · Yann LeCun -
2014 Workshop: Deep Learning and Representation Learning »
Andrew Y Ng · Yoshua Bengio · Adam Coates · Roland Memisevic · Sharanyan Chetlur · Geoffrey E Hinton · Shamim Nemati · Bryan Catanzaro · Surya Ganguli · Herbert Jaeger · Phil Blunsom · Leon Bottou · Volodymyr Mnih · Chen-Yu Lee · Rich M Schwartz -
2014 Poster: Recurrent Models of Visual Attention »
Volodymyr Mnih · Nicolas Heess · Alex Graves · koray kavukcuoglu -
2014 Spotlight: Recurrent Models of Visual Attention »
Volodymyr Mnih · Nicolas Heess · Alex Graves · koray kavukcuoglu -
2013 Workshop: Deep Learning »
Yoshua Bengio · Hugo Larochelle · Russ Salakhutdinov · Tomas Mikolov · Matthew D Zeiler · David Mcallester · Nando de Freitas · Josh Tenenbaum · Jian Zhou · Volodymyr Mnih -
2012 Poster: ImageNet Classification with Deep Convolutional Neural Networks »
Alex Krizhevsky · Ilya Sutskever · Geoffrey E Hinton -
2012 Invited Talk: Dropout: A simple and effective way to improve neural networks »
Geoffrey E Hinton · George Dahl -
2012 Poster: A Better Way to Pre-Train Deep Boltzmann Machines »
Russ Salakhutdinov · Geoffrey E Hinton -
2012 Spotlight: ImageNet Classification with Deep Convolutional Neural Networks »
Alex Krizhevsky · Ilya Sutskever · Geoffrey E Hinton -
2010 Workshop: Deep Learning and Unsupervised Feature Learning »
Honglak Lee · Marc'Aurelio Ranzato · Yoshua Bengio · Geoffrey E Hinton · Yann LeCun · Andrew Y Ng -
2010 Talk: A Probabilistic Approach to Data Visualization »
Geoffrey E Hinton -
2010 Oral: Learning to combine foveal glimpses with a third-order Boltzmann machine »
Hugo Larochelle · Geoffrey E Hinton -
2010 Poster: Learning to combine foveal glimpses with a third-order Boltzmann machine »
Hugo Larochelle · Geoffrey E Hinton -
2010 Poster: Generating more realistic images using gated MRF's »
Marc'Aurelio Ranzato · Volodymyr Mnih · Geoffrey E Hinton -
2010 Poster: Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine »
George Dahl · Marc'Aurelio Ranzato · Abdel-rahman Mohamed · Geoffrey E Hinton -
2010 Poster: Gated Softmax Classification »
Roland Memisevic · Christopher Zach · Geoffrey E Hinton · Marc Pollefeys -
2009 Workshop: Deep Learning for Speech Recognition and Related Applications »
Li Deng · Dong Yu · Geoffrey E Hinton -
2009 Poster: Replicated Softmax: an Undirected Topic Model »
Russ Salakhutdinov · Geoffrey E Hinton -
2009 Poster: 3D Object Recognition with Deep Belief Nets »
Vinod Nair · Geoffrey E Hinton -
2009 Spotlight: 3D Object Recognition with Deep Belief Nets »
Vinod Nair · Geoffrey E Hinton -
2009 Invited Talk: Deep Learning with Multiplicative Interactions »
Geoffrey E Hinton -
2009 Poster: Zero-shot Learning with Semantic Output Codes »
Mark M Palatucci · Dean Pomerleau · Geoffrey E Hinton · Tom Mitchell -
2008 Poster: Using matrices to model symbolic relationship »
Ilya Sutskever · Geoffrey E Hinton -
2008 Demonstration: Visualizing NIPS Cooperations using Multiple Maps t-SNE »
Laurens van der Maaten · Geoffrey E Hinton -
2008 Spotlight: Using matrices to model symbolic relationship »
Ilya Sutskever · Geoffrey E Hinton -
2008 Poster: The Recurrent Temporal Restricted Boltzmann Machine »
Ilya Sutskever · Geoffrey E Hinton · Graham Taylor -
2008 Poster: A Scalable Hierarchical Distributed Language Model »
Andriy Mnih · Geoffrey E Hinton -
2008 Poster: Implicit Mixtures of Restricted Boltzmann Machines »
Vinod Nair · Geoffrey E Hinton -
2008 Poster: Competing RBM density models for classification of fMRI images »
Tanya Schmah · Geoffrey E Hinton · Richard Zemel -
2007 Tutorial: Deep Belief Nets »
Geoffrey E Hinton -
2007 Poster: Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes »
Russ Salakhutdinov · Geoffrey E Hinton -
2007 Poster: Modeling image patches with a directed hierarchy of Markov random fields »
Simon Osindero · Geoffrey E Hinton -
2006 Poster: Modeling Human Motion Using Binary Latent Variables »
Graham Taylor · Geoffrey E Hinton · Sam T Roweis -
2006 Spotlight: Modeling Human Motion Using Binary Latent Variables »
Graham Taylor · Geoffrey E Hinton · Sam T Roweis