Timezone: »

 
Meta-Learning Backpropagation And Improving It
Louis Kirsch

In the past a large number of variable update rules have been proposed for meta learning such as fast weights, hyper networks, learned learning rules, and meta recurrent neural networks. We unify these architectures by demonstrating that a single weight-sharing and sparsity principle underlies them that can be used to express complex learning algorithms. We propose a simple implementation of this principle, the Variable Shared Meta RNN, and demonstrate that it allows implementing neuronal dynamics and backpropagation solely by running the recurrent neural network in forward-mode. This offers a direction for backpropagation that is biologically plausible. Then we show how backpropagation itself can be further improved through meta-learning. That is, we can use a human-engineered algorithm as an initialization for meta-learning better learning algorithms.

Author Information

Louis Kirsch (The Swiss AI Lab IDSIA)

More from the Same Authors

  • 2021 : Exploring through Random Curiosity with General Value Functions »
    Aditya Ramesh · Louis Kirsch · Sjoerd van Steenkiste · Jürgen Schmidhuber
  • 2021 : Introducing Symmetries to Black Box Meta Reinforcement Learning »
    Louis Kirsch · Sebastian Flennerhag · Hado van Hasselt · Abram Friesen · Junhyuk Oh · Yutian Chen
  • 2021 : Introducing Symmetries to Black Box Meta Reinforcement Learning »
    Louis Kirsch · Sebastian Flennerhag · Hado van Hasselt · Abram Friesen · Junhyuk Oh · Yutian Chen
  • 2022 : Meta-Learning General-Purpose Learning Algorithms with Transformers »
    Louis Kirsch · Luke Metz · James Harrison · Jascha Sohl-Dickstein
  • 2022 : Meta-Learning General-Purpose Learning Algorithms with Transformers »
    Louis Kirsch · Luke Metz · James Harrison · Jascha Sohl-Dickstein
  • 2022 : The Benefits of Model-Based Generalization in Reinforcement Learning »
    Kenny Young · Aditya Ramesh · Louis Kirsch · Jürgen Schmidhuber
  • 2022 Poster: Exploring through Random Curiosity with General Value Functions »
    Aditya Ramesh · Louis Kirsch · Sjoerd van Steenkiste · Jürgen Schmidhuber
  • 2021 Poster: Meta Learning Backpropagation And Improving It »
    Louis Kirsch · Jürgen Schmidhuber
  • 2020 : Q/A for invited talk #4 »
    Louis Kirsch
  • 2020 : General meta-learning »
    Louis Kirsch
  • 2019 : Poster session »
    Sebastian Farquhar · Erik Daxberger · Andreas Look · Matt Benatan · Ruiyi Zhang · Marton Havasi · Fredrik Gustafsson · James A Brofos · Nabeel Seedat · Micha Livne · Ivan Ustyuzhaninov · Adam Cobb · Felix D McGregor · Patrick McClure · Tim R. Davidson · Gaurush Hiranandani · Sanjeev Arora · Masha Itkina · Didrik Nielsen · William Harvey · Matias Valdenegro-Toro · Stefano Peluchetti · Riccardo Moriconi · Tianyu Cui · Vaclav Smidl · Taylan Cemgil · Jack Fitzsimons · He Zhao · · mariana vargas vieyra · Apratim Bhattacharyya · Rahul Sharma · Geoffroy Dubourg-Felonneau · Jonathan Warrell · Slava Voloshynovskiy · Mihaela Rosca · Jiaming Song · Andrew Ross · Homa Fashandi · Ruiqi Gao · Hooshmand Shokri Razaghi · Joshua Chang · Zhenzhong Xiao · Vanessa Boehm · Giorgio Giannone · Ranganath Krishnan · Joe Davison · Arsenii Ashukha · Jeremiah Liu · Sicong (Sheldon) Huang · Evgenii Nikishin · Sunho Park · Nilesh Ahuja · Mahesh Subedar · · Artyom Gadetsky · Jhosimar Arias Figueroa · Tim G. J. Rudner · Waseem Aslam · Adrián Csiszárik · John Moberg · Ali Hebbal · Kathrin Grosse · Pekka Marttinen · Bang An · Hlynur Jónsson · Samuel Kessler · Abhishek Kumar · Mikhail Figurnov · Omesh Tickoo · Steindor Saemundsson · Ari Heljakka · Dániel Varga · Niklas Heim · Simone Rossi · Max Laves · Waseem Gharbieh · Nicholas Roberts · Luis Armando Pérez Rey · Matthew Willetts · Prithvijit Chakrabarty · Sumedh Ghaisas · Carl Shneider · Wray Buntine · Kamil Adamczewski · Xavier Gitiaux · Suwen Lin · Hao Fu · Gunnar Rätsch · Aidan Gomez · Erik Bodin · Dinh Phung · Lennart Svensson · Juliano Tusi Amaral Laganá Pinto · Milad Alizadeh · Jianzhun Du · Kevin Murphy · Beatrix Benkő · Shashaank Vattikuti · Jonathan Gordon · Christopher Kanan · Sontje Ihler · Darin Graham · Michael Teng · Louis Kirsch · Tomas Pevny · Taras Holotyak
  • 2018 Poster: Modular Networks: Learning to Decompose Neural Computation »
    Louis Kirsch · Julius Kunze · David Barber