Timezone: »
We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal, we split a deep neural network into a stack of gradient-isolated modules. Each module is trained to maximally preserve the information of its inputs using the InfoNCE bound from Oord et al [2018]. Despite this greedy training, we demonstrate that each module improves upon the output of its predecessor, and that the representations created by the top module yield highly competitive results on downstream classification tasks in the audio and visual domain. The proposal enables optimizing modules asynchronously, allowing large-scale distributed training of very deep neural networks on unlabelled datasets.
Author Information
Sindy Löwe (University of Amsterdam)
Peter O'Connor (Brain Corporation)
Bas Veeling (AMLab - University of Amsterdam)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: Putting An End to End-to-End: Gradient-Isolated Learning of Representations »
Fri. Dec 13th 01:00 -- 03:00 AM Room East Exhibition Hall B + C #66
More from the Same Authors
-
2021 : Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data »
Sindy Löwe · David Madras · Richard Zemel · Max Welling -
2023 Poster: Rotating Features for Object Discovery »
Sindy Löwe · Phillip Lippe · Francesco Locatello · Max Welling -
2023 Poster: PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers »
Phillip Lippe · Bas Veeling · Paris Perdikaris · Richard Turner · Johannes Brandstetter -
2020 : Panel discussion 1 »
Bas Veeling · Olivier Teytaud · Karl Friston · Sindy Löwe · Mateusz Malinowski -
2020 : Introduction: Olivier Teytaud »
Sindy Löwe -
2020 : Invited Talk Bastiaan Veeling »
Bas Veeling -
2020 Workshop: Beyond BackPropagation: Novel Ideas for Training Neural Architectures »
Mateusz Malinowski · Grzegorz Swirszcz · Viorica Patraucean · Marco Gori · Yanping Huang · Sindy Löwe · Anna Choromanska -
2020 : Live Intro »
Mateusz Malinowski · Viorica Patraucean · Grzegorz Swirszcz · Sindy Löwe · Anna Choromanska · Marco Gori · Yanping Huang -
2016 : Deep Spiking Networks »
Peter O'Connor