Timezone: »

Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillations
Axel Laborieux · Friedemann Zenke

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #117
Equilibrium propagation (EP) is an alternative to backpropagation (BP) that allows the training of deep neural networks with local learning rules. It thus provides a compelling framework for training neuromorphic systems and understanding learning in neurobiology. However, EP requires infinitesimal teaching signals, thereby limiting its applicability to noisy physical systems. Moreover, the algorithm requires separate temporal phases and has not been applied to large-scale problems. Here we address these issues by extending EP to holomorphic networks. We show analytically that this extension naturally leads to exact gradients for finite-amplitude teaching signals. Importantly, the gradient can be computed as the first Fourier coefficient from finite neuronal activity oscillations in continuous time without requiring separate phases. Further, we demonstrate in numerical simulations that our approach permits robust estimation of gradients in the presence of noise and that deeper models benefit from the finite teaching signals. Finally, we establish the first benchmark for EP on the ImageNet $32 \times 32$ dataset and show that it matches the performance of an equivalent network trained with BP. Our work provides analytical insights that enable scaling EP to large-scale problems and establishes a formal framework for how oscillations could support learning in biological and neuromorphic systems.

Author Information

Axel Laborieux (Friedrich Miescher Institute for Biomedical Research)
Axel Laborieux

I am a Postdoctoral researcher at the Friedrich Miescher Institute in Computational Neuroscience in the team of Friedemann Zenke. Previously a PhD candidate in hardware-oriented AI at the Center for Nanoscience and Nanotechnology, attached to Paris-Saclay University.

Friedemann Zenke (Friedrich Miescher Institute)

More from the Same Authors