Skip to yearly menu bar Skip to main content


Optimal Architectures in a Solvable Model of Deep Networks

Jonathan Kadmon · Haim Sompolinsky

Area 5+6+7+8 #18

Keywords: [ (Cognitive/Neuroscience) Neural Coding ] [ (Cognitive/Neuroscience) Theoretical Neuroscience ] [ Deep Learning or Neural Networks ]


Deep neural networks have received a considerable attention due to the success of their training for real world machine learning applications. They are also of great interest to the understanding of sensory processing in cortical sensory hierarchies. The purpose of this work is to advance our theoretical understanding of the computational benefits of these architectures. Using a simple model of clustered noisy inputs and a simple learning rule, we provide analytically derived recursion relations describing the propagation of the signals along the deep network. By analysis of these equations, and defining performance measures, we show that these model networks have optimal depths. We further explore the dependence of the optimal architecture on the system parameters.

Live content is unavailable. Log in and register to view live content