Timezone: »

 
Poster
The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
Basri Ronen · David Jacobs · Yoni Kasten · Shira Kritchman

Wed Dec 11 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #236
in Theory -- Regularization »

We study the relationship between the frequency of a function and the speed at which a neural network learns it. We build on recent results that show that the dynamics of overparameterized neural networks trained with gradient descent can be well approximated by a linear system. When normalized training data is uniformly distributed on a hypersphere, the eigenfunctions of this linear system are spherical harmonic functions. We derive the corresponding eigenvalues for each frequency after introducing a bias term in the model. This bias term had been omitted from the linear network model without significantly affecting previous theoretical results. However, we show theoretically and experimentally that a shallow neural network without bias cannot represent or learn simple, low frequency functions with odd frequencies. Our results lead to specific predictions of the time it will take a network to learn functions of varying frequency. These predictions match the empirical behavior of both shallow and deep networks.

Keywords: Deep Learning · Regularization · Theory; Theory
[ Paper]  [ Slides] 

Author Information

Basri Ronen (Weizmann Inst.)
David Jacobs (University of Maryland, USA)
Yoni Kasten (Weizmann Institute)
Shira Kritchman (Weizmann Institute)

More from the Same Authors