Timezone: »

 
Poster
Local Signal Adaptivity: Provable Feature Learning in Neural Networks Beyond Kernels
Stefani Karp · Ezra Winston · Yuanzhi Li · Aarti Singh

Fri Dec 10 08:30 AM -- 10:00 AM (PST) @
in Poster Session 8 »

Neural networks have been shown to outperform kernel methods in practice (including neural tangent kernels). Most theoretical explanations of this performance gap focus on learning a complex hypothesis class; in some cases, it is unclear whether this hypothesis class captures realistic data. In this work, we propose a related, but alternative, explanation for this performance gap in the image classification setting, based on finding a sparse signal in the presence of noise. Specifically, we prove that, for a simple data distribution with sparse signal amidst high-variance noise, a simple convolutional neural network trained using stochastic gradient descent learns to threshold out the noise and find the signal. On the other hand, the corresponding neural tangent kernel, with a fixed set of predetermined features, is unable to adapt to the signal in this manner. We supplement our theoretical results by demonstrating this phenomenon empirically: in CIFAR-10 and MNIST images with various backgrounds, as the background noise increases in intensity, a CNN's performance stays relatively robust, whereas its corresponding neural tangent kernel sees a notable drop in performance. We therefore propose the "local signal adaptivity" (LSA) phenomenon as one explanation for the superiority of neural networks over kernel methods.

Keywords: Theory · Deep Learning · Optimization · Machine Learning · Vision · Kernel Methods

Author Information

Stefani Karp (Carnegie Mellon University/Google)
Ezra Winston (Carnegie Mellon University)
Yuanzhi Li (CMU)
Aarti Singh (CMU)

More from the Same Authors