Timezone: »
In this paper, we study the generalization performance of overparameterized 3-layer NTK models. We show that, for a specific set of ground-truth functions (which we refer to as the "learnable set"), the test error of the overfitted 3-layer NTK is upper bounded by an expression that decreases with the number of neurons of the two hidden layers. Different from 2-layer NTK where there exists only one hidden-layer, the 3-layer NTK involves interactions between two hidden-layers. Our upper bound reveals that, between the two hidden-layers, the test error descends faster with respect to the number of neurons in the second hidden-layer (the one closer to the output) than with respect to that in the first hidden-layer (the one closer to the input). We also show that the learnable set of 3-layer NTK without bias is no smaller than that of 2-layer NTK models with various choices of bias in the neurons. However, in terms of the actual generalization performance, our results suggest that 3-layer NTK is much less sensitive to the choices of bias than 2-layer NTK, especially when the input dimension is large.
Author Information
Peizhong Ju (The Ohio State University)
Xiaojun Lin (Purdue University)
Ness Shroff (The Ohio State University)
More from the Same Authors
-
2022 : Conditional Moment Alignment for Improved Generalization in Federated Learning »
Jayanth Reddy Regatti · Songtao Lu · Abhishek Gupta · Ness Shroff -
2022 Poster: Provably Efficient Model-Free Constrained RL with Linear Function Approximation »
Arnob Ghosh · Xingyu Zhou · Ness Shroff -
2021 Poster: Sample Complexity Bounds for Active Ranking from Multi-wise Comparisons »
Wenbo Ren · Jia Liu · Ness Shroff -
2020 Poster: Overfitting Can Be Harmless for Basis Pursuit, But Only to a Degree »
Peizhong Ju · Xiaojun Lin · Jia Liu -
2020 Spotlight: Overfitting Can Be Harmless for Basis Pursuit, But Only to a Degree »
Peizhong Ju · Xiaojun Lin · Jia Liu -
2019 Poster: On Sample Complexity Upper and Lower Bounds for Exact Ranking from Noisy Comparisons »
Wenbo Ren · Jia (Kevin) Liu · Ness Shroff -
2017 Poster: A New Alternating Direction Method for Linear Programming »
Sinong Wang · Ness Shroff