Timezone: »
Poster
Asymptotic normality and confidence intervals for derivatives of 2layers neural network in the random features model
Yiwei Shen · Pierre C Bellec
This paper studies twolayers Neural Networks (NN), where the first layer contains random weights, and the second layer is trained using Ridge regularization. This model has been the focus of numerous recent works, showing that despite its simplicity, it captures some of the empirically observed behaviors of NN in the overparametrized regime, such as the doubledescent curve where the generalization error decreases as the number of weights increases to $+\infty$.
This paper establishes asymptotic distribution results for this 2layers NN model in the regime where the ratios $\frac p n$ and $\frac d n$ have finite limits, where $n$ is the sample size, $p$ the ambient dimension and $d$ is the width of the first layer. We show that a weighted average of the derivatives of the trained NN at the observed data is asymptotically normal, in a setting with Lipschitz activation functions in a linear regression response with Gaussian features under possibly nonlinear perturbations. We then leverage this asymptotic normality result to construct confidence intervals (CIs) for single components of the unknown regression vector.
The novelty of our results are threefold:
(1) Despite the nonlinearity induced by the activation function, we characterize the asymptotic distribution of a weighted average of the gradients of the network after training;
(2) It provides the first frequentist uncertainty quantification guarantees, in the form of valid ($1\text{}\alpha$)CIs, based on NN estimates;
(3) It shows that the doubledescent phenomenon occurs in terms of the length of the CIs, with the length increasing and then decreasing as $\frac d n\nearrow +\infty$ for certain fixed values of $\frac p n$. We also provide a toolbox to predict the length of CIs numerically, which lets us compare activation functions and other parameters in terms of CI length.
Author Information
Yiwei Shen (Rutgers University)
Pierre C Bellec (Rutgers)
More from the Same Authors

2019 Poster: First order expansion of convex regularized estimators »
Pierre C Bellec · Arun Kuchibhotla