Spectral Asymptotics of Neural Network Jacobians: Convergency, Universality, and Phase Transition

We study the spectral properties of the Jacobian matrix in artificial neural networks and establish both its limiting spectral distribution and the second-order fluctuations of its spectrum. Our analysis uncovers fundamental phenomena, including universality properties with respect to the weight distribution and a phase transition in spectral behavior driven by the choice of activation function. Leveraging tools from random matrix theory, we first analyze a class of nonlinear covariance ensembles, which may be of independent interest in high-dimensional statistics, and subsequently characterize the spectral fluctuations of the Jacobian matrix, highlighting their dependence on weight initialization, layer widths, activation functions, and network depth. These findings provide finite-width distributional calibration for Jacobian-based stability diagnostics and quantify initialization-induced uncertainty in signal propagation, gradient stability, and local sensitivity of high-dimensional neural networks.