Beyond Gaussian Initialization: Activation-Induced Gibbs States for Critical and Width-Efficient Networks

Classical initialization typically searches for a suitable weight scale while keeping the preactivation family Gaussian. We ask whether the activation should also determine the shape of this distribution. For activations $\sigma=\nabla\Psi^* $, the same Legendre potential canonically induces the Gibbs family $\pi_{m,\tau}(dz)\propto\exp(-[\Psi^*(z)-mz]/\tau)dz$, whose score is directly prescribed by $\sigma-m$. We characterize criticality over arbitrary preactivation laws. For Softplus, no finite mean-zero Gaussian preactivation state in a zero-bias network is simultaneously forward-stationary and mean-square critical, whereas the activation-induced Gibbs family provably contains such a state. We then study finite-width fluctuations in an explicit decoupled propagation model: at equal criticality, they depend on $\rho(P)=\mathbb{E}_P[\sigma'(Z)^4]/\mathbb{E}_P[\sigma'(Z)^2]^2$. For sigmoid, the Gibbs critical state has a substantially smaller coefficient than its Gaussian counterpart, predicting a reduced width requirement; an explicit row-wise scale-mixture realization remains favorable after accounting for its additional heteroscedasticity. Preliminary MNIST experiments are consistent with the resulting optimization and width-efficiency predictions.