Improving Deep Learning Speed and Performance through Synaptic Neural Balance

We present experiments and their corresponding theory, demonstrating that synaptic neural balancing can significantly enhance deep learning speed, accuracy, and generalization, particularly on non-traditional compute paradigms. Given an additive cost function (regularizer) of the synaptic weights, a neuron is in balance if the total cost of its incoming weights equals that of its outgoing weights. For various networks, activation functions, and regularizers, neurons can be balanced using scaling operations without altering their functionality, associated with a strictly convex optimization problem. In our simulations, we systematically observe that: (1) Fully balancing before training results in better performance as compared to several other training approaches; (2) Interleaving partial (layer-wise) balancing and stochastic gradient descent steps during training results in faster learning convergence and better overall accuracy (with $L_1 $ balancing converging faster than $L_2$ balancing; and (3) When given limited training data, neural balanced models outperform plain or regularized models. and this is true both for both feedforward and recurrent networks. These balancing operations are entirely local, making them viable for biological or neuromorphic systems. This positions synaptic neural balancing as a promising approach for leveraging the unique characteristics of emerging AI accelerators, advancing the efficiency and sustainability of machine learning.