Adjoint Method: The Connection between Analog-based Equilibrium Propagation Architectures and Neural ODEs

Analog neural networks (ANNs) hold significant potential for substantialreductions in power consumption in modern neural networks, particularly whenemploying the increasingly popular Energy-Based Models (EBMs) in tandem withthe local Equilibrium Propagation (EP) training algorithm. This paper analyzesthe relationship between this family of ANNs and the concept of Neural OrdinaryDifferential Equations (Neural ODEs). Using the adjoint method, we formallydemonstrate that ANN-EP can be derived from Neural ODEs by constraining thedifferential equations to those with a steady-state response. This findingopens avenues for the ANN-EP community to extend ANNs to non-steady-statescenarios. Additionally, it provides an efficient setting for NN-ODEs thatsignificantly reduces the training cost.