Adjoint Method: The Connection Between Analog-Based Equilibrium Propagation Architectures and Neural ODEs

Abstract

Analog neural networks (ANNs) hold significant potential for substantial reductions in power consumption in modern neural networks, particularly when employing the increasingly popular Energy-Based Models (EBMs) in tandem with the local Equilibrium Propagation (EP) training algorithm. This paper analyzes the relationship between this family of ANNs and the concept of Neural Ordinary Differential Equations (Neural ODEs). Using the adjoint method, we formally demonstrate that ANN-EP can be derived from Neural ODEs by constraining the differential equations to those with a steady-state response. This finding opens avenues for the ANN-EP community to extend ANNs to non-steady-state scenarios. Additionally, it provides an efficient setting for NN-ODEs that significantly reduces the training cost.