Why Do Recurrent Neural Networks Suddenly Learn? Bifurcation Mechanisms in Neuro-Inspired Short-Term Memory Tasks

Abstract

Recurrent neural networks (RNNs) are regularly studied as in silico models of biological and artificial computation. Training RNNs requires updating many synaptic weights, making the learning process complex and high-dimensional. In order to uncover learning mechanisms, we investigated the sudden accuracy jumps in RNNs' loss curves. Across several short-term memory tasks, we identified an initial search phase with accuracy plateaus, followed by rapid acquisition of skills. Studying attractor landscapes during learning revealed high-dimensional bifurcations as the links between these phases. Next, we introduced the temporal consistency regularization (TCR), a biologically plausible learning rule that incentivizes formation of memory-subserving attractors. In diverse short-term memory tasks, TCR accelerated (online) training, promoted robust attractors, and enabled networks initialized in a chaotic regime to train efficiently. Our analyses lead to testable predictions for system neuroscientists and highlight the need to study high-dimensional dynamical system theory to uncover learning mechanisms in biological and artificial networks.