Synaptic Strength Controls Trainability and Structural Stability in Rank-Deficient RNNs

Real-world networks, from biological brains to ecological systems, are typically low-rank, yet they continue to learn throughout their existence. Understanding how learning operates in this rank-deficient regime is essential, but existing theory captures only its limits. Classical results describe early training in random high-dimensional networks, while low-rank theory describes its structured endpoint. To bridge these regimes, we introduce rank-deficient RNNs, \textit{i.e.}, networks initialized at low rank with fully trainable weights, in which rank and synaptic strength vary independently. This decoupling reveals that synaptic strength, not rank, primarily governs the learning regime, and that its strong and weak limits confer opposing advantages. Strong synapses produce rich nonlinear dynamics at initialization, enabling rapid learning that matches full-rank training speed at low ranks and triples it in gated architectures (GRUs, LSTMs). Weak synapses, by contrast, distribute computation across the population so that no individual connection is critical, yielding solutions that are structurally stable to neuron loss. Empirically, the weight changes induced by training consistently follow weak synaptic scaling across four tasks, four architectures, and distinct initializations. Overall, our findings identify synaptic strength as a central variable controlling both trainability and structural stability in rank-deficient RNNs.