Skip to yearly menu bar Skip to main content


Transformers learn through gradual rank increase

Enric Boix-Adsera · Etai Littwin · Emmanuel Abbe · Samy Bengio · Joshua Susskind

Great Hall & Hall B1+B2 (level 1) #818
[ ]
Tue 12 Dec 8:45 a.m. PST — 10:45 a.m. PST


We identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank. We rigorously prove this occurs under the simplifying assumptions of diagonal weight matrices and small initialization. Our experiments support the theory and also show that phenomenon can occur in practice without the simplifying assumptions.

Chat is not available.