Memorisation and generalisation in auto-regressive generative models

The transition from memorisation to generalisation in generative models has attracted considerable interest. In diffusion models, a particularly insightful approach defines generalisation as the convergence of independently trained models to the same density. Specifically, two generative models trained independently on disjoint subsets of a data set are said to generalise if they converge to the same model. Extending this idea to autoregressive transformers is challenging: unlike diffusion models, transformers lack a shared latent variable that naturally couples generation and discrete tokens lack an obvious notion of overlap. Here, we show a transition from memorisation to generalisation in autoregressive transformers as a function of sample complexity by introducing the survival probability: the probability that two transformers generate the same sequence under maximal coupling. We further develop a theoretical analysis of convergence for first-order Markov chains. Our framework provides a principled route to studying generalisation through convergence across independently trained autoregressive generative models, and opens up several interesting theoretical questions.