Fixed Points and Scale-Dependent Reversal in a Random Dynamical System of Model Collapse
Lewis Mitchell
Abstract
Iterative fine-tuning on a language model's own synthetic output causes \emph{model collapse}: text diversity narrows across generations as rare patterns are progressively lost. Here we model this process as a discrete-time random dynamical system: the entropy rate of generated text at each generation is the state of an iterated random map, and a training-data filter is an intervention on that map. Fitting this recursion to six-generation QLoRA fine-tuning trajectories across three data-selection conditions, we show that a model-free entropy-rate filter does not merely slow the approach to a degenerate state, it relocates the map's fixed point to a higher-entropy regime ($x^* = 1.71$ vs. $1.15$ bits/word for unfiltered training, non-overlapping bootstrap confidence intervals). We further show that a between-generation cross-entropy filtering criterion fails for a precise, formalisable reason: it is a feedback controller whose reference signal collapses along with the process it is meant to correct, so its correlation with true entropy decorrelates within one generation and briefly inverts. Generalising discrete filtering into a continuous entropy-injection dose, we find no discrete threshold at either of two model scales tested; instead the dose-response is smooth, helping a 3B model while hurting an 8B one -- a reversal in \emph{sign} between scales that, in a single-domain check, persists under a genuinely real-text anchor and that directly-verified training-data bimodality offers a candidate capacity-dependent explanation for. Together these results connect fine-tuning collapse to the broader study of self-consuming generative systems and give a control-theoretic account of when and why model-free entropy interventions succeed or fail.
Chat is not available.
Successful Page Load