The two clocks and the innovation window: When and how generative models learn rules

Generative models trained on finite data face a fundamental tension: their score-matching or next-token objective converges to the empirical training distribution rather than the population distribution we seek to learn. Using rule-valid synthetic tasks, we trace this tension across two training timescales: $\tau_{rule}$, the step at which generations first become rule-valid, and $\tau_{mem}$, the step at which models begin reproducing training samples. Focusing on parity and extending to other binary rules and combinatorial puzzles, we characterize how these two clocks $\tau_{rule}$, $\tau_{mem}$ depend on key aspects of the learning setup. Specifically, we show that $\tau_{rule}$ increases with rule complexity and decreases with model capacity, while $\tau_{mem}$ is approximately invariant to the rule and scales nearly linearly with dataset size $N$. We define the \emph{innovation window} as the interval $[\tau_{rule}, \tau_{mem}]$. This window widens with increasing $N$ and narrows with rule complexity, and may vanish entirely when $\tau_{rule} \geq \tau_{mem}$. The same two-clock structure arises in both diffusion (DiT) and autoregressive (GPT) models, with architecture-dependent offsets. Dissecting the learned score of DiT models reveals a corresponding evolution of the optimization landscapes, where attractors emerge at both timescales: rule-valid samples' basins expand substantially around $\tau_{rule}$, while training samples' basins begin to dominate around $\tau_{mem}$. Together, these results yield a unified and predictive account of when and how generative models exhibit genuine innovation.