Stochastic Dynamics of Long-Horizon Generative Models

Long-horizon generation in autoregressive and iterative generative models can exhibit repetition, reduced contextual sensitivity, and loss of diversity. We study these behaviors from a dynamical perspective by analyzing how separation between generation trajectories evolves in representation space. Using an approximate-contraction condition, in which expected pairwise distance contracts by a factor $\lambda$ up to an additive stochastic scale $\epsilon$, we define an effective relaxation timescale $\tau=1/(1-\lambda)$ and long-horizon separation scale $r=\epsilon/(1-\lambda)$. We evaluate this framework across long-horizon autoregressive language generation and iterative image outpainting. Across models and generation settings, pairwise representation distances initially decrease and subsequently fluctuate around a nonzero long-horizon scale. In language generation, the effective parameters vary systematically with decoding temperature and model architecture: higher temperatures generally produce larger stochastic scales and, in several models, stronger fitted contraction. In image outpainting, $\lambda$ and $\epsilon$ vary substantially with generation model and extension length, while the resulting separation scale remains comparatively stable, with values near $1.3$ across most configurations. Because representations are unit-normalized, these separation scales correspond to mean pairwise cosine similarities and partly reflect static representation geometry such as anisotropy. Noise-reduced Lyapunov-like diagnostics provide a complementary measure of local trajectory sensitivity. The framework provides an empirical description of long-horizon representation dynamics across generative systems, without requiring strict attractors or implying that representation-space contraction causes output-level degeneration.