Inside Emergence: Structure-Behaviour Gaps in Language Model Training

In densely checkpointed language model training, a capability can remain flat and then appear within a narrow window of steps. Behaviour alone cannot tell whether internal structure changes just as abruptly or instead lags earlier structural reorganisation. We study this relationship across 150 main Pythia 70M-410M trajectories (3 scales $\times$ 5 tasks $\times$ 10 seeds), using seed replication to estimate trajectory-level event order. Tracking the spectra of mean-ablation patching matrices throughout training, we find two task-determined dimensions. The discontinuity ratio $\rho$ separates tasks where behaviour changes more abruptly than the structural spectrum from tasks where the two co-evolve, reconciling competing accounts of emergence within this model range. The structure-behaviour gap $\tau$ shows that spectral completion precedes behavioural emergence in 94% of main trajectories, with median absolute leads of 2,000-12,200 training steps (11$\times$-206$\times$ in step ratio). Together these quantities support offline trajectory auditing: $\rho$ computed from 28% of training predicts final $\rho$ at $r = 0.93$, while $\tau$ identifies a pre-emergence checkpoint for circuit inspection; at those checkpoints, Pythia-410M causal ablations show that early-prominent components carry 86-99% of post-emergence capability across the five tasks tested causally.