The Long Delay to Arithmetic Generalization

Grokking in transformers trained on algorithmic tasks shows a long delay between fitting the training set and abrupt generalization, and the cause of that delay is not fully understood. We find a consistent gap between when a model learns the structure a task needs and when it can use it. The delay reflects limited access to already-learned structure rather than a failure to acquire it. The gap holds across encoder–decoder and decoder-only transformers, across tasks from one-step Collatz to multi-step iterates and transfer to GCD, and across scales from 30M to 1.4B parameters. It is sharpest at scale. In a Pythia-1.4B fine-tune, arithmetic features are linearly readable at the pretrained checkpoint while sequence-level accuracy is exactly zero. In one-step Collatz prediction, the encoder organizes parity and residue structure within the first few thousand steps while output accuracy stays near chance for tens of thousands more. Two causal experiments localize the bottleneck to the decoder readout. Across six seeds, transplanting a trained encoder into a fresh model doubles overall accuracy and quadruples odd-branch accuracy over joint training from scratch (mean 0.751 vs. 0.365 overall and 0.524 vs. 0.119 on the odd branch, with non-overlapping 95\% intervals), while transplanting a trained decoder reduces accuracy. Freezing a converged encoder and retraining only the decoder removes the plateau and reaches 97.6\% accuracy, against 86.1\% for joint training. The readout is not equally hard for every input. Across 15 bases, those whose factorization aligns with the Collatz map reach 99.8\% accuracy, while binary fails and its representations collapse without recovering. A model that has not yet generalized may already hold the structure the task needs. What remains is reading it out, and how hard that is depends on how the input is represented.