Halting in Recurrent Sequence Models

Recurrent sequence models, such as recurrent neural networks (RNNs) and state space models (SSMs), are trained to emit an end-of-sequence (EOS) token when their output sequence is complete. During training, EOS contributes to the loss in the same way as the content tokens, without a separate target that explains how the model should decide when to stop. Although this training paradigm is used ubiquitously, we do not understand how the model forms the decision to output EOS in the first place. In this study, we consider a minimal version of the EOS halting problem, isolating it with a copy task in which the model emits a given input sequence $K$ times, followed by EOS. Since two examples can store the same sequence and differ only in $K$, the difference between their hidden trajectories isolates the effect of the stopping instruction. In an RNN with tanh activation, the recurrent matrix mixes this difference, tanh scales each coordinate according to the two current states, and the resulting hidden states form approximate families indexed by sequence position and the number of copies remaining. One complete copy moves the state to the family with one fewer copy remaining, and an affine state edit learned on separate sequences moves the RNN between adjacent families while preserving the copied symbols, reproducing the target sequence correctly in $99.8\%$ of cases. In a linear SSM, the trajectory difference separates across two-dimensional rotation blocks whose contributions to the EOS comparisons cancel at intermediate positions and reinforce near the stopping time, and independently rotating the response within these planes disrupts stopping and preserves the copied content.