PoFormer: Recurrent Pointer Addressing for Length Generalization

Length generalization requires a model trained on short sequences to apply the same algorithm to longer inputs, enabling compact, reusable computational subcircuits rather than length-specific behaviors. In this paper, we introduce POFORMER, which augments recurrent content state with explicit pointers over static input memory. Pointer attention uses the active pointer as the RoPE query position and updates it to the attended physical address, supporting learned relative shifts and content-dependent jumps. We formalize these operations with P-RASP, a recurrent pointer extension of RASP, and establish a finite-horizon simulation result. Across algorithmic tasks, POFORMER substantially improves out-of-distribution length generalization over standard and looped Transformer baselines. Its attention maps preserve the predicted traversal geometry beyond training lengths and expose address drift when generalization fails. These results identify explicit address state as an inductive bias linking recurrent expressivity to length-generalizing behavior.