Rethinking Transfer in Continual Learning: A Replay-Based Realisation

Continual learning studies how deployed language models can continually acquire new tasks without expensive retraining from scratch. Existing methods have largely focused on preventing forgetting, leaving a complementary question underexplored: when can past experience actually help a new task? We identify three measurable conditions for realised forward transfer: the target must have remaining headroom, transferred information must persist through continued optimisation, and replay must come from compatible prior tasks. These conditions motivate Transfer Selective Replay (TSR), which uses a training-free gradient signature to route replay toward sources predicted to help the incoming task, while distillation maintains stability. Across heterogeneous and homogeneous continual-learning streams, model scales, and backbone families, TSR improves forward transfer and final performance without sacrificing retention. More broadly, our results suggest that continual learning should treat transfer alongside forgetting as a first-class design objective.