SARAM: Sparse Adapter Allocation with Reuse and Merging for Lifelong Robot Learning

Lifelong robot learning requires a robot to acquire new tasks sequentially without forgetting previously learned ones. Adapter-based methods have recently addressed this problem effectively. They train task-specific adapters and freeze them, so learning a new task cannot overwrite past parameters. A selection module then infers which task's adapters to activate at deployment. However, their cost grows with the task stream. Per-task adapters and selection modules accumulate on disk, stay resident beside the backbone at deployment, and raise GPU memory with the number of learned tasks against the fixed budget of onboard hardware. To address this, we propose SARAM, an adapter-based continual learning method that maintains performance while keeping storage growth and deployment cost low. Specifically, each new task first fits combination coefficients over the frozen adapters of past tasks. A sparse allocation then adds new adapters at only a few layers, and a duplication penalty discourages them from re-expressing what reuse already provides. After each task is learned, its training demonstrations are passed once through the frozen backbone to build a compact task signature from activation statistics. At deployment, this signature selects the task once per episode, and the selected update is merged into the backbone weights before execution. On four LIBERO suites, SARAM matches or exceeds baselines while adding only a small amount of state per task, and keeps GPU memory and control frequency at the backbone's level regardless of task count.