Where Should Deletion Live? Decision-Exact Deletion and Representation-Level Residuals under Interleaved Continual Learning

Continually adapting agents that interact with people must not only acquire new identities, objects, and skills, but also reliably revoke them when requested. Existing machine-unlearning methods are typically evaluated in a one-shot setting, where learning stops immediately after deletion. We study this more realistic interleaved continual learning–unlearning (CLU) regime, in which learning and deletion requests arrive throughout deployment. Using a vision-based perception and enrollment stack consisting of a frozen ViT-B/16 backbone, a shared LoRA adapter, replay, and an analytic classifier head, we investigate where deletion should be implemented. Our results show that deletion placement can be more important than the choice of gradient-based unlearning operator. Retiring the deleted class’s classifier row already matches the strongest gradient baseline while reducing deletion latency from tens of seconds to milliseconds. We further introduce LEDGER, which re-fits the classifier head in closed form using statistics from retained classes. On CASIA-Face100, LEDGER improves retained-class accuracy by 7.5 percentage points over row retirement and matches a retrain-without-the-deleted-class oracle at a fraction of its cost. However, decision-level deletion does not imply removal of information from the learned representation. An oracle-anchored recovery probe shows that the deleted identity becomes increasingly recoverable as subsequent learning proceeds, despite zero deployed reversal. These findings suggest a practical design principle for continually learning agents: implement operational deletion at the decision layer, while treating representation-level deletion as a separately audited and periodically re-evaluated property.