More Than Enough for the Present: Predictive Semantic Diversity for Continual Learning

Continual learning (CL) aims to enable machine learning models to sequentially acquire new knowledge while retaining previously learned capabilities. While numerous approaches have been developed to mitigate catastrophic forgetting, recent studies increasingly investigate CL from a representation-learning perspective. However, it remains unclear what information should be encoded in a representation to facilitate continual adaptation. In this work, we investigate whether encoding a broader set of predictive semantic factors benefits continual learning. Through controlled experiments on synthetic and natural-image datasets, we show that representations encoding broader predictive semantic information exhibit improved forward transfer and reduced forgetting under fully plastic sequential adaptation. Our findings suggest that predictive semantic diversity is a useful representation property for understanding and designing continually adaptable models.