Time-invariant structure learning via bootstrapping-free retrospective learning

Inspired by recent experimental observations of nontrivial behavioral and dopaminergic learning dynamics during classical conditioning, we introduce a normative principle of time-invariant structure learning. According to the principle, online estimators of environmental relationships should evolve along a common expected learning curve, so that ratios of all estimates stay approximately constant throughout the learning process. We formalize this idea in the framework of successor representation learning and prove that the standard temporal-difference learning algorithm TD($\lambda$) with $\lambda<1$ is unable to adhere to the principle due to bootstrapping. In contrast, bootstrapping-free algorithms that rely on retrospective learning can naturally achieve time-invariant structure learning if the learning rate is adapted based on the frequency of state visitation. Our numerical experiments indicate that methods that adhere to the principle typically lead to lower transient estimation errors compared to the standard bootstrapping methods. We also show that that retrospective learning reproduces spacing effects observed in animals undergoing classical conditioning.