Constructing Memory: Consolidation as Teacher-Student Training of a Generative Model

Human episodic memories are (re)constructed, share neural substrates with imagination, and show systematic biases that increase as they are consolidated into semantic memory. Here we suggest that these main features of human memory are characteristic of 'teacher-student' training of a neocortical generative model by a one-shot memory system in the hippocampal formation (HF). As we simulate with image datasets, the 'students’ (variational autoencoders) in association cortex develop a compressed 'latent variable’ representation of experience by learning to reconstruct replayed samples from the 'teacher' (a modern Hopfield network). Recall and imagination require these representations to be decoded into sensory experience by the HF and return projections to sensory cortex, whereas semantic memory and inference rely directly on the latent variables without requiring the HF.