Invited Talk #3: Predictive Cognitive Maps with Multi-scale Successor Representations and Replay

Reinforcement Learning's principles of temporal difference learning can drive representation learning, even in the absence of rewards. Representation learning is especially important in problems that require a cognitive map (Tollman, 1947), common in mammalian spatial navigation and non-spatial inference, e.g., shortcut- and latent learning, policy revaluation, and remapping. Here I focus on models of predictive cognitive maps that learn successor representations (SR) at multiple scales, and use replay to update SR maps similar to Dyna models (SR-Dyna). SR- and SR-Dyna based representation learning capture biological representation learning reflected in place-, grid-, and distance to goal cell firing patterns (Stachenfled et al. 2017, Momennejad and Howard 2018), the interaction between boundary vector cells and place cells (De Cothi and Barry 2019), subgoal learning (Weinstein and Botvinick 2014), remapping, policy revaluation, and latent learning behavior (Momennejad et al. 2017; Russek, Momennejad et al. 2017). The SR framework makes testable predictions about representation learning in biological systems: e.g., about how predictive features are extracted from visual experience and abstracted into spatial representations that guide navigation. Specifically, the SR is sensitive to the policy the animal has taken during navigation - generating predictions about the representation of goals and how rewarding locations distort the predictive map. Finally, deep RL using SR has been shown to support option discovery, which is especially useful for empowering agents with intrinsic motivation in environments that have sparse rewards and complex structures. These findings can lead to novel directions of human and animal experimentation. I will summarize behavioral and neural findings in human and rodent studies by us and other groups and discuss the road ahead.