Timezone: »

 
Memory in humans and deep language models: Linking hypotheses for model augmentation
Omri Raccah · Phoebe Chen · Theodore Willke · David Poeppel · Vy Vo

@
in Memory in Artificial and Real Intelligence (MemARI) »
Event URL: https://openreview.net/forum?id=5VCN8KbaCO »

The computational complexity of the self-attention mechanism in Transformer models significantly limits their ability to generalize over long temporal durations. Memory-augmentation, or the explicit storing of past information in external memory for subsequent predictions, has become a constructive avenue for mitigating this limitation. We argue that memory-augmented Transformers can benefit substantially from considering insights from the memory literature in humans. We detail an approach to integrating evidence from the human memory system through the specification of cross-domain linking hypotheses. We then provide an empirical demonstration to evaluate the use of surprisal as a linking hypothesis, and further identify the limitations of this approach to inform future research.

Keywords: memory-augmentation · Transformer models · human memory · temporal dependencies

Author Information

Omri Raccah (New York University)
Phoebe Chen (New York University)
Theodore Willke (Intel Corporation)
David Poeppel (New York University)
Vy Vo (Intel Corporation)

More from the Same Authors