Timezone: »

Multimodal neural networks better explain multivoxel patterns in the hippocampus
Bhavin Choksi · Milad Mozafari · Rufin VanRullen · Leila Reddy

Mon Dec 13 12:00 PM -- 01:00 PM (PST) @
Event URL: https://openreview.net/forum?id=6dymbuga7nL »

The human hippocampus possesses concept cells'', neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP [1].Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate thatmultimodality'' stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.

Author Information

Bhavin Choksi (CNRS)
Milad Mozafari (CNRS)
Rufin VanRullen (CNRS - CerCo (Toulouse))
Leila Reddy

More from the Same Authors