Multimodal Neural Networks Better Explain Multivoxel Patterns in the Hippocampus

Abstract

The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP [1].Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that ``multimodality'' stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.

PDF NeurIPSW OpenReview Semantic Scholar

Cite

Text

Choksi et al. "Multimodal Neural Networks Better Explain Multivoxel Patterns in the Hippocampus." NeurIPS 2021 Workshops: SVRHM, 2021.

Markdown

[Choksi et al. "Multimodal Neural Networks Better Explain Multivoxel Patterns in the Hippocampus." NeurIPS 2021 Workshops: SVRHM, 2021.](https://mlanthology.org/neuripsw/2021/choksi2021neuripsw-multimodal/)

BibTeX

@inproceedings{choksi2021neuripsw-multimodal,
  title     = {{Multimodal Neural Networks Better Explain Multivoxel Patterns in the Hippocampus}},
  author    = {Choksi, Bhavin and Mozafari, Milad and VanRullen, Rufin and Reddy, Leila},
  booktitle = {NeurIPS 2021 Workshops: SVRHM},
  year      = {2021},
  url       = {https://mlanthology.org/neuripsw/2021/choksi2021neuripsw-multimodal/}
}