Language Decoding from Human Brain Activity via Contrastive Learning

Abstract

We propose a novel contrastive learning approach to decode brain activity into sentences by mapping fMRI recordings and text embeddings into a shared representational space. Using data from three subjects, we trained a cross-subject fMRI encoder and demonstrated effective sentence identification with a retrieval module. Our model shows strong alignment between brain activity and linguistic features, with top-1 accuracy up to 49.2\% and top-10 accuracy up to 84\%, significantly outperforming chance levels. These results highlight the potential of contrastive learning for cross-subject language decoding,

PDF NeurIPSW OpenReview Semantic Scholar

Cite

Text

Ferrante et al. "Language Decoding from Human Brain Activity via Contrastive Learning." NeurIPS 2024 Workshops: UniReps, 2024.

Markdown

[Ferrante et al. "Language Decoding from Human Brain Activity via Contrastive Learning." NeurIPS 2024 Workshops: UniReps, 2024.](https://mlanthology.org/neuripsw/2024/ferrante2024neuripsw-language/)

BibTeX

@inproceedings{ferrante2024neuripsw-language,
  title     = {{Language Decoding from Human Brain Activity via Contrastive Learning}},
  author    = {Ferrante, Matteo and Toschi, Nicola and Huth, Alexander},
  booktitle = {NeurIPS 2024 Workshops: UniReps},
  year      = {2024},
  url       = {https://mlanthology.org/neuripsw/2024/ferrante2024neuripsw-language/}
}