A Systematic Comparison of fMRI-to-Video Reconstruction Techniques

Abstract

Recent advances in generative models and large-scale neural datasets have brought forth novel methods to reconstruct stimuli from brain activity. This rapidly evolving family of brain-to-stimuli reconstruction techniques has the opportunity to revolutionize fundamental brain sciences and human-computer interaction applications, yet systemic comparisons of these techniques are lacking. Here, we explore a novel method to reconstruct short videos from functional magnetic resonance imaging (fMRI) brain activity of human subjects that achieves state-of-the-art performance as assessed by a suite of evaluation metrics. We perform preliminary comparisons of reconstruction quality within our pipeline by testing different combinations of semantic encoders and video generation models. Lastly, we compare our pipeline's best reconstruction results with previous work. Together, this work comprehensively assesses state-of-the-art methodologies in the increasingly important discipline of brain-to-video reconstruction.