Enriching ConvNets with Pre-Cortical Processing Enhances Alignment with Human Brain Responses

Abstract

Convolutional Neural Networks (ConvNets) are the current state-of-the-art for modelling human visual processing whilst also performing tasks on a human per- formance level. Convolutional features can be seen as analogous to visual recep- tive fields and thus render them biologically plausible. However, spatially-uniform sampling and reuse of features across the entire visual field do not accurately rep- resent structural properties of the human visual system. Here, we present empir- ical findings of incorporating functional and structural properties of the human retina into ConvNets on their alignment with human brain activity. We show that predictions of human EEG data using ConvNets features improve by using foveated stimuli and that differential spatial sampling in ConvNets explains sev- eral qualities of EEG recordings. We also find that color and contrast filtering of inputs in turn do not yield improved predictions. Overall, our results suggest that incorporating biologically plausible spatial sampling is important for increasing representational alignment between ConvNets and humans.