Increasing Video Saliency Model Generalizability by Training for Smooth Pursuit Prediction

Abstract

Saliency prediction even for videos is traditionally associated with fixation prediction. Unlike images, however, videos also induce smooth pursuit eye movements, for example when a salient object is moving and is tracked across the video surface. Nevertheless, current saliency data sets and models mostly ignore pursuit, either by combining it with fixations, or discarding the respective samples. In this work, we utilize a state-of-the-art smooth pursuit detector and a Slicing Convolutional Neural Network (S-CNN) to train two saliency models, one targeting fixation prediction and the other targeting smooth pursuit. We hypothesize that pursuit-salient video parts would generalize better, since the motion patterns should be relatively similar across data sets. To test this, we consider an independent video saliency data set, where no pursuit-fixation differentiation is performed. In our experiments, the pursuit-targeting model outperforms several state-of-the-art saliency algorithms on both the test part of our main data set and the additionally considered data set.