Still Image Action Recognition by Predicting Spatial-Temporal Pixel Evolution

Abstract

Both spatial and temporal patterns provide crucial information for recognizing human actions. However, lack of temporal information in still images is a major obstacle in single-image action recognition. In this paper, (i) We introduce a novel image representation domain, Ranked Saliency Map and Predicted Optical Flow or Rank_SM-POF for short. This domain captures both actor appearance and the future movement patterns of the actor. This is accomplished through capturing the temporal ordering of each pixel by training a linear ranking machine on the predicted tensor of spatial-temporal representation of images. (ii) We employ a transfer learning approach to propose a new spatial-temporal Convolutional Neural Network, named STCNN for the task of single image action classification, by fine-tuning a CNN which is pre-trained specifically for appearance based classification. (iii) Finally, extensive experiments on five benchmarks clearly demonstrate that appearance and motion are complementary sources of information and using both leads to significant performance improvement in single image action recognition, hence outperforming state-of-the-art methods.