Cooking Activities Recognition in Egocentric Videos Using Combining 2DCNN and 3DCNN

Abstract

Recently activities recognition in egocentric videos using wearable camera is one of the hot topic in computer vision. We mainly focus on the problem of cooking activities recognition in egocentric videos. An accurate cooking activities recognition enable us to realize cooking assistance service. In this paper, for the first step to realize cooking assistance service, we make a new dataset which consists of 8 cooking activities which are depending on different hand motions and utensils. Then we propose a new cooking activities recognition method in egocentric videos. Our proposed method has three characteristic points; 1) hand regions are detected in egocentric videos including cluttered background, 2) the region around hand is extracted based on detected hand region to reduce the effects of cluttered background, 3) cooking activities recognition by combining appearance-based approach using 2D convolutional neural networks (2DCNN) and motion-based approach using 3D convolutional neural networks (3DCNN). From experimental results, we have confirmed that our proposed method allows us to recognize cooking activities with 97.5% accuracy on our new dataset and the performance of activities recognition is improved by combined appearance-based 2DCNN and motion-based 3DCNN. We also investigate the environmental robustness of our proposed method.