The Power Is in Your Hands: 3D Analysis of Hand Gestures in Naturalistic Video

Abstract

We study natural human activity under difficult settings of cluttered background, volatile illumination, and frequent occlusion. To that end, a two-stage method for hand and hand-object interaction detection is developed. First, activity proposals are generated from multiple sub-regions in the scene. Then, these are integrated using a second-stage classifier. We study a set of descriptors for detection and activity recognition in terms of performance and speed. With the overarching goal of reducing 'lab setting bias', a case study is introduced with a publicly available annotated RGB and depth dataset. The dataset was captured using a Kinect under real-world driving settings. The approach is motivated by studying actions-as well as semantic elements in the scene and the driver's interaction with them-which may be used to infer driver inattentiveness. The proposed framework significantly outperforms a state-of-the-art baseline on our dataset for hand detection.