Towards Wearable Multi-Modal Human Activity Recognition with Deep Fusion Networks

Abstract

This paper presents a novel multi-modal approach to human activity recognition that capitalizes on the complementary strengths of wearable video and sensor data to enhance accuracy and versatility. We focus on standard deep neural networks suitable for edge devices, which handle the complexities of preprocessing and offer various ways for information fusion. As features, optical flow fields are derived and embedded with a convolutional autoencoder to robustly capture both self-induced scene flow and articulated self-motion. The approach thus transforms a learning problem with multiple heterogeneous modalities into a homogeneous one, facilitating further models specialized in exploiting dependencies in multidimensional time-series data. The competitive performance of this approach in multi-modal human activity recognition scenarios is demonstrated, showcasing its efficacy using various deep learning data fusion architectures on the recent WEAR dataset. The achieved results may provide valuable insights for wearable computing sensor manufacturers or assistive computing developers.