One-Shot Skeleton-Based Action Recognition on Strength and Conditioning Exercises

Abstract

There is a need in the sports and fitness industry for a practical system that can identify and understand human physical activity to enable intelligent workout feedback and virtual coaching. Such a system should be able to classify an athlete’s actions from only limited examples since it is not feasible to collect a large quantity of human data for every action of interest. In this paper, we present SUE-MD, a novel dataset of skeleton motion sequences of seven common strength and conditioning exercises as captured by both a markerless and marker-based motion capture system. We then formulate the one-shot skeleton action recognition problem as a deep metric learning problem. We use the state-of-the-art graph convolutional network (GCN) to project dissimilar actions further away and similar actions closer together in the learned metric space. By training on NTU RGB+D 120, the metric GCN achieves a one-shot performance of 87.4% on all seven never-before-seen actions. In addition, an ablation study reveals the effect of different losses, embedding sizes and augmentations. Our results show that one-shot metric learning method can be used as a means to classify sports actions in a virtual coaching system where users cannot provide many expert examples for the enrolment of new actions.