Scalable Humanoid Whole-Body Control via Differentiable Neural Network Dynamics

Abstract

Learning scalable humanoid whole-body controllers is crucial for applications in animation and embodied intelligence. While popular model-free reinforcement learning methods are capable of learning controllers to track large-scale motion databases, they require an exorbitant amount of samples and long training times. Conversely, learning a robust world model has emerged as a promising alternative for efficient and generalizable policy learning. In this work, we learn a neural dynamics model and propose a novel framework that combines supervised learning and reinforcement learning for scalable humanoid controller learning. Our method achieves significantly higher sample efficiency and lower tracking error compared to prior approaches, scaling seamlessly to datasets with tens of thousands of motion clips. We further show that medium-sized neural dynamics models can serve as a differentiable neural simulator for accurate prediction and effective policy optimization. We also demonstrate the effectiveness of our framework on spare reward tasks and the transferability of learned neural dynamics model to diverse tasks.