Estimating human poses, muscle activations, and ground reaction forces from monocular video

Abstract

Human pose estimation (HPE) generally refers to extracting human kinematics from various inputs, such as videos, inertial measurement units, and marker-based motion cap- ture. This thesis pertains to monocular HPE, where the input source is a single-view video. From an application standpoint, this overcomes many physical and financial limi- tations that other modalities involving wearable sensors or expensive hardware may pose. For example, in sports, monocular HPE allows the user to analyze athletes outside a lab, from opposing teams, or in the past, while being more accessible to amateur and para athletes, or anyone with a smartphone. In this thesis, we focus on physics-based HPE (PHPE), where we also estimate kinetics. Monocular PHPE is a challenging task, as predicting 3D dynamics from a single camera view is an under-constrained problem. To address this, we developed MusclePose as the first pose estimator to incorporate muscle dynamics modeling, via muscle torque generators, for regularization. We showed improvements in biofidelity over state-of-the-art (SOTA) regression-based PHPE methods, with more realistic force and torque predictions. However, regression-based methods, including MusclePose, can have performance drops when applied to new domains. To this end, we developed SPUD as a diffusion-based optimization framework for PHPE, with two novel diffusion priors for kinetics and bone estimation. We showed improved adaptability over SOTA methods via experiments on two disparate sports — fencing and golf. SPUD is also the first PHPE method to incorporate the more anatomically accurate SKEL human model over SMPL’s kinematic skeleton, allowing for better compatibility with common biomechanics standards. Lastly, to overcome the lack of force-annotated video datasets for athletic human move- ments, we collected a motion capture dataset for fencing. This dataset includes over 13 thousand video frames of beginner and national level fencers performing 13 different fenc- ing techniques, synchronized with Vicon markers and force plates. We hope this new evaluation dataset can advance the current state of PHPE research.