Physics-Guided Human Motion Capture with Pose Probability Modeling

1 Introduction

Human motion capture is a fundamental task in sports broadcasting, human behavior understanding, and virtual reality, which require the accurate perception of human pose, position, and contact. Previous kinematics-based works [Kocabas et al., 2020; Arnab et al., 2019; Kanazawa et al., 2019; Rempe et al., 2021] neglect the physical laws when exploring human motion capture from monocular videos and images. As shown in Fig. 1 (a), even state-of-the-art monocular kinematics-based motion capture suffers from artifacts (e.g., floating, foot sliding, and ground penetration) due to the occlusion and depth ambiguity.

To tackle this problem, recent works introduce physical laws in human motion capture. The optimization-based framework [Huang et al., 2022a; Shimada et al., 2020] relieves artifacts by solving a highly-complex formulation. Others rely on Reinforcement Learning (RL) [Yuan et al., 2021; Luo et al., 2021] with non-differentiable physics simulators to obtain a physically plausible human motion. However, these methods all use a physical character to track the kinematic motion, and the physics is treated as a postprocessing module. The noises in the kinematic motion always lead to tracking failure and thus result in a low success rate.

To address these limitations, our key idea is to employ physics as denoising guidance in a reverse diffusion process to reconstruct physically plausible human motion from modeled pose probability distributions. Thus, physics can guide the denoising process to progressively improve the motion quality. Nonetheless, its implementation still faces several technical obstacles. First, since highlyprecise simulators are non-differentiable, the physics module cannot be incorporated into the network to provide explicit gradients to optimize human motions. In the generative task, PhysDiff [Yuan et al., 2022] directly uses a tracked motion as the input of the reverse diffusion step to avoid artifacts. However, this method also requires relatively high-quality reference motion. In addition, the strategy cannot be directly applied in motion capture since it does not consider 2D observations. Second, the denoising models [Ho et al., 2020; Song et al., 2020a] always start from the standard gaussian distribution to generate a sample [Li et al., 2022b; Ramesh et al., 2022; Zhang et al., 2022; Yuan et al., 2022], which ignores the prior knowledge from image observations and may require thousands of denoising steps to reconstruct a satisfactory motion.

To fully utilize motion prior knowledge to improve the denoising efficiency, we first train a latent gaussian model based on Variational Autoencoder (VAE) [Kingma and Welling, 2013]. With the trained VAE encoder, the image features are mapped to a series of gaussian distributions to reflect the 3D probabilistic motion. The modeled probabilistic distributions can be used as good initial values to facilitate the denoising process. To alleviate the artifacts in the reconstructed motion, we further propose a physics module in the reverse diffusion model to provide implicit guidance for the denoising. Different from PhysDiff [Yuan et al., 2022], we utilize the discrepancies between the tracked motion and image observation to guide the reverse diffusion process in the next timestep. Specifically, we project the tracked joint positions to the 2D image plane and calculate the projection loss gradients. Then, we use the combination of the gradients and image features as a condition, and feed the tracked motion to the next reverse diffusion step. After several iterations, physics-based tracking and kinematic denoising can promote each other to obtain a physically plausible human motion. The main contributions of this paper are as follows:

• We construct a physical guidance to combine the physics and image observations for the reverse diffusion process to progressively promote a physically plausible human motion capture.
• We propose a VAE-based latent gaussian distribution to facilitate the reverse diffusion process with motion prior knowledge.
• We incorporate physics and kinematics in the same framework to improve joint accuracy and success rate for physics-based human motion capture.

2 RelatedWork

2.1 Kinematics-BasedMotion Capture

Previous monocular kinematics-based motion capture leverages 3D pose estimation [Li et al., 2021; Li et al., 2022c; Song et al., 2020b] for each frame to construct the human motion. They cannot obtain the temporal information among frames, which leads to obvious jittering. Recent works [Kocabas et al., 2020; Luo et al., 2020] exploit the temporal context of human motion for better temporal consistency. These methods encounter global inconsistency since they can only produce a root-relative motion. Several approaches [Arnab et al., 2019; Xiang et al., 2019] adopt smooth priors over time to reduce jittering. However, the smooth constraints may result in over-smooth and footskate. Since the aforementioned methods do not consider depth ambiguity and occlusion from monocular motion capture, recent diffusion-based works [Gong et al., 2022a; Choi et al., 2022;

Holmquist and Wandt, 2022] model the uncertainty of 2D3D lifting for 3D pose estimation. Although the aforementioned approaches achieve great performance on kinematic metrics, they still encounter artifacts since they ignore the physics laws.

2.2 Physics-BasedMotion Capture

To relieve the artifacts of human motion capture, [Wei and Chai, 2010; Vondrak et al., 2012; Zell et al., 2017; Shimada et al., 2020; Shimada et al., 2021; Li et al., 2022d; Yi et al., 2022] adopt optimization to obtain the physical forces to induce the human motion, which results in high approximation errors since they do not utilize the highlyprecise non-differentiable simulators. While Neural Mocon [Huang et al., 2022a] generates a learned motion distribution for sampling-based motion control with supervision from a non-differentiable simulator. Others [Yuan et al., 2021; Luo et al., 2021; Yuan and Kitani, 2020; Yu et al., 2021; Peng et al., 2018] use Reinforcement Learning (RL) with non-differentiable simulators to obtain physically plausible human motion. However, all the approaches adopt a strong assumption that the reference motion is accurate. Luo et al. [Luo et al., 2022] adopts current character state and environmental cues to promote tracking when the reference pose is unreliable. The accuracy of the estimated motion relies on the 2D observations. Since all these approaches implement physics as a post-processing process, the character always fails to track due to the noises in the reference motion. In contrast, we propose a physics module in the reverse diffusion model to provide implicit guidance for the denoising. In this case, the physics-based tracking and kinematic denoising can promote each other for physically plausible human motion capture.

2.3 Multi-Hypothesis Estimation

Human motion capture from monocular videos and images is an ill-posed problem, for which directly regressing a determinate solution may be inaccurate [Kocabas et al., 2020; Arnab et al., 2019; Kanazawa et al., 2019]. Multi-hypothesis methods [Huang et al., 2022b; Li and Lee, 2019; Jahangiri and Yuille, 2017] are proposed to represent the uncertainty of 2D-3D lifting. Recently, Sharma et al. [Sharma et al., 2019] adopted a conditional VAE to predict 3D pose candidates conditioned on detected 2D poses. Wehrbein et al. [Wehrbein et al., 2021] employs normalizing flow to model the posterior distribution of 3D poses. However, it is difficult to select the best 3D pose from multi-hypothesis. Unlike these works, we propose a physics module to provide implicit guidance for the denoising process to promote a more accurate human motion.

3 Method

We aim to reconstruct the physically plausible 3D human motion from monocular videos. We first design a latent gaussian distribution encoded from image features to provide initial values to facilitate the reverse diffusion process (Sec. 3.2). To alleviate the artifacts in the reconstructed motion, we further propose a physics module to combine the physics and observed 2D poses to guide the denoising of reverse diffusion (Sec. 3.3). With the constructed framework, the physics and kinematics can progressively promote each other to obtain physically plausible motions with a high success rate (Sec. 3.4).

3.1 Preliminaries

MotionRepresentation. The 2Dposes with corresponding confidence detected by AlphaPose [Fang et al., 2017] from color images I 1:H are defined P2D = {ph2D ∈ RJ×3}Hh=1 , where H is the length of motion and J is the number of joints. We adopt SMPL model [Loper et al., 2015] to represent the kinematic human motion x1:H = {xh}Hh=1 , where x denotes the parameters of human pose θ in 6D representation [Zhou et al., 2019]. We also regress SMPL shape β and translation τ . The physics-based human motion is defined as ˜x1:H .

Physics-Based Tracking. We briefly introduce the physical imitation. The character in the physics engine is created based on the SMPL kinematic tree and estimated body shape β. We construct convex hulls to approximate mesh [Luo et al., 2021], which can be simulated in MuJoCo [Todorov et al., 2012] and shares the same pose parameters with SMPL model. The physical imitation aims to control the character to track the reference motion x1:H . With a trained policy π(ah | sh , xh+1 ), we can sample an action ah according to the current state sh and reference pose xh+1 to control the character to move to the next state sh+1 . The state sh = (˜xh , ˙˜xh ) contains the character's current pose ˜xh and joint velocity ˙˜xh . Finally, the physically plausible human motion ˜x1:H can be obtained from the simulated character state. More details can be found in the supplementary material.

Diffusion Model. The diffusion model [Tevet et al., 2022] can generate target data from a simple noise distribution under a condition. Specifically, the forward diffusion process gradually adds infinitesimal gaussian noise ϵ on the data x1:H t ∼ q(x1:H t ) at timestep t, which is formulated as:

where αt is a manually designed constant hyper-parameter, and ˆαt = ∏ti=0 αi, and ¯x1:H 0 is the ground truth original data. The reverse diffusion process samples an initial input from the standard gaussian distribution and progressively denoises it to the target data under the guidance of condition c, which is defined as:

where µα(x1:H t , c) is the estimated mean by a neural network and ˜ βt is the variance which is calculated by the hyperparameters βt, ˆαt and ˆαt-1.

3.2 Probabilistic Modeling of 3DHumanMotion

In this work, we formulate the motion capture as a reverse diffusion process conditioned on image observations. The previous diffusion models [Choi et al., 2022; Gong et al., 2022a] always start from the standard gaussian distribution, which requires thousands of reverse diffusion timesteps to denoise the input to produce a desired result. Although a recent work [Gong et al., 2022a] separately models the initial distribution for each pose coordinate, it neglects the correlation among different joints. To improve efficiency, we leverage prior knowledge from image features to provide initial values for the reverse diffusion process.

Inspired by the previous works [Huang et al., 2022c; Huang et al., 2021], we first extract image features F 1:H with a backbone network, and then fed the features into a VAE model to output 3D motion x1:H , translation τ and body shape β. Since we adopt a Gate Recurrent Unit (GRU) for the VAE encoder, the correlated gaussian distributions {N (µθ (F h ) , σθ (F h ))}Hh=1 in the latent space can describe the motion prior knowledge. During the training phase, we maximize the Evidence Lower Bound (ELBO) to train the model:

The overall loss function is:

where Lmotion and Lshape are used to supervise the estimated human motion and shape:

where ¯xh and ¯ β are ground truth motion and shape.

where ¯ J h 3D and J h 3D are the ground truth and predicted 3D joint positions generated from the corresponding motion. We also follow CLIFF [Li et al., 2022c] to supervise the predicted joints in the original camera coordinates with detected 2D poses:

where Π denotes the projection operation.

which is used to push the output of the encoder to approximate the gaussian distribution.

After the training, we freeze the network parameters of the VAE encoder and use it to generate specific gaussian distributions from image features for the reverse diffusion process.

3.3 Physics-GuidedMotion Diffusion

Although the human motion sampled from the encoded distributions can match the image observations, it is still physically implausible due to the occlusion and depth ambiguity. Thus, we incorporate physics to refine the kinematic motion. Previous works [Yuan et al., 2021; Shimada et al., 2020] directly use the physics module as a post-processing to track the kinematic motion. However, the noises in kinematic motions always result in tracking failure. To address the obstacle, we use physics to guide the reverse diffusion process to denoise the motion.

Unlike previous diffusion-based pose estimation [Choi et al., 2022; Holmquist and Wandt, 2022], we start from the modeled probabilistic distributions, which can provide prior knowledge to improve the efficiency of the denoising process. To train the diffusion model, we first use the trained VAE encoder to produce gaussian distributions N (µ1:H θ , σ1:H θ ) for the diffusion framework. The difference from the standard diffusion process Equ. (1) is that we sample the noise ϵ from the encoded distributions rather than the standard gaussian distribution. We then gradually add the sampled noises on the 3D motion ¯x1:H 0 towards the uncertainty distribution N (µ1:H θ , σ1:H θ ).

In the reverse diffusion process, we train a network to denoise the noisy motion x1:H T to the original data x1:H 0 . Since the distributions have encoded prior knowledge, the noisy motion only contains a few artifacts and is still close to the real motion. Although directly applying the physics-based tracking on the noisy reference motion may fail, the discrepancies between the tracked motion and 2D poses can provide implicit guidance for the next reverse diffusion step to optimize the kinematic motion. Thus, we combine the projection loss gradient and image features as a condition, and feed the tracked motion to the network to predict the distributions of the next step.

We follow Ramesh et al. [Ramesh et al., 2022] to make the diffusion model to predict the target data ˇx1:H 0 , and then construct the mean of the distribution µα ( ˜x1:H t , ct ) in timestep t - 1 according to ˇx1:H 0 . The condition ct is a concatenated vector of image features and projection loss gradient information. The image features vector is encoded from extracted image features with a linear layer, and the gradient vector records the gradient of each frame.

We mask the gradient for the frames that are not successfully tracked. We also fill the gradient factor with 0 when the physical guidance is not executed.

Training Procedure. For the physics-based tracking, we first follow [Luo et al., 2021] to train a policy π on motion capture datasets by maximizing the expected return, and the parameters of the trained policy are fixed in the diffusion model training. To train the diffusion model, the following loss function is adopted:

where F ( ˜x1:H t , t, ct ) is the output of the neural network. The functions Ljoint and Lreproj are the same as Equ. (7) and Equ. (8).

3.4 Reverse Diffusion for Motion Capture

Once the networks are trained, we can construct a physicsbased human motion capture framework. We first extract image features with a backbone network, and then predict 2D poses for each frame. Then, the latent gaussian distributions can be generated from the extracted image features with the trained VAE encoder. A noisy initial motion is sampled from the distributions for the reverse diffusion process. By applying the physics-based tracking, we can combine the physics information with the image features to guide the denoising. Finally, the physically plausible human motion can be obtained after several iterations.

4 Experiments

4.1 Metrics

We report the Mean Per Joint Position Error (MPJPE) and the MPJPE after aligning the prediction with ground truth using Procrustes Analysis (PA-MPJPE) to evaluate joint accuracy. We use the 3D extension of the Percentage of Correct keypoints (PCK) at the threshold of 150mm to evaluate the 3D joint position accuracy. We follow PhysCap [Shimada et al., 2020] to measure motion jitter error by es, which is the deviation of joint velocity between predicted output and ground truth. The es and its standard deviation σs are adopted to assess the motion smoothness. To evaluate foot contact, we adopt ef,z proposed in [Xie et al., 2021], which is the foot position error on the z-axis.

4.2 Datasets

Human3.6M [Ionescu et al., 2013] is an indoor dataset for human motion capture. The videos are captured at 50Hz which includes 7 subjects. Following previous works [Yuan et al., 2021; Shimada et al., 2020], we use 2 subjects (S9, S11) for evaluation, and the others are used for training. We convert the dataset to 30Hz to reduce redundancy.

3DOH [Zhang et al., 2020] is the first dataset to handle the object occluded human body estimation, which contains 3D motions in occluded scenarios. We use the sequence 0013, 0027, 0029 to evaluate our method in occlusion cases.

MPI-INF-3DHP [Mehta et al., 2017] is a single-person 3D pose dataset. Following previous works [Gong et al., 2022b;

Shimada et al., 2021], we use its test set to demonstrate the generalization of our method.

4.3 Comparison with State-of-the-Art Methods

We compared our method with state-of-the-art dynamicsbased human motion capture approaches on Human3.6M dataset, and their average errors are shown in Tab. 1. All approaches in Tab. 1 are dynamics-based methods. EgoPose, SimPoE, and our method rely on RL policy to control the character. When the kinematic motion is inaccurate, EgoPose and SimPoE may fail to track and require re-initialization. In contrast, our method can progressively denoise the artifacts in the reference motion during the reverse diffusion process and promote successful tracking. Thus, our method outperforms these two techniques. In addition, we found that D&D can achieve the best performance in terms of MPJPE and PAMPJPE. However, D&D uses a kinematics-based network to implicitly learn the physical laws, from which the output motion may still contain artifacts. We also use other metrics like motion smoothness and foot contact error to measure the physical plausibility. The results show that our method achieves state-of-the-art on most of the metrics. We further conduct a qualitative comparison with both kinematics-based and physics-based methods in Fig. 3. VIBE [Kocabas et al., 2020] and HuMoR [Rempe et al., 2021] are recent works that exploit temporal information to obtain kinematic motions. The results show that the reconstructed motion in 3D scenes cannot get accurate contact. Besides, PhysCap is an optimization-based method that can obtain almost physically plausible human motion. However, it also does not consider the scene interactions and cannot utilize 2D observations in physics-based tracking. Thus, the results may hover above the floor and deviate from 2D poses.

We further conduct experiments on 3DOH dataset. Since the dataset contains a lot of object occlusions, it is more difficult to reconstruct an accurate motion. In Fig. 3, VIBE and HuMoR cannot get satisfactory results due to the depth ambiguity and occlusions. Although Neural MoCon outputs more precise joint positions, it uses a skeletal character and cannot reconstruct accurate body contact. PhysCap, SamCon, and Neural MoCon formulate the physics-based motion capture as a trajectory optimization, and the tracking results strongly depend on the quality of the estimated reference motion. In contrast, our method can adjust both the kinematic and physical motion in the reverse diffusion process. The quantitative results in Tab. 1 show that our method significantly outperforms these baseline methods in all metrics.

To evaluate the generalization of our method, we use 3DHP dataset as a benchmark. The results in Fig. 4 show that our method can obtain accurate motion with precise contact on different scenes. We also compared our method with PhysCap, NeuralPhysCap, and PoseTriplet on 3DHP. The results in Tab. 1 show that our method gets more accurate joint positions than PhysCap and NeuralPhysCap, but inferior to PoseTriplet. The reason is that PoseTriplet trains the kinematics-based network assisted by physics but discards the physics module in the pose estimation. Thus, the results from the trained network may also contain artifacts.

4.4 Ablation Study

Latent Gaussian Distribution. The conventional diffusion model requires a lot of denoising steps to reconstruct a satisfactory motion, which is inefficient for the motion capture task. To facilitate the reverse diffusion process, we propose a VAE-based latent gaussian distribution to employ the motion prior knowledge for a good initial value. In Tab. 3, we compared the proposed latent distribution with the standard gaussian distribution in our motion capture framework. The results show that we can directly sample plausible motions from the encoded distributions. Although the sampled motion may contain a lot of artifacts due to the depth ambiguity, it still has a relatively high joint accuracy. In contrast, we require more denoising steps to reconstruct a motion when we use the standard gaussian as the initial distribution. In the first several steps, the sampled results are pure noises. The framework needs more than 10 steps to reconstruct the motion.

Physical Guidance. We study the physical guidance in this section. The conventional kinematics-based motion capture predicts the 3D motion from pure image features, which cannot avoid the artifacts due to the depth ambiguity. In Tab. 4, although the PA-MPJPE of the results sampled from the latent distribution is 58.6, it contains a lot of jitters. We can find that the diffusion model with only kinematics cannot remove the artifacts. Existing physics-based motion capture frameworks use physics as post-processing after the kinematics-based prediction. To demonstrate its weakness with a fair setting, we directly add physics-based tracking on the sampled motion. This strategy can alleviate the artifacts, but it has a low success rate since the noises in the kinematic motion always result in tracking failure. We further use the denoising framework to refine the kinematic motion. The results in Tab. 4 show that the kinematics-based diffusion model can improve the joint accuracy, but does not prevent the artifacts. By incorporating the physical guidance, the success rate can be significantly improved. In addition, the results in PA-MPJPE also demonstrate that the physics also provides the correct direction to enhance the motion capture accuracy. To compare the strategy adopted in PhysDiff, we remove the projection loss gradient condition, and use the tracked motion for the input of the next denoising step in Tab. 5. We found that the gradients can provide implicit guidance for the diffusion model to optimize the kinematic motion, and thus improve the success rate in the physics-based tracking.

Denoising Step. We also study the impact of the denoising step. In Tab. 4, we found that the joint accuracy can be significantly improved with the physical guidance. The performance increases at first with more times of physical guidance and then becomes stable. In addition, the gains are also declining with more than 7 denoising steps.

5 Conclusion

In this work, we formulate the physically plausible human motion capture as a reverse diffusion process. Latent gaussian distributions are built based on VAE to utilize the motion prior knowledge to facilitate the reverse diffusion. To employ physics information, we further construct a physics module to combine the physics and image observations to guide the denoising. With the physical guidance, physics and kinematics can promote each other to progressively reconstruct a highquality human motion. Experimental results on several human motion capture datasets demonstrate that our method can achieve competitive performance with a higher success rate.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (No. 62076061) and the Natural Science Foundation of Jiangsu Province (No. BK20220127). The authors would like to thank Yuan Yang for the helpful discussion about physics-based imitation.

Contribution Statement

Jingyi Ju and Buzhen Huang contribute equally to this work. The corresponding author is Yangang Wang.