Temporal Constrained Feasible Subspace Learning for Human Pose Forecasting

1 Introduction

Human pose forecasting is a sequential modeling task that aims to predict future poses from historical motions. This task has received increasing attention in numerous applications, such as autonomous driving [Paden et al., 2016; Mangalam et al., 2020], healthcare [Troje, 2002], teleoperations [Rubagotti et al., 2019], and collaborative robots [Koppula and Saxena, 2013; Unhelkar et al., 2018]. Unlike human pose estimation task [Gu et al., 2019; Gu et al., 2021; Li et al., 2022b; Zhang et al., 2022; Chai et al., 2023] that predicts the pose for observed frames, pose forecasting focuses on future pose estimation. Most of the existing approaches focus on the spatial-temporal neural network model design [Li et al., 2018; Mao et al., 2019; Mao et al., 2020; Sofianos et al., 2021; Li et al., 2022a; Bouazizi et al., 2022; Xu et al., 2022] for learning the movement patterns to reduce prediction errors. Though such data-driven deep learning models can approximate very complex functions and achieve outstanding performance with large-scale datasets, they suffer from less interpretability than conventional simple machine learning models. Unexpected output may occur when the testing data drifts from the training data distribution. Even though a small Mean Per Joint Position Error (MPJPE) is achieved, some of the predicted poses are not temporal feasible solutions. The drawback of commonly used methods is shown in Figure 1. Even though commonly used methods employ temporal constraints in the training stage, the forecasting model may still generate infeasible poses in the prediction since the constraints are NOT employed in the inference stage. For example, since a human cannot make an abrupt movement, there should be constraints on the upper bound of the change of the joint velocities. Such temporal constraints should be satisfied not only in the training stage but also in the inference stage for any unseen testing data. However, the temporal constrained feasible solutions are seldom discussed in the literature for the pose forecasting task.

Physical constraints can alleviate such issues for datadriven deep learning models. By bringing laws from the physical world and general knowledge from humans, predictions and decisions with physical constraints become more trustworthy than those made from purely data-driven models. Some existing approaches take into account physical constraints in deep learning models, typically by adapting ideas from constrained optimization [Amos and Kolter, 2017], training algorithms/loss functions with regularization [Diligenti et al., 2017; Berk et al., 2017], or correcting the model output with iterative projections [Yang et al., 2020; Detassis et al., 2020]. However, most existing approaches use approximation rather than searching for feasible solutions that strictly comply with the constraints. Besides, iterative adaptation may be required in the inference stage, which takes extra computational costs. Furthermore, compared with unconstrained models, constrained models usually find suboptimal solutions, resulting in degradation in performance.

In this paper, we tackle the human pose forecasting problem with the consideration of temporal constrained feasible solutions, where the change of velocities of human joints should follow the continuity property of the physical movement. To deal with this problem, we propose a temporal constrained feasible subspace learning framework for human pose forecasting. Specifically, rather than direct supervision of the prediction in the original pose space, the temporal constrained subspace is explicitly learned by exploiting simple projection functions, such as rectified linear unit (ReLU) and exponential linear unit (ELU). Then the backward transformation can be applied to obtain the final human pose prediction. Since the constraints are guaranteed in both training and inference stages, no extra steps of iterative projections or optimization are required in the inference. Moreover, unlike most existing approaches that usually sacrifice accuracy to satisfy physical constraints, our proposed method can further reduce the prediction error with constrained feasible solutions. Our framework is shown in Figure 2. The key contributions are summarized as follows:

• We apply temporal constraints in the human pose forecasting task and provide feasible solutions for both training and inference stages. Unlike most existing works, the temporal feasible solutions of our proposed method can be obtained without any iteration, adaptation, optimization, or approximation in the inference.
• We propose a novel subspace learning framework for the temporal constraints. Rather than direct supervision of the prediction in the original pose space, a temporal constrained subspace is explicitly learned, followed by an inverse transformation to obtain the final predictions.
• With the STS-GCN [Sofianos et al., 2021] as the encoder backbone network, we achieve state-of-the-art (SOTA) performance on the Human3.6M [Ionescu et al., 2013], AMASS [Mahmood et al., 2019], and 3DPW [von Marcard et al., 2018] datasets in the human pose forecasting task with temporal feasible solutions.

2 RelatedWorks

Human pose forecasting Some works focus on motion modeling for human pose forecasting. For example, [Chiu et al., 2019] propose a new action-agnostic method for shortand long-term human pose forecasting with triangular-prism RNN for modeling the hierarchical and multi-scale characteristics of human dynamics. [Mao et al., 2019] propose a simple feed-forward deep network for motion prediction, which takes into account both temporal smoothness and spatial dependencies among human body joints. [Mao et al., 2020] propose to extract motion attention to capture the similarity between the current motion context and the historical motion sub-sequences. [Sofianos et al., 2021] propose a space-timeseparable graph convolutional network for human pose forecasting. [Adeli et al., 2021] propose a novel trajectory and pose dynamics method based on graph attention networks to model the human-human and human-object interactions both in the input space and the decoded future output space.

Some works combine reinforcement learning in the formulation. Specifically, [Wang et al., 2019] propose a new reinforcement learning formulation for the problem of human pose prediction and develops an imitation learning algorithm for predicting future poses under this formulation through a combination of behavioral cloning and generative adversarial imitation learning. [Yuan and Kitani, 2019] propose the use of a proportional-derivative (PD) control-based policy learned via reinforcement learning to estimate and forecast 3D human poses from ego-centric videos.

Some works consider the action characteristic and scene context information in the setting. For example, [Diller et al., 2020] propose the task of forecasting characteristic 3D poses: from a short sequence observation of a person, predict a future 3D pose of that person in a likely action-defining, characteristic pose. [Adeli et al., 2020] consider incorporating both scene and social contexts as critical clues for the human motion and pose forecasting task.

And some works take account of embedding, constraints, and subtasks in the formulation. For example, [Mangalam et al., 2020] tackle the problem of Human Locomotion Forecasting, a task for jointly predicting the spatial positions of several keypoints on the human body in the near future under an ego-centric setting and presents a method to disentangle the overall pedestrian motion into easier-to-learn subparts by utilizing a pose completion and a decomposition module. [Parsaeifard et al., 2021] propose to learn decoupled representations for the global and local pose forecasting tasks. [Wang et al., 2022] propose the velocity-to-velocity learning paradigm for human motion prediction, which attempts to directly build the sequence-to-sequence model in the velocity space. However, for most existing works, temporal constraints are usually treated as regularization terms in the learning for human pose forecasting. How to guarantee the temporal constrained feasible solutions is not discussed in such approaches.

Physically constrained learning There are some existing works that deal with physical constraints in deep neural networks. Specifically, [Yang et al., 2020] propose a new family of neural networks to predict the behaviors of physical systems by learning their underpinning constraints. [Sangalli et al., 2021] pose the training of deep neural networks for binary classification as a constrained optimization problem using an Augmented Lagrangian method (ALM). [Detassis et al., 2020] use a decomposition scheme alternating master steps and learner steps on the constrained optimization problem. [Diligenti et al., 2017] propose a unified approach to learning from constraints, which integrates the ability of classical machine learning techniques to learn from continuous feature-based representations with the ability of reasoning using higher-level semantic knowledge typical of statistical relational learning. [Amos and Kolter, 2017] present OptNet, a network architecture that integrates optimization problems as individual layers in larger end-to-end trainable deep networks. [Kotary et al., 2021a] connect the variation of the training data to the ability of a model to approximate it and propose a method for producing solutions to optimization problems that are more amenable to supervised learning tasks. [Kotary et al., 2021b] survey the recent attempts at leveraging machine learning to solve constrained optimization problems and focuses on surveying the work on integrating combinatorial solvers and optimization methods with machine learning architectures. [Huang et al., 2021] propose the combinatorially efficient, equivariant, and constraint-aware Graph Mechanics Network (GMN), where the geometrical constraints are implicitly and naturally encoded in the forward kinematic. [Rubanova et al., 2021] present a framework for constraint-based learned simulation, where a scalar constraint function is implemented as a graph neural network, and future predictions are computed by solving the optimization problem defined by the learned constraint. [Zhong et al., 2021] introduce a differentiable contact model, which can capture contact mechanics and accommodate inequality constraints. However, most existing solutions usually require iterative projection steps in the inference stage, and feasible solutions are not strictly guaranteed.

3 ProposedMethod

3.1 Overview of Human Pose Forecasting

We use the joint-skeleton model to represent the human pose. Given the 3D coordinates of V joints for T1 frames, we aim to predict the V body joints for the next T2 future frames. Denote the 3D coordinate of joint v at frame t as xv,t ∈ R3 . The motion history of human poses is denoted by Xp ∈ RV ×T1×3 for all V joints in T1 frames. We aim to learn a forecasting model Fw parameterized by w, i.e.,

where Xf ∈ RV×T2×3 represent the predicted V joints in the future T2 frames.

3.2 Temporal Constrained Subspace Learning

Usually, the learning of pose forecasting can be formulated as an optimization problem, i.e.,

where X∗f is the ground truth of future poses in the training data, and C(Fw(Xp)) ≤ 0 represents the inequality constraint. The constraint can be temporal consistency for predicted poses. For example, the difference in motion velocities of joints in adjacent timestamps cannot exceed a certain threshold. The solution should be temporal feasible and follows the constraint. To take the constraint into consideration, we describe the conventional approaches and our proposed approach to address the problem.

Conventional approaches After combining the constraints into the training stage, the training loss of conventional approaches of pose forecasting can be formulated as follows,

In this formulation, the constraint C(Fw(Xp)) ≤ 0 is converted to a regularization term R(Fw(Xp)) with the hyperparameter λ. For example, R(Fw(Xp)) can be set as [C(Fw(Xp))]+, where [·]+ clamps negative values to zeros. After training, the inference stage can be represented as

where ˆw are the learned model parameters, and ˆXf are the predicted poses in the testing data. However, the constraint is not applied in the inference stage. Due to the gap between the training and testing set, the temporal feasible solution is not guaranteed. In other words, the constraint, C(Fw(Xp)) ≤ 0, may be not satisfied in the inference stage.

Our approach To address the common problem in the conventional approaches, we propose a temporal constrained subspace learning approach. Based on Eq. (3), we formulate our approach as follows,

where the first term in the equation have three additional operations, i.e., a subspace transformation H, a projection P, and an inverse transformation H-1 . Specifically, the subspace transformation H transforms the pose coordinate space to the temporal subspace; the projection P projects the unconstrained subspace to the constrained subspace to ensure the feasibility; the inverse transformation H-1 maps from the temporal subspace back to the pose space. An illustration is shown in Figure 2.

Our proposed inference stage can be represented by

Unlike the inference stage as shown in Eq. (4) of conventional learning approaches that do not take the temporal constraint into consideration, our inference stage uses the projection P to ensure the temporal feasibility explicitly. As a result, the temporal feasibility is always guaranteed in the inference stage in our proposed method.

3.3 Approach Details

Subspace transformation We consider the temporal subspace as the change of velocity of joint coordinated between additional frames, denoted as F ′′w , i.e., F ′′w,t = (Fw,t+1(Xp) - Fw,t(Xp)) - (Fw,t(Xp) - Fw,t-1(Xp)). The subspace transformation H defined in Eq. (5) maps the original pose space Fw to the temporal subspace F ′′w , which follows a linear transformation, i.e.,

where H[t, t : (t+2)] = [1, -2, 1]. Since H is with size (T22) × T2, we concatenate the natural basis e1 = [1, 0, ..., 0]T and eT2 = [0, ..., 0, 1]T with H to form a full rank matrix ˜H = [eT1 ;H; eTT ]. For simplicity, we use H to represent ˜H. Then the temporal subspace transformation H is invertible, denoted as H-1 .

Constraint and projection The temporal constraint C(Fw(Xp)) is set as C(Fw(Xp)) = |F ′′w | - (max(X′′ p ) + δub), where max(X′′ p ) is the maximum change of velocity in the historical poses and δub is the pre-defined upper bound. The constraint assumes that the predicted poses in the temporal subspace should not exceed max(X′′ p ) + δub. For simplicity, we use ˜ δub to represent max(X′′ p ) + δub.

To explicitly follow the constraint in the temporal subspace, we use projection P that maps from unconstrained space to the constrained space. Note that we focus on the temporal feasible solutions, i.e., |F ′′w | - (max(X′′ p ) + δub) ≤ 0. Then the projection function can be explicitly defined as

The function can be easily implemented with the rectified linear unit (ReLU). We denote this projection as Prelu. Other appropriate projection functions can also be applied. Discussions are provided in the Experiments Section.

4 Experiments

4.1 Datasets andMetrics

Human3.6M [Ionescu et al., 2013] It is a large-scale dataset consisting of 3.6 million 3D human poses and corresponding images. It includes 7 actors performing 15 different actions like Walking, Eating and Phoning. Following the current literature [Mao et al., 2020; Mao et al., 2019; Martinez et al., 2017], we use subject 11 (S11) for validation, the subject 5 (S5) for testing, and all the rest of the subjects for training.

AMASS [Mahmood et al., 2019] The Archive of Motion Capture as Surface Shapes (AMASS) dataset has been recently proposed with 18 existing MoCap datasets. Following [Mao et al., 2020; Sofianos et al., 2021], we take 13 datasets from AMASS in the experiment, with 8 datasets for training, 4 for validation and 1 for testing. Then we use the SMPL [Loper et al., 2015] parameterization for the human skeleton and joint rotation angle to represent the human pose based on the shape vector. 3D Human poses are obtained by applying forward kinematics.

3DPW [von Marcard et al., 2018] The dataset consists of in-the-wild video sequences and 3D human poses captured by a moving camera. The dataset includes both indoor and outdoor actions. In total, it contains 51,000 frames captured at 30Hz, divided into 60 video sequences.

Evaluation metrics Following the benchmark protocols, we adopt the Mean Per Joint Position Error (MPJPE) metric for evaluation. It quantifies the error of the 3D coordinate predictions in mm. The MPJPE is defined as follows,

where ˆx and x∗ are predictions and ground truth joint coordinates, respectively. In addition, we also propose a novel metric, namely infeasible rate (IR), to measure the percentage of predicted joints that do not satisfy the temporal constraint C(Fw(Xp)) ≤ 0.

4.2 Implementation Details

Model architecture We adopt the Space-Time-Separable Graph Convolutional Network (STS-GCN) [Sofianos et al., 2021] for human pose forecasting, which is one of the SOTA methods for spatial-temporal graph embedding. Details of STS-GCN can be found in [Sofianos et al., 2021]. We use 4 STS-GCN layers in the encoding, which only differ in the number of channels: from 3 (the input 3D coordinates x, y, z), to 64, then 32, 64, and finally 3, by means of the projection matrices. At each layer we adopt batch normalization [Ioffe and Szegedy, 2015] and residual connections.

Training details We use Pytorch for training the neural networks and use ADAM [Kingma and Ba, 2014] as the optimizer. The learning rate is set to 0.01 and decayed by a factor of 0.1 every 5 epochs after the 20th epoch. The batch size is set to 256. The maximum epoch is set to 50. The constraint L is set to 50. One NVIDIA RTX 3090 GPU is used for training.

4.3 Comparison with State-of-the-Art Methods

Human3.6M The MPJPE error in mm on the Human3.6M dataset is shown in Table 1. Following [Sofianos et al., 2021], we estimate the human pose forecasting for 720, 880 and 1, 000 milliseconds. To show the effectiveness of the model, we also report the performance from ConvSeq2Seq [Li et al., 2018], LTD [Mao et al., 2019], RNN-GCN [Mao et al., 2020] and STS-GCN [Sofianos et al., 2021]. Meanwhile, to verify the effectiveness of our proposed constrained learning approach, we also compare with SBR [Diligenti et al., 2017], INP [Yang et al., 2020] and MT [Detassis et al., 2020]. For the compared constrained methods, they have degradation in the prediction of joint positions, which is a common phenomenon for constrained problems. Since we can learn the constrained feasible solutions in an end-to-end manner with backpropagation, our proposed method can further reduce the prediction error, demonstrating the effectiveness of our method. Except for the superior performance on the prediction error, the proposed method has two additional benefits. First, unlike SBR and INP, which may not strictly comply with the constraints, our method strictly follows the constraints in the inference. Second, unlike INP and MT, our method does not need extra iterative steps in the inference stage, showing the efficiency of our method.

AMASS Similar to Human3.6M, we conduct experiments on AMASS for all the comparison methods. The results are shown in the left part of Table 2. We have achieved competitive results compared with the baseline methods like ConvSeq2Seq, LTD, RNN-GCN, and STS-GCN. Though the improvement is not significant, our solution can generate temporal constrained feasible solutions without iterative steps. Besides, the proposed method also outperforms the constraintbased approaches such as SBR, INP and MT, demonstrating the effectiveness of our subspace learning strategy.

3DPW To test the generalizability of our proposed method, we use AMASS dataset for training and the 3DPW dataset for testing, as shown in the right part of Table 2. Since AMASS is an extremely large-scale dataset that contains many sub-datasets, the generalizability among different approaches does not have a significant difference. With our temporal constrained feasible subspace learning, we can still achieve competitive results compared with the baseline methods like ConvSeq2Seq, LTD, RNN-GCN, and STS-GCN. Similarly, the proposed method also outperforms the other constraint-based approaches, demonstrating the generalizability of our proposed method.

Comparison on infeasible rate (IR) We show the percentage of infeasible solutions that do not satisfy the temporal constraint on Human3.6M testing set in Table 3. Since our proposed method strictly follows the temporal constraint in the inference stage, the IR keeps 0% in all experiments. Compared with the baseline method STS-GCN, our method is over 3% better on 1000 ms forecasting.

Comparison on the predicted distributions We show the distributions of predicted X′′ f and the ground truth X′′ f on Human3.6M testing set in Figure 3. The top sub-figure shows the comparison between distributions of the ground truth and STS-GCN predictions. The bottom sub-figure shows the comparison between distributions of the ground truth and our constrained predictions. It is obvious that our constrained predictions are much closer to the ground truth distribution.

4.4 Qualitative Results

Qualitative results of the predicted poses We show some qualitative results of the predicted pose on Human3.6M in Figure 4. Each subfigure represents the predicted pose along with the time. Black, red and blue colors represent the ground truth pose, the predicted pose by STS-GCN, and the predicted pose by our method. From the results, the generated poses by our proposed method are closer to the ground truth compared with the baseline method, demonstrating the effectiveness of our method.

Examples of the temporal constraints on joints To verify the effectiveness of our temporal constrained solutions, we visualize the change of velocities of each joint of the predicted poses in Figure 5. We calculate the change of velocity, i.e., X′′ f , on the predicted pose along x, y, z and then draw arrows on the joints, where the arrow direction represents the direction of X′′ f and the arrow length represents the magnitude of X′′ f . The first and second row shows pose examples predicted by STS-GCN and our method, respectively. Obviously, the predictions by STS-GCN have many abrupt changes that do not obey the temporal constrained rules. In contrast to STS-GCN, our method can generate poses with only subtle changes in the velocities, which are closer to real-world movement, showing the effectiveness of our constrained solution.

4.5 Ablation Study

Regularization loss functions As defined in Eq. (5), we consider several regularization terms Rw in the training. The regularization terms include the MSE of the change of velocities with respect to the ground truth, Lv; and the regularization on the change of velocities, Lr. These regularization losses are defined as follows,

Note that the prediction in the inference stage is defined as ˆXf = H-1 (P(H(Fˆw(Xp)))), where the temporal feasibility is not affected by the regularization terms. The results with different combinations of the regularization terms training on the Human3.6M dataset are shown in Table 4. The weight λ in Eq. (5) is set to 1 by default. For each column in the table, ✓ mark represents the adopted regularization term in training. For the rows of Prelu and Pelu, the best performance is shown in bold. For both the projection functions Prelu and Pelu, smaller errors are achieved when Lv and Lr are both used as the regularization, which shows the regularization in the constrained subspace can improve the learning performance.

Projection functions Many projection functions can be applied as long as they are differentiable in the training stage. Except for the projection function described in Eq. (8), we also consider the function constrained by the exponential lin-

where ELU(x; α) = max(0, x) + min(0, α(exp(x) - 1)) and α is set to 1 by default. To verify the effectiveness of different projection functions, i.e., Prelu and Pelu, we show the results in Table 4, along with the combination of different regularization losses. The results that Pelu achieves worse performance than Prelu for each loss combination are reasonable. When the target output of Pelu is touching the bound of the constraint, it will push the input to positive infinity or negative infinity in the back-propagation, causing instability in training and degradation on the performance.

5 Conclusion

In this paper, we propose a temporal constrained feasible subspace learning for the human pose forecasting task. The temporal constrained subspace is explicitly formulated with projection operations. Then the final prediction is obtained by an inverse transformation. The proposed method can guarantee feasible solutions, and no iterative steps are required in the inference stage. The efficiency and effectiveness are verified on the Human3.6M, AMASS, and 3DPW datasets. Ablation studies show the importance of different projection functions and regularization terms. We hope this research can open up more exploration of real-world constrained problems.

Limitations and future work Though good performance has been achieved for the proposed feasible solutions, there are a few limitations. First, we only consider the temporal constraints in the current work. For human pose models, there are also spatial constraints among different joints. Second, only human pose forecasting problems are considered in the current work. We plan to consider spatial constraints and extend our method to other tasks in future work.

Acknowledgments

This work is supported by the National Key R&D Program of China under Grant (2022ZD0162000), the National Natural Science Foundation of China (62106219), and the Fundamental Research Funds for the Central Universities (2262023-00045, 2021FZZX001-23, 226-2023-00048).