MotionMixer: MLP-based 3D Human Body Pose Forecasting

1 Introduction

Forecasting 3D human motion is at the core of many different applications ranging from virtual reality to autonomous driving [Wiederer et al., 2020] and robotics [Gui et al., 2018]. Fundamentally, the task of human motion prediction is defined as the prediction of future body poses from past ones. To model the human spatial-temporal dynamics, classic approaches adopted Hidden Markov Models [Kuli´c et al., 2012] or Gaussian Processes [Wang et al., 2007]. Despite the tangible progress in predicting periodic motion, these approaches impose strong assumptions on body pose, leading to performance degradation. Furthermore, it is difficult for these methods to forecast reliable 3D poses because of the complicated bio-mechanical kinematics. Even manually imposing expert knowledge does not help prior-based approaches to generalise to new environments and subjects.

∗Contact Author

†Most of this work was done while Vasileios Belagiannis was with Ulm University.

Recently, the availability of large-scale datasets, e.g. Human3.6M [Ionescu et al., 2013], AMASS [Mahmood et al., 2019] or 3DPW [von Marcard et al., 2018], the development of human pose estimation algorithms [Belagiannis et al., 2014; Bouazizi et al., 2021] and the advent of deep learning methods pushed the evolution towards forecasting future 3D poses with less priors. Several learning-based approaches were proposed to tackle the problem of 3D human motion prediction. Methods like [Fragkiadaki et al., 2015; Martinez et al., 2017; Tang et al., 2018] build upon the success of recurrent neural networks (RNNs) to better model the temporal correlation between the human body joints. Nevertheless, the use of RNNs come with many drawbacks, including vanishing or exploding gradients [Fragkiadaki et al., 2015], first frame discontinuity, problems in processing longer input sequences as well as shorter forecast horizons leading to stationary predictions or frozen motion [Martinez et al., 2017]. More recent works [Li et al., 2018; Sofianos et al., 2021] propose to replace RNN models with dedicated temporal convolutional architectures. In contrast to RNNs, Convolutional Neural Networks (CNNs) maintain a hierarchical structure, enabling them to capture both spatial and temporal correlations effectively. Lately, graph convolutional networks (GCNs) have received increasing attention. Several works [Mao et al., 2019; Mao et al., 2020; Sofianos et al., 2021; Liu et al., 2021] attempted to utilize GCNs to learn fine-grained spatial relationships among joints. [Mao et al., 2019] for instance, encoded the joints history in the frequency domain and proposed a GCN with learnable connectivity to predict the future motion. Although effective, these methods still need structural priors [Aksan et al., 2020] or frequency transformation [Mao et al., 2019] to address the inherent spatial-temporal dependency of the human motion.

While RNNs, CNNs and GCNs led to a significant performance gain in forecasting 3D human body poses, the existing methods are unnecessarily complex. In this paper, we present MotionMixer, the first model using exclusively MLPs to address the inherent problems of human motion. Equipped with a simple, yet effective architecture, the model aims to learn the spatial-temporal dependencies of the human body pose. Inspired by [Tolstikhin et al., 2021], we propose two types of layers: one with MLPs applied independently to time steps (i.e. "mixing" the temporal information) and another with MLPs applied across body poses (i.e. "mixing" the spatial information). The interchangeable spatial and temporal mixing operations allow the model to access current and past information directly and capture both the structural and the temporal dependencies explicitly.

Our contributions are summarized as follows: 1) We propose to jointly model the spatial locations of the body joints and their temporal dependency with a spatial-temporal MLP. To the best of our knowledge, MotionMixer is the first 3D body pose forecasting approach based solely on MLPs. 2) We design an efficient architecture, that significantly reduces the computational cost of the pose forecasting model. 3) An extensive evaluation on three challenging large-scale datasets demonstrates state-of-the-art performance for short-term and long-term motion prediction.

2 RelatedWork

Recurrent-based Motion Prediction. 3D human motion prediction with RNNs has been widely studied in the last years. [Fragkiadaki et al., 2015] proposed a recurrent encoder-decoder model, which incorporates nonlinear encoder and decoder networks before and after recurrent layers. A curriculum learning strategy was adopted to prevent error accumulation during the training. To better model the long-term temporal dependency, [Martinez et al., 2017] incorporated a residual connection between the RNN units. To make reliable future predictions, [Tang et al., 2018] proposed a motion context modeling by summarizing the historical human motion with respect to the current prediction within a recurrent prediction framework. Though these methods have also incorporated different modules into RNNs, exploring a recurrent-free backbone to address human motion prediction tasks is rarely studied.

Convolutional-based Motion Prediction. Thanks to their hierarchical structure and their effectiveness in capturing spatial and temporal correlations, there has been recently a great interest in integrating CNNs in human motion prediction. [Li et al., 2018] for instance, encoded the history motion into a long-term hidden variable, which is used with a decoder to predict the future sequence. The decoder itself also has an encoder-decoder structure, with a short-term encoder and a long-term decoder. [Sofianos et al., 2021] proposed a spacetime-separable GCN, where the space-time graph connectivity is factored into space and time affinity matrices. [Mao et al., 2019] designed a fully connected GCN to adaptively learn the spatial connectivity of the human skeletons and converted the joint trajectory to the frequency domain to handle the temporal information. [Dang et al., 2021] proposed a multi-scale residual graph network with descending and ascending GCNs to extract features in both fine-to-coarse and coarse-to-fine manners. Despite the advantages in capturing long-range temporal correlations, the quite high computational cost of convolution-based approaches remains a bottleneck. Unlike these approaches, we propose an MLP-based model with lower computational complexity, that better exploits the spatial-temporal dependencies of the body pose.

Attention- and MLP-based Architectures. Inspired by the success of the self-attention mechanism [Vaswani et al., 2017] in natural language processing, many works have explored its application in human motion prediction. [Mao et al., 2020] proposed to capture the similarity between the current motion context and the historical motion sub-sequences in the frequency domain with the attention mechanism. [Aksan et al., 2020] proposed to autoregressively learn spatialtemporal representations with decoupled temporal and spatial self-attention. The key role of self-attention is to re-weight the relative importance of each pose in the sequence with respect to all other poses. This resulted in a high computational memory overhead with increasing number of history poses. On the other end of the spectrum, there have been new works that support replacing self-attention with MLPs. MLP-Mixer [Tolstikhin et al., 2021] for instance, relaxed the quadratically increasing computational memory by replacing the selfattention module with a two-layer MLP. The idea behind the Mixer architecture is to learn to separate the per-location operations and cross-location operations, allowing communication between different image patches. With a design based solely on MLPs, the model was originally developed for visual recognition tasks. However, unlike image classification, where only spatial correlations exist, there exist complicated spatial-temporal dynamics in human motion. In this work, we delve deeper and propose a new architecture based on MLPs to learn the spatial-temporal dependencies of the human body. Unlike the above-discussed approaches, we show that MLPs are effective in learning human dynamics.

3 Method

We define the human body motion as a sequence of Th + Tf consecutive frames, where each frame parameterizes the angles or 3D coordinates of the human body joints. Let X1:Th = {x1, x2, . . . , xt, . . . xTh } ∈ R3×J×Th be the historical motion sequence until the current time step Th, with the 3D body pose xt ∈ R3×J and J is the number of body keypoints. Our goal is to learn the mapping that bridges the history sequence X1:Th to the future sequence XTh+1:Th+Tf = {xTh+1, xTh+2, . . . , xTh+t, . . . xTh+Tf } ∈ R3×J×Tf with a pure MLP-based network. Below, we provide the details of our network architecture.

3.1 MotionMixer

MotionMixer is a sequence to sequence model with mainly three modules: pose embedding, spatial-temporal mixing, and pose prediction, as illustrated in Fig. 2. The pose embedding and the spatial-temporal mixing are coupled together to encode the spatial-temporal dependencies of the human body joints. The pose prediction module, which consists of two fully-connected layers decodes the future 3D motion. Given the historical sequence, each pose is first embedded by a fully-connected layer and given to repeated N STMixer blocks, each of which includes two MLPs with skip connections. The interaction of the body joints over time is modelled by two mixing operations within a single spatial-temporal MLP. The spatial-mixing allows the interplay between the spatial location of the joints, whereas the temporal-mixing allows the long-range interactions of the observed motion. In the pose prediction module, the outputs of the mixing are finally aggregated into a global vector and fed to an MLP to forecast the future motion. Below, we describe each module in detail.

Pose Embedding

Given the observed motion sequence X1:Th , the skeleton of each time step xt is flattened into a vector of length K = 3×J. This yields a two-dimensional tensor X1:Th ∈ RTh×K with one temporal dimension Th and one spatial dimension K. For simplicity, we omit the subscript Th, thus replacing X1:Th with X. The flattened sequence X is then processed by a learnable embedding, which linearly projects each body skeleton xt ∈ RK through a single fully-connected layer to the hidden dimension C. We refer to the output of the learnable pose embedding Y = {y1, . . . , yt, . . . yTh } ∈ RC×Th as:

where W0 ∈ RC×Th×K and b0 ∈ RC×Th are weights of the fully-connected layer.

Spatial-Temporal Mixing

The proposed mixing module is motivated by the fact that the human spatial-temporal dynamics are contiguous. The spatial-temporal mixing stacks N STMixer blocks of identical size and structure. Each block, as shown in Fig. 2 includes two types of MLP layers: spatial-MLP and temporalMLP, each followed by a squeeze-and-excitation block [Hu et al., 2018]. The spatial-mixing aims to learn fine-grained spatial dependencies between the body joints by acting on the columns of the pose embedding Y. Each column encodes the spatial information of one timestep. The spatial-mixing operation can be written as follows:

where W1 ∈ RC×C , W2 ∈ RC×C , σ(·) is a GELU activation function [Hendrycks and Gimpel, 2016] and LN(·) denotes the layer normalization [Ba et al., 2016]. Driven by the fact that the human body joints contribute unequally to the forecasted motion, mixing the columns of Y allows the communication between different spatial pose embeddings. To enable the interchanging between the spatial and temporal domains, the spatial-mixed skeleton features ˆY are transposed and fed to the temporal-mixing MLP. The temporal-mixing MLP, by acting on rows of ˆY aims to learn the temporal correlation of the spatial-mixed features. The temporal-mixing operation is shared across all rows, which encode the temporal information of one body joint. Formally, this can be written as follows:

where W3 ∈ RTh×Th , W4 ∈ RTh×Th . Each linear operator of the temporal-mixing assigns each time step as a linear combination of all frames where the linear weights depend on the frame's location. As such, the temporal information can be maintained in each mixing step, allowing the model to capture long-term dependencies by applying long-range interactions between frames.

In the motion prediction, each time step has a different contribution that is not known in advance. We introduce a squeeze-and-excitation (SE) block [Hu et al., 2018] into STMixer to automatically regulate the input importance. The SE block, as shown in Fig. 2 is added after each mixing operation helping the network to re-weight the influence of each time step. Formally, this is defined as:

where δ(·) and σR(·) are respectively the Softmax and ReLU activation functions. The weights Ws ∈ Rs×e and We ∈ Re×s are shared across the spatial and temporal mixing units. After the mixing operation, an MLP-based pose prediction learns to generate the future human motion.

Pose Prediction

Let ˜Y be the output features of the spatial-temporal mixing. An MLP-based decoder further propagates the mixed features for future pose forecasting. Each ˜yt of ˜Y is projected to a vector of length Tf based on a two-layer non-linear feedforward network. The computation of the predicted body poses ˆXTh+1:Th+Tf is described as:

where Wp1 ∈ RC×Tf , Wp2 ∈ R3×J×Tf and bp2 ∈ R3×J , bp2 ∈ RC are the weights of the fully-connected layers.

3.2 Training

Each joint is represented by the position displacement between two adjacent frames. We train our model by predicting future displacements, which are then added to the most recent pose to get the full-body pose sequence. The loss in terms of Mean Per Joint Position Error (MPJPE) is given by:

with ∆ˆxt,j and ∆xt,j denoting the predicted and groundtruth displacement of a joint j between two adjacent frames. || · ||2 indicates the ℓ2 norm. For the angle-based representation, the loss between the predicted joint angles and the ground truth in the exponential map representation is given by:

where ˆxt,j denotes the predicted angle of the joint j at frame t and xt,j the corresponding ground-truth.

4 Experimental Evaluation

We evaluate our model on three large-scale public benchmarks. Below, we first introduce the datasets, the evaluation metrics and the baselines we compare with. We then present our results using 3D coordinates and joint angles.

4.1 Datasets andMetrics

Human3.6M. [Ionescu et al., 2013] consists of 7 actors performing 15 different actions. The original data is transformed from exponential map to 3D joint coordinates. We consider 22 joints for forecasting the 3D body poses and 16 for the angle-based prediction. Following [Sofianos et al., 2021; Mao et al., 2020], we use the subject (S11) for validation, (S5) for testing, and the rest of the subjects for training.

AMASS. [Mahmood et al., 2019] is a recently published dataset, which consists of 40 subjects performing the action of walking. Following [Sofianos et al., 2021; Mao et al., 2021], we select 8 datasets for training, 4 for validation, and one (BMLrub) as the test set. For each body pose, we consider 18 joints.

3DPW. The 3D Pose in the Wild dataset [von Marcard et al., 2018] consists of video sequences acquired by a moving phone camera. Overall, it contains 51,000 frames of indoor and outdoor actions captured at 30Hz. We use the official test set to test the generalization of a model trained on AMASS.

Metrics. Following the standard evaluation protocol [Li et al., 2018; Mao et al., 2020; Sofianos et al., 2021], we report the euclidean distance between the predicted and groundtruth joint angles. Due to the inherent ambiguity of the Eulerangle representation [Mao et al., 2019; Mao et al., 2020], we further report results in terms of 3D error. We make use of the Mean Per Joint Position Error (MPJPE) in millimeters. We provide the results at the particular frame, as well as the average over all frames following [Sofianos et al., 2021; Zhong et al., 2022]. For the particular frame evaluation, the MPJPE is measured between the predicted pose sequence and the corresponding ground truth at each frame, whereas for the average frame evaluation, the errors in all previous frames w.r.t. a considered one are computed and then averaged.

4.2 Implementation Details

MotionMixer contains three STMixer blocks with C = 60 channels. In each MLP block, a dropout layer with a rate of 0.1 is added to prevent overfitting. We use Adam [Kingma and Ba, 2014] as the optimizer. During training, the learning rate is set to 10-2 and decayed by a factor of 0.1 every 10 epochs. We train our model for 50 epochs with a batch size of 50 for Human3.6M and 256 for AMASS.

4.3 Baselines

Following previous works [Martinez et al., 2017; Li et al., 2018; Mao et al., 2019], we quantitatively evaluate our proposed model on all kinds of actions against the state-of-the-art for 400ms short-term (i.e., 10 frames) and 1000ms long-term (i.e., 25 frames) predictions. We include nine methods with recurrent [Martinez et al., 2017; Mao et al., 2020], convolutional [Li et al., 2018; Tang et al., 2018], graph-convolutional [Mao et al., 2019; Sofianos et al., 2021; Dang et al., 2021; Zhong et al., 2022], and attention-based architectures [Mao et al., 2021] in our comparison .

4.4 Results

Human3.6M. In Tab. 1, we provide the results for each activity of the Human3.6M dataset using 3D body poses. MotionMixer outperforms all previous methods for short-term and long-term prediction in the average-frame evaluation. In particular, we outperform the current best-performing approach [Zhong et al., 2022] by at least 1.3% over all time horizons. The gain over the second-best approach [Sofianos et al., 2021] ranges from 13% in the case of 400ms, up to 5% for 1000ms. In the particular frame evaluation, we reach very competitive results to the state-of-the-art approaches. Only [Mao et al., 2021] outperforms MotionMixer in longterm prediction. Our method, however, yields larger improvements on activities with more complex dynamics such as WalkingDog or Purchase. Also on highly aperiodic actions like Posing, our model still produces accurate predictions.

Fig. 1 illustrates the future predictions of the Posing action. The predicted skeletons accurately match the ground-truth body poses. This demonstrates the effectiveness of spatialtemporal mixing in learning fine-grained motion patterns. In Tab. 2, we additionally provide the results over all actions using respectively the 3D body poses and the joint angles. Despite the inherent ambiguity of the angle-based representation, our method outperforms the compared methods in shortterm and yields the lowest angle error of 0.2 at 80ms. In longterm prediction, we reach comparable results with previous approaches.

AMASS&3DPW. The results of short-term and long-term prediction in 3D on AMASS and 3DPW are shown in Tab. 3. MotionMixer gets the best average error at all short-term forecast times on the AMASS dataset. For long-term prediction, our method consistently outperforms all previous approaches. The performance gain ranges from 4% for 1000ms up to 36% for 80ms, which further shows the benefits of the proposed spatial-temporal mixing. We further test the generalization of a model trained on AMASS on 3DPW. Without any finetuning, our approach outperforms previous approaches at different forecast times and can therefore better generalize to complex outdoor environments.

4.5 Ablation Studies

Model Architecture. We first study the influence of individual components of the proposed method through different ablation studies. Specifically, we report the impact of the number of layers N on the motion prediction. In Fig. 3, we show the average prediction errors at different forecast times on the Human3.6M dataset. MotionMixer yields the best average error with N = 3. Stacking more than three layers did not empirically improve the performance. To verify the impact of predicting the pose displacement instead of the 3D

body pose, we train our model directly with 3D joints. With a 5mm performance gain, our model takes advantage of the pose displacement representation. This is reasonable, since such transformation may help the network focus more on motion patterns rather than the appearance of the body pose, hence, generalizing better to new environments and subjects.

Spatial-TemporalMixing. To demonstrate the effect of the STMixer, we train the Spatial-Mix MLP and the TemporalMix MLP independently for short-term and long-term prediction and report the results in Tab. 4. By removing the temporal or the spatial mixing, the error increases at 1000ms by 4% and 6%, respectively. The best results are achieved when simultaneously mixing the body pose in time and space. This is expected since the human spatial-temporal dynamics are interleaved. We also empirically evaluate the effect of the squeeze-and-excitation (SE) blocks, which shed new light on LSTMs hidden-state weighting by giving higher importance to influential time steps. With a 2mm performance gain over all time horizons, the SE-blocks help the network re-calibrate the influence of each pose in the sequence and predict more accurate motion patterns.

Computational Complexity. We also evaluate the tradeoff between the model's computational cost and performance. The results are shown in Tab. 5. We report the number of parameters and an estimate of the floating operations FLOPs to predict 25 frames (1000ms). We compare our model with the current best approaches. In comparison to [Mao et al., 2021], MotionMixer reaches nearly the same performance with only 1.4% of the parameters. We outperform [Sofianos et al., 2021] and [Dang et al., 2021] respectively by 4% and 3%, while using only 40% and 0.5% of the parameters.

4.6 Limitations

In addition to the qualitative results in Fig.1, we examine some failure cases of MotionMixer. Fig. 5 illustrates an example of the predicted skeletons for the WalkDog action. As can be seen, the last three frames do not match the groundtruth poses. This failure is also common for previous methods [Mao et al., 2021; Sofianos et al., 2021] since various actions in Human3.6M are performed in different arts in the training dataset. In addition, the human motion is highly uncertain. A sequence of past poses may imply various possible futures. Thus, predicting the inter-joint and inter-frame dependencies in long-term becomes even more complex.

5 Conclusion

In this work, we presented an MLP-based pose forecasting approach that effectively exploits the spatial-temporal dependencies of the 3D human body pose. By learning to mix features across the spatial and temporal domains, our method improved the state-of-the-art for short-term and long-term forecasting on three large-scale benchmark datasets. Enhanced by squeeze-and-excitation (SE) blocks, which aim to calibrate the influence of each time step in the pose sequence, our model has much less parameters than current best-performing approaches.

Acknowledgments

The research leading to these results is funded by the German Federal Ministry for Economic Affairs and Energy within the project "KI Delta Learning" (F¨orderkennzeichen 19A19013A). The authors would like to thank the consortium for the successful cooperation.