Few-shot Human Motion Prediction via Learning Novel Motion Dynamics

1 Introduction

Human motion prediction is the task of forecasting future human pose based on the observed pose data. It is one of the hallmarks of human intelligence, which has a wide range of applications such as autonomous driving [Behl et al., 2017], motion simulation [Vondrak et al., 2008], and human-robot interaction [Koppula and Saxena, 2015]. Taking humanrobot interaction as an example, robots are supposed to greet us immediately once we are raising our hand to greet them, to help us take the cup once we are walking to get it, and to stop once their next motion may hurt others.

In contrast to human intelligence which acquires such a prediction capability from just a few experiences, existing methods for human motion prediction [Fragkiadaki et al., 2015; Martinez et al., 2017; Liu et al., 2019] still rely on

∗This work was completed while Chuanqi Zang was at Rochester Institute of Technology as a visiting student.

̂

Figure 1: Overview of our few-shot human motion prediction architecture (MoPredNet). Future poses are predicted from observed poses. There are three modules in our architecture: DSTCN is used for motion generation. Sequence mask (of length K) is used to recall referable motion to get the explicit guidance for prediction. The parameter generation is used for quickly updating parameters of the DSTCN.

extensive training data in order to be adapted to novel motion dynamics. This limits their applications in few-shot task domain. In addition, even though these methods continuously improve the performance in motion prediction on public datasets, their prediction capability is limited in long-term prediction [Martinez et al., 2017] mainly because they do not explicitly learn sub-motion patterns and do not receive explicit guidance from past related motion. Here, sub-motions represent the decomposition of complex motion, e.g., raising, holding, and lowering hands are sub-motions of eating.

To deal with these issues, we propose a Motion Prediction Network (MoPredNet) that can dynamically select the most informative poses as prediction reference and adaptively learn spatio-temporal motion structure for long-term motion prediction. Besides, MoPredNet has the ability to predict new motion categories based on accumulated information and novel information from a few new samples. As shown in Fig. 1, our MoPredNet can be divided into three modules: Deformable Spatio-Temporal Convolution Network (DSTCN), a sequence mask module and a parameter generation module.

The DSTCN is proposed to adaptively model sub-motion dynamics and spatial correlation in the entire motion sequence to capture long-term dependency. In the spatial domain, human's skeletal joints are directly related to their parent joints but indirectly related to other joints (symmetry or regularity in motion). For traditional convolution, it is hard to model sub-motion patterns in a suitable starting and ending frames and flexibly capture spatial correlation based on motion dynamics. The DSTCN can adaptively extend in the past time and establish selective connections with its neighboring joints. Besides, because the joint is embedded with highdimensional representation compared to pixels, we present a local geometric extraction layer before the deformable spatiotemporal convolution layer.

The sequence mask module is used to recall a referable motion state from history, which provides explicit guidance for long-term motion prediction at specific moments. Many researchers extract the guidance information of motion dynamics from different cues, such as speed [Martinez et al., 2017], incorporating derivative [Gopalakrishnan et al., 2019], and smooth trajectory [Wei et al., 2019]. However, a single physical property does not explicitly guide the architecture to generate the next moment pose as prediction errors accumulate. In this work, we use the attention mechanism to calculate the correlation scores between the past poses and the current pose, which are then used to pick key frames. This information combined with observed motion information helps the prediction module generate human-like motion.

The parameter generation module is inspired by the human learning process. When facing a new motion category with a few samples, we analyze and predict motion using accumulated information from an external memory at first. Then we will pay attention to novel motion dynamics to acquire new knowledge. In the parameter generation module, we store the basic knowledge learned from large-scale dataset in the memory to instruct the model to generate human-like motion. Besides, we present a learner that can learn novel motion dynamics and quickly update the parameter of the DSTCN. Therefore, the DSTCN can predict motion by a new context vector with unique characteristics for a specific category.

2 RelatedWorks

2.1 HumanMotion Prediction

In early work, hand-craft features [Pavlovic et al., 2001; Wang et al., 2006; Taylor et al., 2007] are proposed to model the motion patterns. However, human motion incorporates far more complicated changing, which is hard to be modeled in a fixed, manually designed model.

With the collection and publication of large-scale human motion datasets in recent years, many methods based on the deep neural network are proposed and achieve promising results. Recurrent Neural Network (RNN) based methods [Fragkiadaki et al., 2015; Martinez et al., 2017; Liu et al., 2019] are popular in motion prediction and encode temporal structure by a hidden state. [Li et al., 2018] presents a temporal convolution seq2seq Network with GAN to generate human-like motion. Recently, [Wei et al., 2019] proposes a feed-forward network based on graph convolutional networks and improves the performance of fixed length motion prediction within 1000 ms.

The prediction capability of most current methods is limited to short-term predictions, mainly because they do not explicitly learn sub-motion patterns and do not dynamically select important motion segmentation as a reference in the past. Facing few-shot new motion categories, these methods, which have been trained with large amounts of annotated

̂

data, will quickly become overfitting. In this paper, we design a MoPredNet for few-shot human motion prediction.

2.2 Few-shot Learning

The goal of the few-shot learning is to learn new concepts with limited samples. Recent work on few-shot learning can be classified into three categories: metric-based [Koch, 2015; Vinyals et al., 2016], gradient-based [Finn et al., 2017], and memory-based [Ravi and Larochelle, 2016; Gidaris and Komodakis, 2018]. In few-shot human motion prediction , [Gui et al., 2018] proposes to combine the model-agnostic metalearning MAML [Finn et al., 2017] and model regression networks (MRN) [Wang et al., 2017] to jointly learn generic model initialization and adaptation strategy during the metatraining phase.

Unlike [Gui et al., 2018] that slowly updates parameters, when facing novel features extracted from a new category, we directly update a specific model for this motion category from the external memory. Then instead of just using stored fixed parameters [Gidaris andKomodakis, 2018], we also generate target parameters by learning novel motion dynamics from new motion categories.

3 Approach

In human motion prediction task, we forecast future pose sequence based on the observed pose sequence. In this paper, the pose at each time step is defined by x (i.e., mocap representation for skeleton joints). For observed motion pose, the motion sequence from time 1 to t is expressed as x1:t. The predicted motion sequence from t + 1 to t + T is expressed as ˆxt+1:t+T , which is expected to be close to the ground truth xt+1:t+T . To predict the correct motion of a new category from very few observed samples, as illustrated in Fig. 2, the proposed new architecture consists of three parts. DSTCN (Section 3.1) adaptively captures motion information from observed poses and mask poses (Section 3.2) through encoders and generates new pose through the decoder. In the few-shot motion prediction, the parameter generation module (Section 3.3) generates new parameters by combining information from the memory and generation module, enabling fast adaptation to new motion categories.

3.1 Deformable Spatio-Temporal Convolution

To adaptively model sub-motion dynamics and spatial correlation in the entire motion sequence, inspired by [Dai et al., 2017], we propose a Deformable Spatio-Temporal Convolution Network (DSTCN) based on Temporal Convolution Network (TCN) [Bai et al., 2018]. Same as [Li et al., 2018], the observed skeletons data are represented in a 2D format: temporal sequence domain t and spatial skeleton domain s.

Compared with TCN, our DSTCN can adaptively model sub-motions, and can flexibly capture spatio-temporal structure. We show the first two layers of DSTCN in Fig. 3. The feature Fl+1 ∈ Rns×nt (feature map size: n) of location P0 in l + 1-th layer includes not only original receptive field O information in l layer but also its neighbor's information. The selection of neighbor information for motion modeling is guided by the offset of receptive field ∆P. Then feature Fl+1 (P0) convolves (∗) this deformable information by a 2D convolution kernel Kl ∈ Rms×mt (kernel size: m). We formulate the deformable convolution process as follows:

where Pts = (Pt, Ps). Pt is the temporal dimension and Ps is the spatial dimension. Due to the limitation of the fixed kernel, TCN does not have the offset ∆P . Unlike the original application of deformable convolution using image data [Dai et al., 2017], the DSTCN is designed to fit the temporal data, where the information from the previous moment cannot be leaked to the next moment.

In our work, we use the nonlinear activation function (ReLU) to limit the direction of convolution expansion. We also adaptively expand the receptive field in space to capture the correlation of joints that are not directly connected, e.g., the symmetry of arms. We have the spatio-temporal offset:

However, high-dimensional 3D skeleton data are different from the explicit relationship of adjacent pixels. Adjacent pixels are not coaxial, so it is difficult to extract spatial correlation directly. Therefore, reasonable deformation expansion should occur on the same geometric pixels. In this work, we present a local geometric extraction layer before deformable spatio-temporal convolution layer to encode coordinates data ∆Ps = Pjoint.

The extension of convolution to the past is beneficial for the modeling of periodic and aperiodic sub-motions. The captured motion dynamics in base motion categories can be easily generalized to new motion categories.

3.2 Sequence Mask

Correlated motion sequence is selected in past sequences based on attention mechanism to provide explicit motion information guidance. As shown in Fig. 2, the observed pose and masked pose (masked by M) are encoded by observed encoder fα and masked encoder fm respectively. At each time step, by combining observed motion context vector vo and masked motion context vector vm, a reasonable future motion sequence can be predicted by decoder function fβ. The generation of ˆxt+k can be presented by

where θα, θm, θβ are parameters of deep networks. ⊙ denotes Hadamard product. [x, ˆx] is the observed and generated motion before time t+k. The temporal maskM in this paper is binary which depends on the correlation score ai between past poses and the current pose. The process of selection can be given by

where ai is defined as

Here, δ and K are hyper-parameters that determine the number of reference frames. In summary, different from previous work [Martinez et al., 2017; Gopalakrishnan et al., 2019] in which a single physical property is selected, we enhance correlated motion sequence encoding to provide explicit guidance for long-termmotion prediction.

3.3 Few-shot Learning

Fig. 4 summarizes our new memory-based parameter generation module for the few-shot motion prediction. Following related few-shot work [Gidaris and Komodakis, 2018; Sun et al., 2019], we divide large human motion dataset D into two subcategory datasets: base categories split Dbase = {Di}Cb c=1 of Cb categories and novel categories split Dnovel = {Di}Cn c=1 of Cn categories. In few-shot motion prediction, the training data for a new category are limited. Therefore, we sample a few prediction task T (tr) = ⋃Cb c=1 {xc,s}Ss=1 of S

samples from Dbase and T (ts) = ⋃Cn c=1 {xc,s}S′ s=1 of S′ samples fromDnovel for meta-train and meta-test separately.

In our work, we train a model suitable for a new category of motion prediction in three phases: a pre-training phase, a meta-training phase, and a meta-test phase. In the pretraining phase, to simulate the accumulation of basic human knowledge, we pre-train our motion prediction module on a large dataset with base categories Dbase. Pose encoder (including whole past sequence encoder fα (·; θα), masked sequence encoder fm (·; θm)), and pose decoder fβ (·; θβ) are trained by the loss function LD:

where ▽LD is computed by the L2 loss ℓ2 on frames:

and γ1 denotes the learning rate. In addition to pre-training general parameters [θα; θm; θβ] of the model for all motion categories, the parameters θβ of the prediction decoder are also kept in an external memory with its n-dimensional context vector v = fα (·; θα) ∈ Rn (including general parameters and specific category parameters). So, similar to memory networks [Sukhbaatar et al., 2015], the external memory stores key-value pairs: (V, θ), where V = {vc}Cb+1 c=1 and θ = {θβc}Cb+1 c=1 .

The second phase is the meta-training phase, as shown in Fig. 4. To quickly adapt the motion prediction module to new motion categories, we use the parameter generation module to generate decoder parameters θβ instead of optimizing them slowly [Gui et al., 2018]. A crucial difference between our network and previous work [Gidaris and Komodakis, 2018] is that our proposed parameter generation module not only references a credible parameter set θ in the memory but also relies on novel information in a new motion category. The parameters in the memory have the ability to extract shared features in the motion but cannot capture unique information for new motion. Therefore, we propose a parameter learner to extract novel motion dynamics and generate a novel weight for fg (·; θg), which enables the motion prediction module decode new motion. Combining the base parameters and novel parameters, the p-dimensional target parameters θβ ∗ ∈ Rp are computed as:

where the context vector vnovel is encoded by observed encoder fα (xt+1:t+T ; θα) and is the same as vo in base categories. vnovel ∈ Rn denotes novel feature as a query. h (·, ·, ·) computes the content-based attention scores by normalized cosine similarity and used to weight the basic parameters. Therefore, the Motion Prediction Module will use the updated parameters [θα; θm; θβ ∗ ] of the DSTCN to generate future pose.

For parameter update in the meta-training phase, parameters [θα; θm] of the pose encoder in DSTCN are frozen, and the learnable generated parameter θβ ∗ of the pose decoder in DSTCN is optimized by gradient descent:

where θβ ∗ass denotes the assigned parameters before the optimization and θβ ∗upd denotes the updated parameters after the optimization. θβ ∗ass and θβ ∗upd are used for guiding the optimization of θg:

where loss function LT (tr) is computed by the L2 loss between θβ ∗ass and θβ ∗upd :

In the meta-test phase, we train the parameter generation module on the support data of a new prediction task T (ts) . With the motion of the new category as input, the parameter generation module is adjusted by new training data. To avoid the problem of "catastrophic forgetting" [Lopez-Paz and Ranzato, 2017], the encoder's parameters of the motion generation module are frozen.

Our parameter generation module can quickly update parameters of motion prediction module for new motion categories. The memory module helps generate model parameters for human-like motion, and the parameter learner can help generate model parameters for decoding novel motion.

3.4 Schedule Sampling

There are many training strategies for human motion prediction, including open-loop prediction with noise [Fragkiadaki et al., 2015], closed-loop prediction [Martinez et al., 2017] and fixed sampling [Li et al., 2018]. All these training strategies focus on the challenging error accumulation problem, which ignores the model's convergence efficiency and convergence effect. Inspired by [Bengio et al., 2015], we use scheduled sampling to balance the convergence and generalization during the training phase. Let f denote the entire prediction function, including encoding function (fα, fβ) and decoding function (fθ). We have

where ϕ decays after each series of iterations I (or epoch), like the learning rate:

Here, s determines the decay speed of the function, with the range of [0, 1]. Hence, in the early stage of training, the network can quickly converge relying on the guidance of ground truth. As the training proceeds, the network can generate plausible motion based on motion model and dynamics. The network then reduces the sampling rate to focus on error accumulation and improves its generalization ability.

4 Experiments

We evaluate the effectiveness of our MoPredNet for few-shot motion prediction on two popular human motion datasets:1) Human 3.6M dataset [Ionescu et al., 2013] and 2) CMU MOCAP. We follow [Martinez et al., 2017] on Human 3.6M dataset, where the sequence of subject 5 is selected for test (same in meta-learning and meta-test) and others are selected for training. Besides few-shot motion prediction, we also report the performance of the sequence mask and DSTCN module on long-term prediction using Human 3.6M dataset. Similar to [Li et al., 2018], 8 representative motions in CMU MOCAP are used and split into training and test set.

4.1 Experimental Setup

In our Motion Prediction Network (MoPredNet), pose encoder consists of whole past sequence encoder and masked motion sequence encoder with the longest mask K set as 20 and mask threshold γ set as 0.0666. They share the parameters of the convolution layer in the encoder, including 1 convolution layer with 1×3 convolutions for joints coding and 3 deformation convolution layers with 2 × 3 convolutions, followed by leaky ReLU nonlinearity. The feature channels of each convolution layer are set as 16, 64, 128, 256 respectively. After that, the feature is encoded to a 512-dimensional vector by a fully-connected layer. The pose decoder network contains three fully-connected layers with the size of 512, 128, and 54, respectively. Leaky ReLU action function and drop out are both set as 0.5 in the first two layers. For the parameter learner in the parameter generation network, it contains two fully-connected layers. We adopt the ADAM optimizer with the initial learning rate set as γ1 = γ2 = 1e-4, γ3 = 1e-8.

The initial sampling rate and decayed rate are set as 0.8 and 0.7, respectively.

In order to fairly evaluate previous work, we report experimental results by implementing their released code with their original setting. The input window is set as 50 frames (2s), and the output window is set to 25 frames (1s) for training (GCNs' output window is set to 100 frames as it required). It is noteworthy to mention that both in few-shot motion prediction and long-term motion prediction, we do not use any supervised label for motion prediction.

4.2 Evaluation on Human 3.6MDataset

We first evaluate our MoPredNet architecture for few-shot human motion prediction on Human 3.6M dataset. Following previous work [Gui et al., 2018], we sample 6 motion sequences (5 sequences for training, 1 sequence for test) from each of the 11 categories (except walking, eating, smoking and discussion) as base prediction task T (tr) during metatraining phase. The remaining categories of motion are used as novel prediction task T (ts) with its small training set and test set like T (tr) . The performance of our meta-learning architecture is shown in Table 1. Obviously, our method outperforms PAML [Gui et al., 2018] almost all the time in all novel motion categories, which proves that encoding novel motion dynamics is beneficial for few-shot motion prediction.

We construct two baseline methods (MoPredNet without parameter generator module): Single means training a single model with few samples on every specific motion category, and Transfer means training one model for all basic motion categories with large data and fine-tuning new motion categories with few samples. Compared with baseline methods, we can see that our meta-learning method improves the performance of our MoPredNet on novel motion categories.

We also apply the DSTCN and the sequence mask module to long-term motion prediction (4000 milliseconds) [Gopalakrishnan et al., 2019] and compare with the state-ofthe-art methods. Table 3 shows the Mean Angle Error (MAE) results on the Euler angles on the Human 3.6M dataset. From the experiment results, we can see that our architecture achieves the best prediction results. Especially at 560ms, 1000ms and 2000ms, our errors are 0.07, 0.08 and 0.1 lower than HMR results respectively.

The qualitative results are illustrated in Fig. 5, which shows the long-term human motion prediction over 4000ms for two actions: walking and walking together. We can see that poses predicted by our MoPredNet are closed to ground truth in short-term prediction and more like human motion. Take walking as an example, our generated pose sequence is consistent with ground truth before 1 second. After that, our MoPredNet hardly holds the same motion frequency with ground truth, but can still capture the motion direction changes.

4.3 Evaluation on CMUMOCAP Dataset

Using similar architecture settings for Human 3.6M, we report our results on the CMU MOCAP dataset in terms of angle errors in Table 2. The CMU MOCAP dataset is divided into base categories and novel categories. The base categories include walking, jumping, basketball signal, and directing traffic. The novel categories include running, soccer, basketball, and wash window. The experiment results show that our method outperforms other baselines in all the new categories.

4.4 Ablation Study

We use the parameter learner to update the parameters of decoder according to novel motion dynamics beyond the memory combination. We remove it from our MoPredNet to verify its effectiveness as shown in Table 1. The results show that using the parameter learner provides a significant boost in performance. We also show the influence of the sequence mask module, the DSTCN module, and the local geometric extraction layer in Table 3. The def.cnn in Table 3 indicates DSTCN without local geometric extraction layer before the deformable convolution layer. Results show that the sequence mask, the DSTCN and the local geometric extraction layer all contribute to the improvement of the performance.

In the training stage, we use schedule sampling to cope with the error accumulation problem. We compare the performance in Human 3.6M dataset using schedule sampling with open-loop prediction, closed-loop prediction, and fixed sampling in Table 4. We can see that using schedule sampling can converge to a better local optimal solution.

5 Conclusion

In this paper, we propose MoPredNet for few-shot motion prediction. In addition to the base knowledge accumulated through base motion categories, MoPredNet can capture novel motion dynamics from a few new motion samples. This improves its performance in new motion categories. MoPredNet can select the correlated motion frames of past motion to provide explicit guidance for long-term pose prediction. In addition, the MoPredNet can adaptively capture past motion spatio-temporal structure to improve its encoding capability. Experiment results demonstrate that our method achieves state-of-the-art performance in human motion prediction.