ReliaAvatar: A Robust Real-Time Avatar Animator with Integrated Motion Prediction

1 Introduction

Virtual reality, augmented reality (AR), and mixed reality (MR) are rapidly evolving fields that offer new dimensions for human communication and interaction. A critical aspect of enhancing these immersive experiences lies in the ability of models to generate seamless and realistic avatar motions, ideally using user-friendly devices such as head-mounted displays (HMD) or non-wearable WiFi devices [Yan et al., 2024]. Despite notable advancements in using sparse observations to animate full-body avatars, a significant research gap exists to effectively drive avatars in scenarios plagued by low-quality signals.

In practical applications, poor-quality signals are a common challenge, particularly when wearable devices are intended to be as convenient as possible. Various factors, including network fluctuations, occlusion in motion capture systems, and limited visibility of interactive handles in HMDs, can degrade signal integrity. Previous works such as AGRoL [Du et al., 2023] and HMD-NeMo [Aliakbarian et al., 2023] have addressed some aspects of these challenges. However, systematic exploration and comprehensive solutions for diverse data-loss scenarios remain underdeveloped.

∗Corresponding author: Xing Wei.

In this work, we delve into potential data-loss scenarios that could occur in real-world applications. We specifically identify and focus on two scenarios: instantaneous data-loss and prolonged data-loss scenarios. Our response to these challenges is two-fold, encompassing both model architecture and training methodology innovations. We introduce Relia- Avatar, a real-time, robust, autoregressive Avatar Animator, integrating full-body joint motion prediction with an innovative training approach that simulates data-loss conditions.

We develop two distinct data-loss scenarios: instantaneous, resembling tracker signal loss like network packet loss, and prolonged, simulating long-last loss of specific motion capture joints, such as a hand-held controller's prolonged invisibility. Our model operates through two pathways: a regression pathway for conventional full-body avatar pose estimation tasks and a prediction pathway that predicts motions in the absence of tracker signals, ensuring continuity in avatar movements. Both pathways leverage GRU-based models for feature extraction. The combined features are then transformed into a decoding token sequence representing 22 SMPL joints, which will be used in a Transformer encoder to model the inter-joint relationships. We also propose an autoregressive training pipeline encompassing three preprocessing methods aligned with standard, instantaneous, and prolonged data-loss scenarios. During training, each signal sequence is processed using one of these preprocessing methods, enabling the model to adapt to the challenges posed by different data loss scenarios.

We compare ReliaAvatar with existing methods in all scenarios on the AMASS benchmark dataset [Mahmood et al., 2019]. The results showcase our model's state-of-the-art performance in both standard and data-loss scenarios. We also provide a comparison with AvatarPoser [Jiang et al., 2022] in Figure 1 in the prolonged data-loss scenario. These qualitative and quantitative results collectively demonstrate that we are the first method, as stated in Table 1, that has robustness to data quality. Our model also demonstrates impressive performance during the online inference stage, achieving a remarkable 109 fps. This outstanding speed surpasses other avatar pose estimation methods, highlighting the superiority of our model in real-time applications.

Our contribution can be summarized as follows:

• We pioneer a comprehensive investigation into practical data-loss scenarios, identifying and focusing on two key scenarios: instantaneous and prolonged data loss.
• We propose ReliaAvatar, a real-time, robust avatar animator with integrated full-body joint motion prediction, and an autoregressive training pipeline tailored for dataloss scenarios. This approach significantly enhances the model's adaptability and performance.
• Our experimental results demonstrate that ReliaAvatar not only achieves top-tier performance in the standard scenario but also represents, to our knowledge, the first method adept at managing various data-loss scenarios effectively. Furthermore, ReliaAvatar outperforms other methods in terms of computational efficiency.

2 RelatedWork

2.1 Full-body Avatar Pose Estimation

In the past, motion capture systems required users to wear numerous devices, as seen in early implementations [Xsens, 2013; Vlasic et al., 2007]. However, this requirement often compromises the immersive experience, highlighting the need for more compact approaches. Consequently, the focus shifted to generating a full-body pose using sparse observations, a topic that has garnered significant attention and led to numerous advancements [Von Marcard et al., 2017; Huang et al., 2018; Yi et al., 2021; Yang et al., 2021; Jiang et al., 2022; Zheng et al., 2023; Du et al., 2023; Aliakbarian et al., 2023]. These innovative methods allow users to wear tracking devices on only a limited number of joints, such as the head, hands, and pelvis, thus improving the overall user experience. Among these developments, AvatarPoser [Jiang et al., 2022] introduced a Transformer-based architecture combined with an inverse kinematics (IK) module, paving the way for more sophisticated avatar control techniques. AGRoL [Du et al., 2023] utilized diffusion models to drive avatars, showing notable robustness against instantaneous data loss. Similarly, HMD-NeMo [Aliakbarian et al., 2023] made strides in avatar control by focusing on scenarios where the hands are partially or entirely obscured. Building on these foundational works, our research delves deeper into the realm of full-body avatar estimation under the conditions of low-quality data.

2.2 3DHumanMotion Prediction

A key innovation of our model is the motion prediction pathway in the architecture, which is used to provide additional cues to avatar pose estimation. There are many motion prediction methods available for reference, ranging from nonlinear Markov models [Lehrmann et al., 2014] and Restricted Boltzmann Machines [Taylor et al., 2006], to more recent developments like Graph Convolutional Networks [Ma et al., 2022; Mao et al., 2020; Mao et al., 2019] and Recurrent Neural Networks [Jain et al., 2016; Liu et al., 2019; Martinez et al., 2017]. In the prediction pathway, we use a GRU module[Cho et al., 2014] to extract features and a Transformer [Vaswani et al., 2017] module to model the interjoint relationships. The prediction pathway can be aggregated with the regression pathway at the joint-relation Transformer to form our reliable avatar animator.

3 Method

3.1 ProblemDefinition

Task. The core objective of our task is to accurately predict the 3D full-body human poses yt at any given time-step t, using sparse joint signals xt and supplementary information et . This supplementary information, et , may include data derived from historical movements or other available motion signals at time-step t. Therefore, our objective is formally defined as follows.

Traditionally, this task is approached as an upsampling challenge, where the goal is to learn the mapping from sparse joint signals to a complete full-body joint configuration. In these conventional methods, inputs are configured as a series of continuous sparse signals over a time window to facilitate the extraction of temporal features, leading to the following model formulation.

Here, the supplementary information et is represented as et = {xi}t-1 i=t-T , with T denoting the window size.

Those approaches neglect the continuity of the prediction process, which has been proven to be crucial in visual tracking[Wei et al., 2023; Bai et al., 2024]. Unlike them, we treat the task as an integrative process of both upsampling and full-body joint motion prediction. To this end, we incorporate historical trajectory states as an additional input variable. We utilize a Gated Recurrent Unit (GRU) to extract temporal features from both the tracker signals and the historical trajectory states. This approach allows us to redefine et as a composite of previous pose predictions and historical hidden states, i.e., et = {ˆ yt-1 , ht-1 x , ht-1 y }. Consequently, our model's formulation is as follows:

Here, htx , hty represent the hidden states generated by the GRU, essential for capturing the complex temporal dynamics of human motion.

Representation. ReliaAvatar's input signals consist of four parts: the tracker signals xt , hidden tracker states ht-1 x , the historical trajectory states ˆ yt-1 , and hidden historical states ht-1 y . Here, ˆ yt-1 represents the output of the model at timestep t - 1, and ht-1 x and ht-1 y are the hidden states returned by the GRU module.

The input tracker signals xt at time-step t are composed of the orientation 1 xr ∈ R|J|×6 , rotation velocity ∆xr ∈ R|J|×6 , position xp ∈ R|J|×3 , and linear velocity ∆xp ∈ R|J|×3 of the trackers. J represents the set of all trackers, ∆xp = xtp-xt-1 p , ∆xr = f6D(R-1 (xt-1 r )R(xtr )), and R and R-1 , respectively, represent converting the 6D representation of rotation into the corresponding rotation matrix and inverse matrix, where f6D represents converting the rotation matrix into a 6D representation. Therefore, xt = {xtr ,∆xtr , xtp ,∆xtp } ∈ R|J|×18 .

The historical trajectory states ˆ yt at time t + 1 are also composed of the four parts: 22 SMPL joints' local rotation relative to their parent joints ˆ ytr ∈ R22×6 , position ˆ ytp ∈ R22×3 and corresponding velocity ∆ˆ ytr ∈ R22×6 and ∆ˆ ytp ∈ R22×3 . Therefore, the overall output can be represented as ˆ yt = {ˆ ytr , ∆ˆ ytr , ˆ ytp , ∆ˆ ytp } ∈ R396 .

Scenarios. Current models often operate under the assumption that tracker signals are consistently available, promptly received, and accurate. However, real-world applications present more complex and varied scenarios in which the quality and consistency of tracker signals cannot always be guaranteed. To address this, we have identified three primary scenarios in which our model and similar models may need to operate:

• Standard scenario: This scenario represents ideal conditions where the model successfully receives all required tracker signals without any loss or degradation. This is the standard operational scenario for most existing methods.

1 The orientation is represented by the 6D representation of rotation [Zhou et al., 2019].

• Instantaneous data-loss scenario: Real-world applications are prone to transient disruptions, such as network fluctuations, leading to the momentary loss of specific tracker signals. To simulate these sudden and brief interruptions in signal transmission in this scenario, we make each tracker signal at any given time t subject to a probability p of being lost, as illustrated in Fig 3.A.
• Prolonged data-loss scenario: There are instances where certain tracker signals may be consistently unavailable over a long-lasting period 2 . Such situations can arise from continuous occlusion of infrared markers or sustained invisibility of hand-held controller signals in an HMD kit. This scenario is characterized by a continuous loss of all tracker signals for a specific duration, as depicted in Fig 3.B. It represents challenges posed by prolonged data unavailability.

The detailed configurations and implications of these scenarios will be further explored in Section 3.4 for the training stage and Section 4.3 for the testing stage, respectively.

3.2 Overview

ReliaAvatar adopts an autoregressive pipeline during both the training and testing stages. The process of ReliaAvatar's handling continuous tracker signal sequences can be represented by the following equations:

...

During the training stage, given a tracker signal sequence {xt }L-1 t=0 as input, the training process is divided into L sequential steps with each step building upon the previous one. L is the length of the sequence.

As depicted in Figure 2, our model features two distinct yet interconnected pathways: the regression pathway (Regression Encoder→Joint-relation Transformer→Decoder) and the prediction pathway (Prediction Encoder→Joint-relation Transformer→Decoder). The regression pathway operates similarly to the traditional avatar animator. Its primary function is to regress full-body poses from sparse joint information. This process can be formulated as:

The prediction pathway, on the other hand, aligns more closely with motion prediction models. It utilizes known sequences of past motion to predict the current full-body motion. The formulation for this pathway is as follows:

Both pathways converge at the Joint-Relation Transformer. This convergence results in the complete model represented by Eq. (3). The Joint-Relation Transformer and the decoder are shared modules utilized by both pathways, while each pathway has its unique components - the regression encoder for the regression pathway and the prediction encoder for the prediction pathway.

2 The term "long-lasting period" refers to a period that is significantly longer than the frame interval, for example, 1 second.

3.3 Model Architecture

Regression Encoder and Prediction Encoder. As shown in Figure 2, tracker signals are initially processed through a sparse fully connected layer (SFC) to extract initial features Et0 ∈ R|J|×d , where d is the embedding dimension of the model. The purpose of SFC is to merge multiple linear layers into a sparse linear layer, thereby reducing the number of loops in the forward process and improving the runtime efficiency. In the regression encoder, SFC is formed by merging four linear layers that handle orientation xr, rotation velocity ∆xr, position xp, and linear velocity ∆xp. This module can be formulated as:

Then, the GRU module will extract the temporal features, as described in the following formula:

After the regression encoder, the features of all tracker signals Etx ∈ R|J|×d are obtained. Subsequently, an upsampling module will be used to increase the dimension to 22×d, thus obtaining the decoding token sequence {tix }21 i=0 which will be input to the Joint-Relation Transformer. Each token represents a SMPL joint.

The prediction encoder is similar to the regression encoder. After passing through an SFC and a GRU module, ˆ yt-1 is mapped to a token Ety ∈ Rd . Then a fully connected layer is applied to expand it to 22 joint tokens {tiy }21 i=0 .

Joint-Relation Transformer. In human kinematics, the motions of the entire body are not simply a combination of independent motions of multiple joints. In contrast, the actions performed by the human body are strongly correlated with the relationship between joints. Therefore, it is evident that equipping the model with the ability to model joint relationships can enhance the driving effectiveness. A Transformer encoder is used to model joint relationships, which has been proven to be effective in PETR [Shi et al., 2022].

The input of the Transformer is a token sequence {ti }21 i=0 = Concat (tx, ty) ∈ R22×(2d) representing 22 SMPL joints, which is fused from the tokens generated by the previous two encoders.

Decoder. The decoder is responsible for decoding rotation ˆ ytrot and position ˆ ytpos directly from the output of the token sequence of the Transformer. The decoder is composed of two layers of MLP. There are three decoder variants:

• SharedDecoder. Since the pelvic joint parameter represents the global orientation of the entire body, while the other joint parameters represent local rotations relative to their parent joints, a dedicated decoder is used for the pelvic joint, while the remaining joints share a common decoder.
• Multi-FC Decoder. Each joint has its own dedicated decoder. During the inference process, the decoding of each joint is done sequentially.

• SFC Decoder. Use an MLP composed of SFCs as a unique decoder. As a result, all joints can be decoded simultaneously in parallel.

We adopted SFC decoder due to its advantages of high accuracy and high efficiency. The comparison of the three decoders will be mentioned in Sec 4.5.

3.4 Simulation Training

We simulate the three scenarios mentioned in Sec during training ReliaAvatar.

• Standard scenario. In the standard scenario, all tracker signals can be fully and correctly transmitted to the model. So in this scenario, there is no need for any masking of the signals.
• Instantaneous data-loss scenario. In the instantaneous data-loss scenario, we use a dropout layer to randomly mask the signals of each tracker in each frame with a probability of 0.1.
• Prolonged data-loss scenario. In the prolonged dataloss scenario, we mask out the latter half of each input sequence {xt}L-1 t=0 . That is to say, {xt}L-1 t=L/2 = 0.

Each sequence has an equal probability (1/3) to undergo the above three simulation treatments.

3.5 Loss Function

We adopt orientation loss Lori, rotation loss Lrot, SMPL position loss L SMPL pos , decoded position loss L dec pos , and velocity loss Lvec during the optimization process. All losses are computed using the L1 loss. Taking the rotation loss as an example, its loss function is as follows:

Here, ˆ yrot, yrot represent the model output and corresponding ground truth, respectively. Other losses are computed in the same way. The whole loss function can be formulated as:

Here, λori, λrot, λSMPL pos , λdec pos , λvec are weights of corresponding losses. The difference between L SMPL pos and L dec pos is that L SMPL pos is computed based on the position SMPL(ˆ yrot) inferred by SMPL and the ground truth ypos, while L dec pos is computed based on the position ˆ ypos decoded by the decoder and ypos.

4 Experiments

4.1 Implementation Details

For all GRUs in the model, the number of layers is set to 1 and the dimension is set to 256. The joint-relation Transformer has 4 layers with a dimension of 512. The initial learning rate is set to 5 × 10-4 , and is halved every 15000 iterations. The length of the input sequence L is set to 32. {λori, λrot, λSMP L pos , λdec pos , λvec} = {0.02, 1, 1, 1, 0.5}. The model is trained on two GeForce RTX 3090 GPUs for a total of 90000 iterations, with a batch size of 32 on each GPU. The training process costs approximately 32 hours to complete.

4.2 EvaluationMetrics

We adopt the following evaluation metrics.

• MPJPE: Mean Per Joint Position Error [cm].
• MPJRE: Mean Per Joint Rotational Error [degree].
• MPJVE: Mean Per Joint Velocity Error [cm/s].
• fps: Frames Per Second [frame].

4.3 Evaluation Protocols

Standard scenario. Following the experimental setup outlined in AvatarPoser [Jiang et al., 2022], we partition the CMU [graphics lab., 2000], BMLrub [Troje, 2002], and HDM05 [M¨uller et al., 2007] subsets into 90% training data and 10% testing data. For comparison, we evaluate the results using both three inputs (head and wrists) and four inputs (head, wrists, and pelvis). It is worth noting that, unless explicitly stated otherwise, all other results are based on the usage of three inputs.

Instantaneous data-loss scenario. AGRoL [Du et al., 2023] proposed a setting for instantaneous data-loss, which randomly masks out 10% of the input sequence. However, in practical applications, an instantaneous data-loss does not necessarily mean the complete loss of all tracker signals. Instead, it might indicate the loss of information from a specific joint. Therefore, we propose a new setting where signals of each tracker signal have a certain probability p of being lost at each moment. We evaluated each model five times at p = 0.1, 0.5 and 0.9, and took the average as the result.

Prolonged data-loss scenario. In the scenario of prolonged data-loss, data is continuously lost for a long-last period. For evaluation purposes, we set a protocol to mask out subsequent M frames every 80 frames. We evaluated each model in M = 20, 40 and 60.

4.4 Comparison to the State-of-the-art

The evaluation results of each model in the standard scenario are extracted from the reports in their respective papers. For comparison in data-loss scenarios, we select several publicly available state-of-the-art methods (AvatarPoser [Jiang et al., 2022], AGRoL [Du et al., 2023] and AvatarJLM [Zheng et al., 2023]). We conduct a fair comparison by retraining their publicly available code and evaluating them under the protocols for data-loss scenarios using the retrained parameters.

Standard scenario. To verify the effectiveness of our model, we perform a fair comparison with the state-of-the-art methods in the standard scenario. The results, as shown in Table 2 and Table 3, demonstrate that our model achieves state-ofthe-art performance in three- and four-input conditions. This indicates that ReliaAvatar, despite being primarily designed to tackle data-loss scenarios, surpasses other models even in the standard scenario.

Instantaneous data-loss scenario. We compare ReliaAvatar with other models under three conditions: p = 0.1, p = 0.5, and p = 0.9. As shown in Table 4, ReliaAvatar demonstrates its superior robustness compared to other models in the instantaneous data-loss scenario. Under the conditions of p = 0.1 and 0.5, ReliaAvatar is hardly affected. Even when the signals of each tracker have only a 10% probability of being received, ReliaAvatar can still operate normally.

Prolonged data-loss scenario. We compare our model with the previous state-of-the-art models in the prolonged dataloss scenario under three conditions: M = 20, 40, and 60. Table 5 indicates that existing models are unable to handle scenarios where there is a continuous loss of more than 40 frames of signals. In contrast, ReliaAvatar exhibits superior robustness compared to other methods, as it can function nearly normally even with a continuous loss of 20 frames. It is capable of operating even when the signal is continuously lost for 60 frames.

ReliaAvatar's robustness to low-quality signals stems from three aspects: 1) Our model abandons windowed inputs. Models that solely rely on a window of tracker signals as input (e.g., AvatarPoser with a window size of 40) exhibit obvious malfunctions when facing prolonged data-loss since the signals within the window are all set to zero. 2) Our model integrates motion prediction into the avatar animator. When the regression pathway fails to function properly, the prediction pathway can still predict the current full-body motion based on the historical trajectory states. 3) The autoregressive training paradigm allows for simulating abnormal scenarios that may occur in applications.

4.5 Ablation Studies

To illustrate the roles of various designs in our model, we conduct the following ablation experiments.

Input Signals. As shown in Table 6, we validate the performance gain of incorporating historical trajectory states into the input. Here, X represents the input used by most current methods, i.e. X = {xr,∆xr, xp,∆xp}.

If the historical trajectory states are not used as input, the regression pathway remains inactive. The experimental results demonstrate that incorporating yrot,∆yrot, ypos,∆ypos into the input leads to performance gains. Due to the continuity of motion, the current full-body is naturally influenced by and aligned with the previous trajectories. So this improvement can be attributed to the previous trajectories providing cues for the generation of the current full-body motion.

Decoder. As shown in Table 7, we compare the performance of different decoder designs in our model. The model using a

shared decoder has larger errors but shorter runtime while the model using multi-FC decoders has smaller errors but suffers from poor real-time performance. The model using a Sparse-FC decoder has the advantages of both low error and fast running speed.

Fusion Method. The methods to integrate features from two pathways are worth exploring. We have explored three different methods for feature integration: addition, concatenation along the feature dimension, and concatenation along the joint dimension. As shown in Table 8, the fusion method of concatenating features along the embedding dimensions achieves the best performance.

Simulation Training. An important distinction of ReliaAvatar compared to other methods is its ability to simulate various scenarios during the training process. This is attributed to the model architecture with joint motion prediction and the autoregressive training paradigm. The results presented in Table 9 demonstrate that simulation training for data-loss scenarios does not adversely affect performance in the standard scenario.

4.6 Analysis

Ability of motion prediction. An important capability that we expect ReliaAvatar to achieve is the ability to predict fullbody movements of the current frame using historical trajectory states when tracker signals are lost. This enables us to drive avatars smoothly even in the absence of real-time tracker information. We calculate MPJPE for the 40 frames after losing real-time tracker signals. The results presented in Table 10 demonstrate that ReliaAvatar is capable of predicting relatively accurate poses within several tens of frames after the disappearance of tracker signals.

Inference Time. To ensure a fair comparison of the operational efficiency of different models, we conduct speed tests on state-of-the-art models under identical operating modes and hardware conditions. For each model, we construct a zero input and run it 10,000 times to obtain the average time as its inference time. All tests are conducted on a single GeForce RTX 3090 GPU. As shown in Figure 4, ReliaAvatar surpasses other models in both operational efficiency and accuracy.

5 Conclusion

We conducted a comprehensive exploration of the full-body avatar pose estimation problem under low-quality signal scenarios, which had not been systematically investigated before. We summarize three scenarios that may be encountered in practical applications: standard scenario, instantaneous dataloss scenario, and prolonged data-loss scenario. To address these challenges, we proposed ReliaAvatar, a real-time, robust, autoregressive avatar animator. We consider the fullbody avatar pose estimation problem as a combination of joint upsampling and motion prediction. Therefore, ReliaAvatar possesses an upsampling pathway and a prediction pathway. Furthermore, we incorporate simulation training for data-loss scenarios on top of autoregressive training. Experimental results demonstrate that ReliaAvatar not only outperforms other methods in the data-loss scenarios but also achieves state-ofthe-art performance in the standard scenario. In addition to its outstanding accuracy and robustness, ReliaAvatar also exhibits superior running efficiency compared to other methods. These advantages reduce the hardware requirements for animating a full-body avatar. As a result, more affordable and convenient devices can be used to drive avatars, which contributes to the wider adoption of related technologies.

Acknowledgements

This work was supported by National Key R&D Program of China under Grant No. 2021ZD0110400, the Fundamental Research Funds for the Central Universities No. xxj032023020, and sponsored by the CAAI-MindSpore Open Fund, developed on OpenI Community.