Copy Motion From One to Another: Fake Motion Video Generation

1 Introduction

The recent advancement in artificial intelligence has taken the world by storm and given rise to a lot of interesting and compelling applications [Liu et al., 2019b; Liu et al., 2021d; Song et al., 2021; Wang et al., 2021]. Motion copy is one emerging research problem which looks at generating a fake video of a target person performing arbitrary motion, usually extracted from a source person. While the two person may be vastly different in body shape and appearance, motion copy allows retargeting the motion to the target, empowering an untrained person to be depicted in videos dancing like pop stars [Chan et al., 2019], playing like NBA players, and maneuvers in gymnastics and martial arts.

∗Corresponding Authors

Fundamentally, copying the motion from a source person to a target person amounts to learning a mapping between their video frames. Due to the high dimensionality of this mapping (which renders it almost intractable), it would be more economical to leverage an intermediate representation to bridge the problem. Typically, such approaches include pose-guided and mesh-guided representations, which respectively extract pose or mesh representations from the source person and then learn a generative model that maps this intermediate representation (pose or mesh) to the appearance of the target person. We focus on pose-guided target video generation in this paper.

Upon scrutinizing and experimenting with the released implementations of the state-of-the-art methods, we observe that there are two key issues to be addressed. (1) Whereas existing methods achieve plausible results on a broad stroke, the issues of distorted face, hands, and feet are quite rampant and such defects are highly noticeable to the human observer. (2) Current approaches tend to heavily rely on GANs with L2 loss to learn generative models for pose-to-appearance generation. When the training samples for the target person are scarce (typically only a single target video is available), it would be inadequate to learn a competent GAN that typically requires a large amount of training samples to discriminate the authenticity of the generated frames satisfactorily.

To tackle the challenges, we propose a novel motion copy framework FakeMotion, consisting of three key designs: 1) We first perform a foreground and background separation. Focusing on foreground generation fundamentally reduces the complexity and dimensionality of the problem. 2) We propose a theoretically motivated Gromov-Wasserstein loss to guide pose-to-appearance generation. The GromovWasserstein loss constitutes an optimal transport objective that learns the pose-to-appearance mapping in a pair-wise fashion, and is more data-efficient and well-suited than conventional GANs for learning the generative model. 3) We refine the textural details of the face, feet, and hands with multiple dedicated GANs. In particular, our face refinement adopts a self-supervised approach where we employ geometric cues such as facial orientations to serve as guidance for mining real facial appearances of the target person. These are combined with the coarse generated face for refinement, and the encoding of similar oriented real faces results in more delicate and realistic facial textures. Extensive experiments demonstrate that our method outperforms existing methods.

To summarize, the key contributions of this work are: (1) We propose a novel architecture for fake motion video generation, which exploits a Gromov-Wasserstein loss to frame pairwise distance constraint in pose-to-appearance generation. We also present a dense skip connection structure in the generator to facilitate filling the gap between pose and appearance. (2) We further introduce a local enhancement module to explicitly handle the face, feet, and hands, which are visually prominent body parts and are often affected by artifacts. (3) Extensive experiments on benchmark datasets demonstrate that our approach achieves new state-of-the-art performance. As a side contribution, we crafted a new dataset for evaluating motion copy methods where 50 subjects in the videos perform complex motions. Our code is released to facilitate future research.

2 Method

Problem Statement. Given two videos, one for the target person whose appearance we want to synthesize, and the other for the source person whose motion we seek to copy [Chan et al., 2019], the goal is to synthesize a video of the target person enacting the same motion as the source person.

Method Overview. An overview of our method Fakemotion is outlined in Fig. 1. Overall, Fakemotion consists of three key stages: a) Data Pre-processing, b) Pose-Guided Appearance Generation, and c) Local Parts Enhancement. Specifically, we first extract the pose sequence from the video of the source person. Then, we feed the poses into a pose-toappearance network to synthesize the foreground images of the target person. Further, we engage in multiple local GANs to polish the local body parts. Finally, the polished foreground is fused with the background, which is extracted from the video of the target person, to approach the final result. We would like to highlight that our generators have an edge in adopting dense skip connections and are equipped with a Gromov Wasserstein loss, while our discriminator games in spatial and temporal dual constraints. In what follows, we elaborate on the key components in detail.

2.1 Data Pre-processing

Presented with a video Vs of the source person and a video Vt of the target person, we perform the following in parallel: (1) extracting the human pose from each frame of Vs, and (2) disentangling each frame of Vt into foreground (person) F and background B. The motivations are twofold: (1) The extracted poses of source Vs unambiguously characterize the motion and can be used to guide appearance generation. (2) Generating the entire image conditioned on a pose might be too ambitious. Synthesizing only the foreground conditioned on a pose input would reduce the complexity and result in better texture details. Technically, we could adopt off-theshelf pose estimators [Cao et al., 2019; Liu et al., 2021b; Liu et al., 2022] for pose extraction. For foreground and background decoupling, we follow [He et al., 2017].

2.2 Pose-to-appearance Generation

Now, our goal is to generate an appealing foreground image in accordance with a given pose. Particularly, we design a pose-to-appearance generation network, consisting of a generator that incorporates dense skip connections and a Gromov-Wasserstein loss, as well as a discriminator that enforces spatio-temporal dual constraints.

Generator with Dense Skip Connections. As illustrated in Fig. 2, we engage a U-shaped architecture for our poseto-appearance network. In vanilla U-Net, a decoder layer Di is connected to a single encoder layer Ei and is unable to access low-level features from encoders {Ej}i-1 j=1 . Consequently, different level features are isolated to each other and their connections are ignored. Building upon the U-Net, we propose to add extra skip connections that help a decoder layer to access lower level features. Instead of connecting a decoder Di at layer i with only the encoder Ei at layer i, we allow Di to access encoder layers {Ej|j < i}. In this way, each decoder could integrate and fuse feature representations frommultiple layers.

Gromov-Wasserstein Loss to Facilitate Pose-toappearance Generation. Our generator network is designed to learn a mapping from a pose Pt to the corresponding foreground image Ft of the target person. To enhance temporal consistency, instead of a single pose Pt, we input a pose context window 〈Pt-1, Pt, Pt+1〉 to the generator. Current methods typically approach this pose-to-appearance problem with a conventional GAN, and measure the discrepancy between the generated foreground and ground truth via a pixelwise L2 loss. However, this inherently requires a large number of training samples in order to generate viable and realistic images. Furthermore, training such GAN models is often precarious and tends to be unstable when adapted to a wide range of target videos. To alleviate this issue, inspired by optimal transport, we propose to measure the generated and ground truth distance via a Gromov-Wasserstein loss, which essentially defines a pairwise distance at the feature-level.

Interestingly, our experiments reveal that our pairwise loss plays an unexpectedly significant role in the model performance. Heuristically, our proposed Gromov-Wasserstein loss looks at the overall distance structure perception. We first extract the VGG features for the generated foreground F and ground truth F :

where Ψ(·) represents a pre-trained VGG network. The Gromov-Wasserstein loss defines a distance between the metrics on the two feature spaces. We consider two batches of feature vectors {zx i }ni=1 , {zy i }ni=1 as two discrete empirical distributions τ , υ:

where δ denotes the Dirac delta distribution. Formally, the Gromov-Wasserstein loss for our task is defined as

where Π denotes the set of point distributions with marginals τ and υ. To obtain the Gromov-Wasserstein distance for these two point distributions, we solve for the optimal transport matrix π that minimizes the squared distance between the intraspace L1 costs. We follow [Peyr´e et al., 2016] in introducing an entropic regularization term which guarantees tractability (as well as being amenable to backpropagation) in the optimization. We then adopt the Sinkhorn algorithm and projected gradient descent to optimize the entropy regularized Gromov-Wasserstein objectives.

where C denotes the cost matrix for a given batch of the generated foreground features and the ground-truth features. ε denotes a regularization coefficient.

Optimizing the Gromov-Wasserstein distance via above equations (Eqs.4-7) updates the generator network parameters through backpropagation. Effectively, this effectuates an alignment of these two feature embedding spaces that facilitates matching of the generated foreground image with the ground-truth. Even when there is scarce training samples for the target person, this pairwise feature matching approach could train the network sufficiently and yield a more detailed and realistic generation. Furthermore, the GromovWasserstein distance is also tractable and stable, and empirically generalizes well to a variety of target video domains.

Discriminator with Dual Constraints. Humans generally judge the authenticity of a video from two aspects: the quality of the images and the temporal consistency across frames. The continuity between frames plays a crucial role in visual judgment. Yet, this is often neglected in many existing methods [Dong et al., 2018; Kappel et al., 2021] which tend to focus on single frame quality and cannot explicitly consider temporal continuity. In our approach, we devise dual constraints which are composed of a quality discriminator Dq and a temporal discriminator Dt. (1) The quality discriminator Dq focuses on the single frame quality to assess the authenticity of the forged foreground. (2) The temporal discriminator Dt is equipped with a set of parallel dilated convolutions that capture temporal information across frames at various temporal scales. As illustrated in Fig. 1, Dq takes (Pt, Ft) and (Pt, Ft) as input pairs while the input to Dt consists of ({Pi}t+1 i=t-1 , {Fi}t+1 i=t-1 ) and ({Pi}t+1 i=t-1 , {Fi}t+1 i=t-1 ). Both discriminators are trained to classify the authenticity of the generated frame or context window.

2.3 Local Parts Enhancement

While the global quality of the generated frames is certainly essential when judging the authenticity of a video, our visual system is also sensitive to prominent body parts, especially the face as well as the hands and feet. When these body parts appear unnatural, our mind would be quick to identify them and judge the videos as fake. Upon experimenting with existing methods following their released codes [Chan et al., 2019; Balakrishnan et al., 2018; Wang et al., 2019; Liu et al., 2019a; Wei et al., 2020], an important insight and conclusion that we draw are that even the state-of-the-art methods still have difficulties in generating natural and realistic details for the face, hands, and feet. Therefore, we introduce a self-supervised face enhancement and additional dedicated local GANs to refine these body parts, with the goal of polishing the texture details on top of the generated results from the pose-toappearance generation module.

Self-Supervised Face Enhancement. Intuitively, the facial images of the same person should look alike when they are viewed from the similar orientation or viewing angles. With this geometric guidance, we search from the given video of the target person to extract facial images that exhibit similar facial orientations and alignment as that of the synthesized image. This is then leveraged to refine the final face output.

To measure face orientation similarity, we do not simply compute the (cosine) similarity between facial features as the full facial features would usually contain redundant information, e.g., color and eye shape, that are irrelevant to face orientation. Instead, we represent facial orientation with vector fields. Specifically, as shown in Fig. 3, we employ 5 vectors, including v1: right eye → left eye, v2: left eye → left side of the mouth, v3: left side of the mouth → right side of the mouth, v4: right side of the mouth → right eye, v5: the center of eyes → nose. Given two face orientations {vi}5i=1 and {vi}5i=1 , their similarity can be conveniently computed as:

Subsequently, we select top m ground-truth face images f = {f1 ,..., fm} with the highest similarities as auxiliary faces. Finally, the initially generated face F ′1 is enhanced into:

where α, λi, and β are hyperparameters.

Multi-Local GANs for body parts. The face enhancement enforces the generated face to be similar to faces with close orientations. We further polish the face and limbs. Similar to the divide-and-conquer strategy in [Liu et al., 2021c], we model each body part with a dedicated local GAN to perform refinement. Specifically, we crop the five key parts F ′1 (face) and {F i }52 (hands, and feet) of the generated foreground image F t, where i denotes the ith key body part. We further feed them into corresponding GANs, which are trained to output a residual image F i r . The residual image reveals the difference in color and texture details between the generated image of the body part and the ground truth. We add them to the generated images to yield the final foreground:

Foreground and Background Fusion. Finally we may combine the polished foreground ˜ Ft with the inpainted background Bt. We employ a linear sum to fuse ˜ Ft and Bt into the final video frame as:

where M is the mask obtained during the pre-processing phase highlighting the region of the foreground.

In practice, for background Bt, we filled the removed foreground pixels in Bt following [Yu et al., 2019].

2.4 Loss Functions

Now, we zoom in on the loss functions. The proposed Gromov-Wasserstein loss promotes a better learning of the embedding correspondence between the generated foreground space and the ground-truth space. This facilitates extracting commonalities and shared structural similarities between the two spaces.

We further introduce two standard spatial-temporal adversarial losses for Dq and Dt, where Dq enhances the frameby-frame spatial constraints and Dt reinforces temporal constraints for every three consecutive frames:

The Multi Local GANs polish the details of key body parts and we use critics Dj to present supervision on local details:

3 Experiments

3.1 Experimental Settings

Datasets. Experiments are conducted on two benchmark datasets, iPER [Liu et al., 2019a] and ComplexMotion. The iPER dataset contains 30 subjects of different shapes, heights and genders. Each subject wears diverse clothes and performs different motions. The whole dataset contains 241,564 frames from 206 videos. Although there are a large number of videos in the iPER, the motions in the videos are relatively common. In order to further examine the performance of the methods on complex motion scenes, such as doing sophisticated and rapid motion, we collected a benchmark video dataset, ComplexMotion, consisting of 122 inthe-wild videos for more than 50 human subjects. These videos are acquired from various video repositories including TikTok and Youtube. In particular, humans in the dataset wear different clothes and perform complex and rapid movements such as street dance, sports, and kung fu.

Implementation details. During training, all the images are resized to 512 × 512. We utilize OpenPose to detect the 18 human joints in each frame, which form the pose skeleton. We employ the Mask-RCNN to disentangle each video frame into the foreground and background. We train our model with a mini-batch of 10 for 120 epochs on a Nvidia RTX 2080-Ti GPU. The initial learning rate is set to 1e-4. We employ the Adam optimizer with β1 = 0.9 and β2 = 0.999.

3.2 Comparisons to State-of-the-art Human Motion Copy Methods

To evaluate our approach FakeMotion, we compare it with state-of-the-art approaches including Everybody Dance Now (EDN) [Chan et al., 2019], PoseWarp [Balakrishnan et al., 2018], Few-Shot Video2Video (FSV2V) [Wang et al., 2019], Liquid Warping GAN (LWG) [Liu et al., 2019a], C2F-FWN [Wei et al., 2020], SS-FR, MLG, and TD. Specifically, SSFR, MLG and TD are three variants of FakeMotion, which correspond to removing the self-supervised face refinement, multiple local GANs, and temporal discriminator from our default method, respectively.

Quantitative Comparisons. Following the protocol of existing methods [Liu et al., 2019a; Huang et al., 2021], we quantitatively benchmark our method and current approaches on two scenes: Person Reconstruction and Action Imitation. For Person Reconstruction, we perform self-imitation where a person mimics motions from themselves. We adopt Structural Similarity (SSIM) [Wang et al., 2004], Peak Signal to Noise Ratio (PSNR) [Korhonen and You, 2012], and Learned Perceptual Image Patch Similarity (LPIPS) [Zhang et al., 2018] to evaluate the quality of the generated image. For Action Imitation, we perform cross-imitation where a person mimics another person's movements. We employ the Inception Score (IS) [Salimans et al., 2016] and Frechet Inception Distance score (FID) [Heusel et al., 2017] as evaluation metrics. To measure the quality of the video instead of image, we also adopt a Temporal Consistency Metric (TCM) [Yao et al., 2017] which concerns temporal consistency. The experimental results on ComplexMotion and iPER are reported in Table 1. Our approach consistently outperforms state-of-the-art methods in both Person Reconstruction and Action Imitation.

Qualitative Comparisons. In addition to quantitative comparisons, we also analyze the generated images visually. Fig. 4 illustrates visual results of six different methods. We observe that PoseWarp and FSV2V suffer from distorted human body and absent hands. EDN achieves plausible visual results, however, the face of the person is blurred. Warpingbased LWGAN and C2F-FWN are able to effectively copy the motion of source person, but fail to generate fine-grained parts such as hairs and clothing. Our method, in contrast, yields more robust and realistic results on human subjects. We attribute the performance gain to the proposed dense skip connections, Gromov-Wasserstein objective, and the body part GANs.

Ablation Study. Extensive ablation experiments are also conducted to investigate the effects of various components. We first investigate the effect of removing GromovWasserstein loss in the generator. Specifically, we adopt a traditional L2 loss instead of the Gromov-Wasserstein loss. From Table 2, we observe that SSIM and PSNR fall from 0.883 and 27.15 to 0.836 and 25.06, respectively. These performance deteriorations upon removal of the GromovWasserstein loss suggest the efficacy of the proposed loss. In addition to Gromov-Wasserstein loss, we also investigate the impact of removing dense skip connections in the generator. From Table 2, we observe clear performance dips upon removal of the dense skip connection component. This is in line with our intuition that integrating and fusing feature representations frommultiple encoder layers are beneficial.

To evaluate the self-supervised face refinement module, we consider 1) selecting similar real face images based on facial image features instead of face orientations, 2) using a different number of face vectors, and 3) employing a different number of similar real images. The results of these ablation experiments are reported in Table 2. We observe significant performance drops of SSIM and PSNR upon adopting the first setting. This suggests the effectiveness of using face orientations for similar face searching, which avoids being interfered by unimportant features such as color and eye shape. Furthermore, we observe that the quality of the generated image gradually increases with the growing of the number of face vectors. Similarly, the increase in the number of used similar real images also leads to better results.

Finally, we also try removing the multiple local GANs to evaluate their contributions. Upon the removal, the FID dramatically increases from 48.03 to 61.49 (the smaller the FID value, the better the image quality). This highlights the importance of local refinement. In particular, the residual images, which are produced by the multiple local GANs, disclose the difference in fine-scale texture details between the generated local body parts and the ground-truth, facilitating the generation of more lifelike local images.

4 RelatedWork

Pose-guided approaches. The researches learn mapping functions between a pose and its corresponding image [Ren et al., 2020; Zhang et al., 2021; Kappel et al., 2021]. [Ma et al., 2017; Wang et al., 2018] seek high-resolution video2video synthesis by adopting GAN or Two scale-GAN networks. [Yang et al., 2020] performs human video motion transfer in an unsupervised manner, taking advantage of the independence of three orthogonal factors, motion, skeleton, and view. However, these methods fail to take into account the importance of maintaining facial details and the end effectors (i.e. hand and feet) in motion transfer. [Chan et al., 2019] introduces a face enhancement module that adopts a traditional GAN architecture.

Warpping-based approaches. The studies aim to warp the appearance of the target person to copy motions from another person [Balakrishnan et al., 2018; Dong et al., 2018; Liu et al., 2021a; Siarohin et al., 2021]. [Liu et al., 2019a] disentangles the human image into action and appearance, and achieves motion copy using a Warping GAN which warps the image of the target person performing standard actions. [Wei et al., 2020] extracts the motion manifold of the target human and accordingly refines the detailed motion. However, these methods are mesh based, and have inherent difficulties in dealing with irregular shapes and clothing, and also suffer major performance deterioration during rapid motions (unless we have an unlimited number of meshes).

5 Conclusion

In this work, we propose a novel fake video generation framework termed FakeMotion, which is equipped with dense skip connections, Gromov-Wasserstein GANs, and local enhancement modules. A particular highlight of our method is the Gromov-Wasserstein loss which enables a robust and stable generator training. We further propose a self-supervised face refinement module that resorts to real faces with similar orientations to polish facial details (textures), and multiple local GANs which enhance local details of body parts. Extensive experiments show that our method achieves notable improvements and is able to handle rapid-motion and complex scenes.

Acknowledgements

This research is supported by the National Key R&D Program of China (Grant no.2018YFB1404102), the National Natural Science Foundation of China (No. 61902348), the Key R&D Program of Zhejiang Province (No. 2021C01104), and the Postgraduate Research Innovation Fund Project of Zhejiang Gongshang University.