Generating Person Images with Appearance-aware Pose Stylizer

1 Introduction

Generating realistic-looking human images is of great value in many tasks such as surveillance data augmentation [Zheng et al., 2017] and video forecasting [Walker et al., 2017; Wang et al., 2018b]. In this work 1 , we focus on the poseguided person image generation [Ma et al., 2017] which aims to transfer person images from one pose to other poses. The generated person is expected to accord with the conditioned pose structure as well as preserving the appearance details of the source person. Fig. 1 provides some pose transfer results generated by our proposed framework as examples. The pose-guided person image generation is very challenging due to the following aspects: (1) The distributions of clothes, body appearance, backgrounds, and poses vary largely between human images; (2) One person of different poses may have very different visual features; (3) The generative model usually needs to infer the appearance details of body parts which are unobserved in input images.

1 Code is available at https://github.com/ siyuhuang/PoseStylizer

To address the above challenges, it is essential to decouple the complex entanglements, such as the interplay between appearance and pose [Esser et al., 2018], in person generation procedure. Towards that goal [Ma et al., 2018] introduced a learning-based pipeline to disentangle and encode three factors: image foreground, background, and pose into separated representations and then decode them back to a person image. Although the three factors are successfully decoupled by the encoder, the representations from three factors are simply concatenated into latent codes before they are fed into the decoder, resulting in a lack of controllability and interpretability in the decoding phase. We note that such a mode decoupling is significant for the decoding phase. From the application perspective, the users prefer a more controllable image editing process brought by flexible input modes. From the engineering perspective, the experts would lean to a generator with more interpretability, such that distinct feedback can be collected from disentangled modes to better guide model design, tuning, and optimization.

Motivated by the above observations, we investigate a strategy to decouple and re-couple the entanglement factors including appearance-pose, foreground-background, and local detail-global structure in person images. We propose a novel end-to-end person image generation framework consisting of an appearance encoder and an Appearance-aware Pose Stylizer (APS): The appearance encoder learns the appearance representation of the person image; APS, as shown in Fig. 2, is the image generator which progressively couples the target pose map with the appearance representation, enabling a natural re-coupling of pose and appearance. We additionally adopt an attention mechanism in both encoder and generator to disentangle the foreground and background. In APS, the image is progressively synthesized from small to large scale, thus local details and global structures are fused and preserved in a multi-scale approach..

In summary, the proposed end-to-end framework can effectively decouple the entanglements between appearance, pose, foreground, background, local details, and global structures, and re-couple them in the generator, thus generate highquality person images in a highly flexible and controllable way. The contributions of this paper are summarized below. (1) We propose a novel person image generation framework to make explicit disentanglement of the complex factors in human images. (2) We propose a new normalization method called adaptive patch normalization. It enables normalization within local regions and is suited to the spatially-dependent generative tasks including person image generation. (3) We have conducted extensive quantitative and qualitative experiments, and ablation studies to validate the effectiveness of the proposed methods.

2 RelatedWork

Person image generation is very valuable in many real-world applications [Wei et al., 2018; Chan et al., 2019]. Various settings of person image generation have been proposed in the literature. [Lassner et al., 2017] proposed 3D pose representation to generate images of a person with different clothes. [Zhao et al., 2018] generated multi-view cloth images from a single view cloth image. Several work on virtual try-on [Han et al., 2018; Wang et al., 2018a; Zanfir et al., 2018; Dong et al., 2019] including FashionGAN [Zhu et al., 2017] proposed to manipulate the clothes of a given person while maintaining the person identity and pose.

In this work we focus on the pose-guided person image generation [Ma et al., 2017] which aims at generating images of a person with different poses but with the same clothes and identity. Based on generative adversarial networks (GANs) [Goodfellow et al., 2014; Mirza and Osindero, 2014], several efforts [Neverova et al., 2018; Song et al., 2019; Zhou et al., 2019] have been made towards this goal. More related to this work, [Ma et al., 2018] proposed to disentangle image foreground, background, and pose into separated representations by the encoder. [Zhu et al., 2019] proposed to transfer person pose in the encoding phase using an attention-based progressive model [Karras et al., 2018]. Both the above methods disentangles image factors in the encoding phase and combines the disentangled representations before decoding, lacking explicit cross-modal re-coupling in the decoding phase.

Our approach is also related to adaptive normalizationbased GANs [Karras et al., 2019; Park et al., 2019] which progressively transforms a constant vector or a random noise (namely, content) into an image using a stack of convolutional layers and normalization layers with learned coefficients (namely, style). More recently, [Yildirim et al., 2019] applied StyleGAN [Karras et al., 2019] to fashion image generation, where the content input is a constant vector and the style input is a combination of clothes and pose information. Different from existing normalization-based generative models, we set pose and appearance information of a person as content and style inputs, respectively. Our approach enables a more natural re-coupling of cross-modal representations, thus it is more effective and efficient in articulated-object generation problems.

3 Our Approach

The goal of pose-guided person image generation is to generate a person image Ig which is expected to follow a given person pose Ps while keep the appearance details of a given source person image Is. In this paper, we propose an end-toend generative framework including an appearance encoder and an Appearance-aware Pose Stylizer (APS) to address this challenging task.

3.1 Appearance Encoder

We use an attention-based appearance encoder to learn the appearance representation of the source image Is. The appearance encoder is built upon a stack of L encoder blocks and the architecture is shown in the left part of Fig. 3. The appearance encoder has two network streams, the image stream and pose stream. The two streams take the outputs of two streams in layer l -1 as their inputs and output I ls and P l s , respectively. The input of image stream is source image Is such that image stream extracts the visual representation of source image. The input of pose stream is source pose Ps such that pose stream learns the structure of source pose to guide the information flow in image stream with attention mechanism [Zhu et al., 2019]. More specifically, the pose input P l s from previous layer goes through a convolutional layer and a sigmoid activation layer to obtain foreground attention masks M ls ∈ (0, 1) as

Then, M ls masks the image stream using element-wise multiplication as

The residual connection in Eq. 2 eases the learning of deep generative models in practice. With pose-guided attention, image stream is forced to focus on the foreground of image, i.e., the person.

3.2 Appearance-aware Pose Stylizer

Appearance-aware Pose Stylizer (APS) is a novel generator for realistic person image generation. As shown in Fig. 2, the pose stylizer stylizes the target pose map Pt under guidance of appearance representation, progressively. Similar to appearance encoder, APS consists of L repetitive generator blocks to restore images from small sizes to large sizes. In the following we discuss more details of the generator blocks.

Generator Block

In a generator block, there are two network streams including foreground image stream and synthesized image stream, as shown in the right part of Fig. 1. Foreground image stream If generates foreground person images based on pose maps Pt and appearance representations I L s . Synthesized image stream Ig synthesizes complete images including both foregrounds and backgrounds. In the foreground stream, we adopt adaptive normalization mechanism (AdaNorm) to stylize pose maps based on appearance representations, such that our generator is named as Appearance-aware Pose Stylizer. AdaNorm(x, y) accepts content feature x and style feature y

deconv is a deconvolutional layer, concat is the channel-wise concatenation operation, embed is a convolutional layer with a kernel size of 1×1. I 0 f = Pt such that content x is derived from pose map Pt. Style y is derived from appearance representation I L s . Note that the setting of content and style in this work is distinctly different from existing normalizationbased generative models such as StyleGAN [Karras et al., 2019] and SPADE [Park et al., 2019], in which the content x is generally set as a constant vector or a random vector. In our setting, AdaNorm naturally disentangles pose and appearance of a person as its content and style inputs, leading to a reasonable feature fusion as well as an effective person generation pipeline.

After computing z = AdaNorm(x, y), the foreground stream outputs I lf = z + x. The foreground stream fuses into the synthesized stream with attention mechanism,

The synthesized stream outputs I lg as

Specifically, I 0 g is feature maps with zero values. I L f and I L g output by the last generator block are concatenated and decoded as the generated image with an 1×1 convolutional layer.

Adaptive Patch Normalization

Here we discuss the AdaNorm module used in our method. In existing literature, StyleGAN and SPADE successfully applied adaptive instance normalization (AdaIN) [Huang and Belongie, 2017] to progressive generative models. AdaIN is formulated as

where x and y are content and style features respectively. c denotes the channel number, β and γ denote mean and standard deviation. The weight term yw c and bias term yb c are embedded from input style feature y. In Eq. 7, content feature x is first normalized by instance normalization (IN) and then scaled by parameters conditioned on style feature y.

In this paper we develop a new normalization method, i.e., adaptive patch normalization (AdaPN)

where i, j denote the spatial position of a patch on feature map. The content feature is first normalized by IN and then scaled with region-specific parameters yw c,i,j and yb c,i,j . 2 Fig. 4 illustrates the difference between existing normalization methods. The proposal of patch normalization is motivated by the observation that human body parts within a specific spatial region are relatively fixed among well-cropped person images. For instance, the head usually appears in the topcenter area of an image while the legs usually appear in the bottom area of an image. Therefore, it is natural to normalize spatial regions with different factors. Compared with AdaIN, AdaPN induces the generator to learn appearance details of different body parts more effectively. In Section 4.4, we take insights into AdaPN by conducting more empirical studies.

3.3 Training and Optimization

The loss function that we use to train our person image generation model is comprised of the adversarial loss LGAN, the reconstruction loss L1, and the perceptual loss Lper as follow:

In Eq. 9, L1 loss LL1 = ‖Ig -It‖1 where Ig is the generated image and It is the ground-truth target image. The perceptual loss [Johnson et al., 2016] computes L1 distance over feature maps. The adversarial loss LGAN consists of the appearanceconsistency term and the pose-consistency term, formulated as

where Da and Dp are discriminators. (Is, It) ∼ Ireal, Ig ∼ Ifake, Pt ∼ P. Ireal, Ifake, P are the distribution of real images, fake images, and person poses, respectively.

2 In implementation of AdaPN, in each generator block we embed I L s to latent features and then tile the features into yw and yb . The sizes of yw and yb are the same to the size of corresponding x.

Network architectures. For both encoder and generator, we adopt a total block number L = 4 on Market-1501 dataset and L = 5 on DeepFashion dataset, respectively. The first layer of encoder and the last layer of generator has 64 channels. The number of channels in every block is doubled/halved in encoder/generator until a maximum of 512. The size of feature map in every block is halved/doubled in encoder/generator using stride-2 convolutions/deconvolutions. The appearance representation I L s has a shape of 512× H 2L × W 2L , where H and W is height and width of the input image. Every AdaNorm module is built up with AdaPN-Conv-AdaPN in practice for establishing a deeper model.The standard/adaptive normalization layer and ReLU are applied after every convolutional or deconvolutional layer.

Training details. We implement our model on the PyTorch framework [Paszke et al., 2017]. The model is trained with an Adam optimizer [Kingma and Ba, 2014] for 800 epochs. The initial learning rate is 0.0002 and it linearly decays to 0 from 400 epochs to 800 epochs. Following [Zhu et al., 2019], the loss weights (α,λ1,λ2) are set as (5, 10, 10) on Market-1501 and (5, 1, 1) on DeepFashion. In training Market-1501, we additionally apply Dropout [Hinton et al., 2012] with a rate of 0.5 after every generator block in case of overfitting.

4 Experiments

4.1 Experimental Setups

Datasets. We conduct experiments on two benchmark person image generation datasets including Market-1501 [Zheng et al., 2015] and DeepFashion (In-shop Clothes Retrieval Benchmark) [Liu et al., 2016]. Market-1501 is a challenging person re-identification dataset which contains 32,668 images of 1,501 person identities. The images in Market-1501 are low-resolution (128×64 pixels) and the person pose, viewpoint, illumination, and background vary largely. DeepFashion contains 52,712 in-shop clothes images (256×256 pixels). We adopt OpenPose [Cao et al., 2017] as our pose keypoints detector. By following the settings in [Zhu et al., 2019], for Market-1501, we collect 263,632 training pairs and 12,000 testing pairs. For DeepFashion, we collect 101,966 pairs for training and 8,570 pairs for testing. Each pair is composed of two images of the same identity but different poses. The person identities in training sets do not overlap with those in testing sets.

Evaluation metrics. In this work we use Structural Similarity (SSIM) [Wang et al., 2004] to measure the structure similarity between images, i.e., the appearance-consistency. We use the Inception Score (IS) [Salimans et al., 2016] to measure the image quality. Following PG2 [Ma et al., 2017] we adopt their masked versions mask-SSIM and IS to evaluate the image foreground only via masking out the background, since no background information is provided for person generation models. In addition, we use Percentage of Correct Keypoints (PCKh) [Andriluka et al., 2014] to measure the pose joints alignment, i.e., the pose-consistency.

4.2 Results

Qualitative comparison. Fig. 5 shows a qualitative comparison of state-of-the-art person image generation methods on DeepFashion. The source images, pose maps, target images, and the generations are shown from the left to the right, respectively. We compare our method with Variational U-Net (VUNet) [Esser et al., 2018], Deformable GANs (Deform) [Siarohin et al., 2018], and Pose-Attentional Transfer Network (PATN) [Zhu et al., 2019]. The example images shown in Fig. 5 vary in poses, scales, and colors and types of clothes. Our method shows appealing results in the following aspects: (1) Realistic generations: The generated images show natural facial details, body postures, clothing collocations, and skin appearances; (2) Clothing-consistency: Clothes colors and styles are consistent with those in source images; (3) Identityconsistency: Person identity details including facial appearances, body figures, and skin colors are well maintained in generated images; (4) Pose-consistency: Poses of generated persons well follow the conditioned poses.

Fig. 6 shows qualitative results on Market-1501. Although the images of this dataset are low-resolution and their visual details are somewhat blurry (as shown in Source and Target columns), we observe that our method shows a good performance in comparison with the other methods considering the sharpness of generated images.

Quantitative comparison. Table 1 quantitatively evaluate the person generation methods under a series of metrics. Our method shows a competitive performance compared with the existing methods. It achieves the best PCKh on both datasets, indicating that the generations have a good consistency with the conditioned poses. We attribute it to the natural disentanglement of pose and appearance in our APS model. On Market-1501, our model performs well on SSIM, maskSSIM, and mask-IS. On DeepFashion, our model performs well on SSIM. However, it is relatively worse on IS. We conjecture it is because the model well restores the visual appearance details, while, a certain level of overfitting may hurt the realness performance.

Figure 6: Qualitative comparison of existing pose-guided person generation methods on Market-1501.

4.3 Ablation Study

We conduct ablation study on our person generation model to evaluate the efficacy of different components proposed in this paper. The first part of Table 2 evaluates different encoders and decoders. With the same encoders (Conv encoder or PATN encoder), our APS decoder shows large improvements over Deconv decoder. The second part of Table 2 evaluates different components of APS generator. In StyleGAN [Karras et al., 2019], the content input is a constant vector and the style input contains both pose and appearance representation. APS shows significant improvements over StyleGAN, demonstrating that the distinct disentanglement of pose and appearance in generator can benefit the performance of person image generation. APS w/ AdaIN replaces the AdaPN with AdaIN in APS, and the results show that our proposed AdaPN is more suited to spatially-dependent generative tasks. APS w/o attention removes all the attention modules in encoder and decoder, and the results indicate that attention mechanism can slightly help the APS model. Decoding both I L f and I L g is better than only decoding the synthesized stream I L g .

4.4 Study on Adaptive Normalization

Adaptive normalization is the core of our APS model. Fig. 7 compares AdaIN and AdaPN on testing set of DeepFashion, vs., the training epochs. Before 400 epochs, AdaIN and AdaPN shows similar L1 scores. After 400 epochs, L1 scores of AdaIN does not decrease anymore, while, L1 scores of AdaPN is decreasing continuously until the end of training. It reveals that AdaPN has a larger model capacity than AdaIN in learning, thus to better reconstruct the high resolution images. AdaPN also shows better Inception Scores on most of the epochs, demonstrating its superiority in person generation model.

In Fig. 8 we visualize the AdaPN statistics of a fully trained model. The statistics of individual generator blocks, including bias yb and scaling factor yw , are shown at right. We notice that the biases of different positions are similar within a layer. Conversely, the scaling factors of different positions vary largely, suggesting that the local-version scaling is valuable in normalization-based person generation, while, the local-version translation is not much necessary. Intuitively, the scaling operation is related with the input variables. Compared with the bias term, the scaling term contributes more to variations within the output. This leads to the necessity of locally sensitive scaling factors.

5 Conclusion

In this paper, we have presented a novel framework for generating realistic person images. The framework decouples the image factors by an attention-based appearance encoder and re-couples the image factors by an APS generator. It is effective, controllable, and flexible since it makes explicit disentanglement of the complex factors in human images. We have also proposed AdaPN which enables local-specific normalization for spatially-dependent generative tasks. Extensive experiments on benchmark datasets have validated the effectiveness of our approach over existing methods.