Deliberation Learning for Image-to-Image Translation

1 Introduction

Unsupervised image-to-image translation is an important application task in computer vision [Zhu et al., 2017; Choi et al., 2018]. The encoder-decoder framework is widely used to achieve such translation, where the encoder maps the image to a latent representation and the decoder translates it to the target domain.

Considering labeled data is costly to obtain, unsupervised image-to-image translation is widely adopted, which tries to uncover the mapping without paired images in two domains. CycleGAN [Zhu et al., 2017] is one of the most cited model that can achieve unsupervised translation between two domains. Denote Xs and Xt as two image domains of interests, and no labeled image pair in Xs×Xt is available. As the name suggests, the two ingredients to make CycleGAN successful are: (1) Minimizing the reconstruction loss between two models Xs → Xt → Xs and Xt → Xs → Xt; such a loss can ensure the main content of an image is kept; (2) the adversarial network [Goodfellow et al., 2014], that can verify whether an output image belongs to specific domain. DualGAN [Yi et al., 2017] and DiscoGAN [Kim et al., 2017] also adopt similar ideas. Another important model StarGAN [Choi et al., 2018] is designed for multi-domain image-to-image translation with one encoder and one decoder. The motivation of StarGAN is to reduce parameters by using a single model to achieve multi-domain translation instead of using multiple independent models.

∗Equal contribution.

†Corresponding author.

Although the aforementioned techniques achieved great success, they lack an obvious step compared with human behaviors: deliberation, which means reviewing and keeping polishing the output. For example, to draw a paint, the artist first sketches the outline to get an overall impression of the image, and then gradually enrich more details and textures. Such a behavior broadly exists in human's behavior, but still lacks in unsupervised image-to-image translation literature.

Deliberation is important for image translation as illustrated in Figure 1. For the Winter→Summer translation (first row), there are still some snow in the direct output of the decoder (middle). Similarly, for Summer→Winter translation (second row), we can still find green trees in the output (middle). This shows the necessity of deliberation in image translation. To leverage such an important property, we propose deliberation learning for image-to-image translation, that can further polish and improve the output of a standard model. Take the translation from Xs to Xt as an example. Our deliberation learning framework consists of an encoder, a decoder and a post-editor: The encoder and decoder are the same as those in CycleGAN or StarGAN, serving for encoding the image as a hidden vector and decoding the image conditioned on the hidden vector. They work together to translate x ∈ Xs to a ˆ y ∈ Xt. The post-editor will eventually output another y∗ ∈ Xt, with both x and ˆ y as inputs. Compared with the standard encoder-decoder framework, the post-editor can have an overall impression of ˆ y, which is the mapping of x in the target domain, and then keep refining on it, while the standard one cannot. As shown in Figure 1, after using posteditor, the snow are erased for Winter→Summer translation and the green parts are enveloped with white frost.

2 RelatedWork

The generative adversarial networks [Goodfellow et al., 2014] (briefly, GAN) has enabled significant progress in unsupervised image-to-image translation [Zhu et al., 2017; Choi et al., 2018]. There are two essential elements in a GAN: a generator, used to map a random noise to an image; and a discriminator, used to verify whether the input is a natural image or a faked image produced by the generator.

Unsupervised image-to-image translation is an important application of GAN, which means to map an image from one domain to another and has received considerable attention recently. According to the number of domains involved in the translation, the related work can be categorized as twodomain translation and multi-domain translation.

2.1 Two-Domain Translation

Let Xs and Xt denote two different image domains. Our target is to learn a mapping f : Xs ↦→ Xt. A common technique adopted in two-domain translation is the conditional GAN, who have made much progress recently [Mirza and Osindero, 2014; Isola et al., 2017]. In these frameworks, the input image is compressed into a hidden vector via a series of convolutional layers and then converted to the target domain by several transposed convolutional layers. The generated images will be fed into the discriminator to ensure the generation quality. Additional input like random noise [Isola et al., 2017], text [Reed et al., 2016] can also be included.

Inspired by the success of dual learning in neural machine translation [He et al., 2016; Wang et al., 2019], learning two dual mappings f : Xs ↦→ Xt and g : Xt ↦→ Xs together are introduced [Kim et al., 2017; Zhu et al., 2017; Yi et al., 2017; Lin et al., 2018; Mejjati et al., 2018] to image translation. CycleGAN [Zhu et al., 2017] is one of the most cited work with such an idea: Given an x ∈ Xs, it is first mapped to

ˆ y by f(x) and then mapped back to ˆx by g(ˆ y). The cycleconsistent loss ‖x- ˆx‖1 is equipped to minimize the distance between x and ˆx, allowing f and g to obtain feedback signal from its counterpart. Similar idea also exists in DualGAN [Yi et al., 2017] and DiscoGAN [Kim et al., 2017].

2.2 Multi-Domain Translation

Great performances were achieved in dual-domain translation. However, while applying in multi-domain translation, training models for each pair of domains incurs much more resource consumption. To alleviate this limitation of scalability, several works extend dual-domain translation to multidomain translation by learning relationships among multiple domains in an unsupervised fashion [Choi et al., 2018; Liu et al., 2018; Anoosheh et al., 2018; Pumarola et al., 2018]. Instead of employing a generator and a discriminator for each domain, StarGAN [Choi et al., 2018] learns all mappings f with only one generator and one discriminator.

Although deliberation learning is not widely studied in image generation, it has been used in many natural language processing tasks including neural machine translation [Xia et al., 2017], grammar check [Ge et al., 2018], review generation [Guu et al., 2018]. It is beneficial to introduce this idea into image generation tasks.

3 Framework

In this section, we introduce the framework of deliberation learning for image translation, including both two-domain translation (based on CycleGAN) and multi-domain image translation (based on StarGAN).

3.1 Two-Domain Translation

Given two image domains Xs and Xt, the architecture of our proposed deliberation network achieving Xs to Xt translation is shown in Figure 2. There are three components: an encoder Es, a decoder Gt 1 , and a post-editor G∗t . The three components work together to achieve the translation, which is shown in Eqn. (1): for any x ∈ Xs,

1 We use G· instead of D· to represent the decoder, in order to avoid confusion with the discriminator D·.

The y∗ is used as the output of the translation. Note that ˆ y is the output of the conventional models like CycleGAN and DualGAN, without deliberation included. To generate y∗ , the information from both raw input x and the first-round output ˆ y are leveraged, carried by hx and hˆ y, and thus leading to better translation results.

The reason we use one encoder Es to encode both x and ˆ y is that, in standard CycleGAN, there is an identity mapping loss ‖Gt(Es(y)) - y‖1, y ∈ Xt. That is, an encoder can naturally encode the images from the target space. To reduce memory cost, we reuse the encoder. We found using two encoders will not bring much additional gain, see Section 4.3.

Following the common practice in image translation, we apply the cycle consistency loss between the translation of two domains as well as the adversarial loss. To achieve Xt → Xs translation, another groups of encoder EB, decoder GA and post-editor G∗A are needed, which work as follows: for any y ∈ Xt,

For ease of reference, let f ∗ denote Xs → Xt translation with Eqn. (1) and g∗ denote Xt → Xs translation using Eqn. (2). Note that Gt ◦Es and Gs ◦Et can also achieve the above two translations, we ◦ denotes the cascade of functions. Denote Gt ◦ Es as f and denote Gs ◦ Et as g respectively.

To stabilize the training, the Es, Et, Gs and Gt are pre-trained following standard CycleGAN until convergence. Only G∗s and G∗t are updated. An empirical study is shown in Section 4.3. With a little bit confusion, in the next context, Xs and Xt also refer to the training datasets of images in the source and target domains.

Cycle consistency loss. Following the common practice in image translation, the cycle consistency loss is defined as

where |Xs| refers to the number of images in Xs, and so does |Xt|. Both f ∗ and g∗ jointly work to minimize the reconstruction loss, which is exactly what is used standard CycleGAN.

Adversarial loss. To force the generated images belonging to the corresponding domains, we need to use the adversarial loss. Define Ds and Dt as two discriminators of domain Xs and domain Xt, which can map an input image to [0, 1], indicating the probability that the input is a natural image in the corresponding domain. The adversarial loss is

Therefore, the training objective functions for f ∗ and g∗ is to minimize

while the Ds and Dt will work on maximize ℓadv. In experiments, we fix λ as 10 following [Zhu et al., 2017].

3.2 Multi-Domain Translation

Several works target at image translation among multiple domains [Choi et al., 2018; Liu et al., 2018]. Our proposed deliberation learning framework also works for this setting.

Let X1, X2, · · · , XN be N domain of interests (N ≥ 2). Our target is to achieve N(N - 1) mappings among the N image domains. It is impractical to learn so many mappings, especially when N is large. An light-weight way is to use a StarGAN like structure, where there is only one encoder, one decoder and one discriminator. Adapted to the deliberation learning framework, there are one encoder E, one decoder G, one post-editor G∗ and a discrimiator D. Each image domain has a learnable embedding ti i ∈ [N], representing the domain characteristic. The E can map images from any domain to hidden representations conditioned on the embedding; G and G∗ can map the hidden representation guided by the domain embedding to the target space. Take the mapping from Xi to Xj as an example (i ̸= j): for an x ∈ Xi,

where concat represents padding the second input to the first one along the last dimension. y∗ is eventually used as the output of x in Xj. In this case, for ease of reference, denote the translation from Xi to Xj following Eqn. (6) as f ∗ i,j . Correspondingly, the generation function based on G◦E is denoted as fi,j . For multi-domain translation, E and G are pre-trained and then fixed too.

Similar to two-domain translation, the training loss of multi-domain translation consists of two parts too: the cycle consistency loss and adversarial loss. The training process of multi-domain deliberation is shown as follows:

1. Randomly choose two different domains Xi and Xj, where i ̸= j; randomly sample two batches of data Bi ⊂ Xi and Bj ⊂ Xj;
2. Formulate the cycle consistency loss as follows:

1. The discriminator slightly differs from those in twodomain translation. It consists of two parts: Dsrc, which is used to justify whether the input is a natural image; Dcls, which is used to verify which domain the image belongs to. Dsrc and Dcls share the basic architecture

expect for the last few layers. The adversarial loss is

f ∗ ·,· will work on minimizing ℓadv while the Dadv and Dcls will try to enlarge it. When optimizing the discriminators, the Dcls is fixed.

1. Minimize ℓcyc + λℓadv on Bi and Bj, where λ = 10. Repeat step (1) to (4) until convergence.

Compared with two-domain deliberation learning, each component of our model in multi-domain setting has to deal with images from different domains, while that in twodomain setting only works for two-domain.

3.3 Discussion

The idea of deliberation learning also exists in super resolution (briefly, SR), whose task is to convert a lowresolution image to a high-resolution one. A bicubic interpolation [Dong et al., 2014] is first applied to the low-resolution image, then followed by a neural network to reconstruct the high-resolution one. Indeed, the task of SR itself and our framework share the high-level idea, but there are still several differences: (1) Image translation covers at least two domains with different semantics, while SR works on one domain only; (2) When making deliberation, our framework takes the information from two domains as input and further deliberates them by a post-editor. In comparison, SR conducts one-pass operation, where the interpolated image is directly used in the subsequent module.

Another work leveraging deliberation learning is [Xu et al., 2018], which attacks the text-to-image problem: the images are generated from low-resolution to high-resolution, with multi-modal loss as constraints. Different from our work, our model is optimized in an fully unsupervised manner, and what we focused is to polish a generated image (with canonical output resolution) to a better one.

4 Application to Two-Domain Translation

For two-domain translation, we work on four datasets to verify the effectiveness of our algorithm.

4.1 Settings

Tasks. We select four tasks evaluated in CycleGAN [Zhu et al., 2017]: semantic Label↔Photo translation on Cityscapes dataset [Cordts et al., 2016], Apple↔Orange translation, Winter↔Summer translation, and Photo↔Paint translation.

Model architecture. For the encoder, decoder and the discriminator, we adopt the same architectures as those in CycleGAN for consistency. In addition, we need two posteditors G∗s and G∗t . We split the generator of CycleGAN into two components: The first one serves as the encoder in our scheme, which contains two stride-2 convolutional layers and four residual blocks. The remaining part serves as a decoder, which contains five residual blocks and two 1 2 -strided convolutional layers. Therefore, the total number of layers are consistent with CycleGAN. The architecture of post-editor is the same as the decoder in CycleGAN as we introduced before. For the discriminator, we directly follows CycleGAN.

Implementation details. We follow the offical CycleGAN 2 to implement our scheme in PyTorch. To stabilize the training, We first pre-train the Es, Gt, Et and Gs using standard CycleGAN code until convergence. After that, we start to train G∗s and G∗t . We use Adam with initial learning rate 2×10-4 to train the models for the first 100 epochs. Then we linearly decay the learning rate to 0 in the next 100 epochs.

2 https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

Metrics. Fr´echet Inception Distance (briefly, FID) is first proposed by [Heusel et al., 2017], recently becomes a commonly adopted approach to evaluate generative models [Lucic et al., 2018; Brock et al., 2019]. For FID measurement, the generated samples and the real ones are first mapped into feature space by an Inception-v3 model [Szegedy et al., 2016]. Then Fr´echet distance between these two distributions is calculated to obtain FID score. The authors demonstrated that FID score has a reasonable correlation with human judgment [Heusel et al., 2017]. Smaller FID scores indicate better translation qualities. In addition, we follow [Zhu et al., 2017] and provide FCN score [Isola et al., 2017] 3 in our framework analysis for fair comparison, including per-pixel accuracy (Pixel Acc.), per-class accuracy (Class Acc.) and mean class Intersection-Over-Union (Class IOU). Higher accuracies indicate better translation qualities.

4.2 Results

Qualitative evaluation. Figure 3 shows three groups of translation results on Label↔Photo (the first two rows), Apple↔Orange (the second two rows) and Paint↔Photo (the last two rows). In general, deliberation learning can: (1) Generate images with rich details. (2) Correct many failure cases (e.g.missing items, flawed translation, etc.) generated by CycleGAN. (3) Make the generated images more realistic.

Quantitative evaluation. FID scores of four datasets are shown in Table 1. We can see that deliberation learning significantly surpasses the baseline across all datasets.

4.3 Framework Analysis

We carry out detailed analysis of our proposed framework. We implement another six settings on Label→Photo translation and the scores are listed in Table 2.

(1) To verify whether different ways to combine the features from the encoder and decoder will influence deliberation quality, we concatenate the two features hx and hˆ y in Eqn. (1) instead of adding them. (Ours-1)

3 We directly use the code and pre-trained FCN model in https: //github.com/phillipi/pix2pix/tree/master/scripts/eval cityscapes.

1. To verify whether better accuracies are brought by larger models, we increase the depth of decoder as double, i.e.ten residual blocks. (Ours-2)
2. To verify whether we need two different encoders to encode two different outputs, we try to use two encoders to encode x and ˆ y separately. (Ours-3)
3. To verify whether we need identity loss, we remove it from our framework. (Ours-4)
4. To verify whether we need to fix encoder and decoder, we update all parameters in the encoder, decoder and post-editor. (Ours-5)
5. To verify whether we need to fetch raw input for the posteditor, we only feed the post-editor with hˆ y. (Ours-6)

For fair comparison, we also provide the FCN scores in Table 2. We list the existing results of CoGAN [Liu and Tuzel, 2016], BiGAN [Donahue et al., 2017; Dumoulin et al., 2017], standard CycleGAN [Zhu et al., 2017] and pix2pix [Isola et al., 2017]. Note that pix2pix is trained in a supervised way.

5 Application toMulti-Domain Translation

In this section, we conduct our deliberation learning on multidomain image translation. We also give qualitative and quantitative analysis on the performance of our scheme.

5.1 Settings

Tasks. We used the publicly available CelebA dataset [Liu et al., 2015] for facial attributes translation. CelebA contains 10, 177 number of identities and 202, 599 number of facial images. The original images are cropped to 128 × 156. The test set is randomly sampled (2, 000 images) and the remaining images are used for training. For directly comparison with StarGAN [Choi et al., 2018], we perform our experiments on the same three attributes: hair color (black, blond and brown), gender (male / female) and age (young / old).

Model architecture. For fair comparison, we adopt the same architectures as StarGAN [Choi et al., 2018] for the encoder and decoder. We additionally need one post-editor G∗ for our method. Similar to two-domain image translation, we split the generator of StarGAN into encoder and decoder. We adopt the same architecture of decoder for the post-editor G∗ . Different from conventional GAN [Goodfellow et al., 2014], the discriminator is enhanced with the ability of distinguishing which domain the translated images belong to.

Implementation details. Our implementation based on official code of StarGAN 4 . Similar to CycleGAN, we first pretrain the Es and Gt with standard StarGAN until convergence. The pre-trained parameters is loaded as initial model to optimize G∗ . We use Adam with initial learning rate 1 × 10-4 to train the models for the first 100, 000 iterations. Then we linearly decay the learning rate to 0 in the next 100, 000 iterations. The batch size is set to 16.

Metrics. To evaluate qualities of generated images quantitatively, we follow [Choi et al., 2018] to train a classification model on three facial attributes we used (i.e.hair color, gender and age). We directly adopt the model architecture of discriminator used in StarGAN for the classifier and use the same training set as our image translation models. Higher classification accuracies on the translated facial attributes indicates the better translation qualities.

5.2 Results

Qualitative evaluation. We provide facial attributes translation in Figure 4. Generally, deliberation learning effectively boost the performance of StarGAN on facial attributes, making the generated images more natural and indistinguishable from the real images. From Figure 4, we can observe that, deliberation learning successfully generates facial images with rich details, especially on semantic regions like eye, cheek etc. In addition, in many cases, StarGAN suffers from color shifting, while our scheme correct this fault and preserve the reality without impact on background or hair color.

4 https://github.com/yunjey/stargan

Quantitative evaluation. The quantitative evaluation of three facial attributes translation are illustrated in Table 3. All classification accuracies are measured on translated images with our pre-trained facial attributes classification model. In terms of classification accuracies, our method surpasses the StarGAN baseline by 3.11%, 5.79% and 2.16% points for hair color, gender and age respectively.

6 Conclusion and FutureWork

In this paper, we introduced the concept of deliberation learning for image translation, which shares high-level sense with human behaviors: reviewing and keeping polishing. We implemented deliberation learning on image-to-image translation. The experimental results demonstrate that our method generates more natural images and preserves crucial details.

There are many interesting directions to explore in the future. First, we will apply the idea of deliberation learning to more tasks like supervised image classification and segmentation. Second, deliberation learning can be iteratively used to keep polishing an image to get a better output. Third, we will study how to speed up the algorithm in the future.

Acknowledgments

This work was supported in part by NSFC under Grant 61571413, 61632001. We thank all the anonymous reviewers for their valuable comments on our paper.