Generative Image Inpainting with Submanifold Alignment

1 Introduction

Given a corrupted image where part of the image is missing, image inpainting aims to synthesize plausible contents that are coherent with non-missing regions. Figure 1 illustrates such an example, where Figure 1a and 1b show two original (uncorrupted) images and their corrupted versions with missing regions respectively. The aim is to restore the missing regions of corrupted images in Figure 1b, such that the restored regions contain contents that make the whole image look natural and undamaged (high structural and textural consistency with the original images in Figure 1a). With the help of image inpainting, applications such as restoring damaged images or removing blocking contents from images can be realized.

∗Corresponding authors.

Figure 1: Comparison between the state-of-the-art GMCNN and our proposed model.

Generative Adversarial Networks (GANs) are powerful models for image generation [Goodfellow et al., 2014; Radford et al., 2016]. From the perspective of GANs, image inpainting can be viewed as a conditional image generation task given the context of uncorrupted regions [Pathak et al., 2016; Iizuka et al., 2017; Yang et al., 2017; Yeh et al., 2017; Yu et al., 2018; Wang et al., 2018]. A representative GANbased inpainting model is Context Encoder [Pathak et al., 2016], where a convolutional encoder-decoder network is trained with the combination of a reconstruction loss and an adversarial loss [Goodfellow et al., 2014]. The reconstruction loss guides to recover the overall coarse structure of the missing region, while the adversarial loss guides to choose a specific distribution from the results to promote natural-looking patterns. Lately, Wang et al. [2018] propose a generative multi-column neural network (GMCNN) for better restoration.

In spite of the encouraging results, the restored images by current GAN-based inpainting models may lead to structural or textural inconsistency compared with the original images. Specifically, current inpainting models may significantly alter the structure or the texture of the objects involved in the missing regions, and therefore the restored contents visually mismatch their surrounding undamaged contents. For example, in Figure 1c, the restored regions by the state-of-the-art inpainting model GMCNN [Wang et al., 2018] exhibit segmentation structure misalignment (part of the tower is missing) in the top image and texture mismatch (inconsistent with respect to 'beard' pattern) in the bottom image. This problem can be ascribed to the following two limitations.

First, GAN-based formulation has the ill-posed nature: the one-to-many mapping relationships between a missing region and its possible restorations. This can lead to sub-optimal inpainting results as in the above examples, especially when the inpainting model was not properly regularized for structural/textural consistency. The commonly used loss combination of a reconstruction loss and an adversarial loss, however, fails to address this limitation. The reconstruction loss mainly focuses on minimizing pixel-wise difference (very low-level), while the adversarial loss determines the overall similarity to real images (very high-level). Intuitively, mid-level (between the pixel and the overall realness levels) regularizations that can provide more explicit emphases on both structural and textural consistencies are needed for natural restorations.

Second, most existing inpainting models [Yang et al., 2017; Yu et al., 2018] adopt a two-stage "coarse-to-fine" inpainting process. Specifically, these models generate a coarse result at the first stage (which also suffers from the first limitation), then refine the coarse result at the second stage by replacing patches in the restored regions with their nearest neighbors found in the original patches. Due to the cosinesimilarity based searching of nearest patches, these models tend to produce repetitive patterns. Moreover, when structures in missing regions are originally different from those in the background, these models may significantly alter or distort the structure of the restored contents and end up with unrealistic restorations, for example in the top image of Figure 1c, filling missing part of the 'tower' with 'sky' background.

To address the two limitations, we propose to enforce the alignment (or closeness) between the local data submanifolds (or subspaces) around restored images and those around the original images, during the learning process of GAN-based inpainting models. Our intuition is that, a restored image or patch of the image will look more natural if the image or the patch is drawn from a data submanifold that is closely surrounding the original image or patch, that is, data submanifolds around the restored images/patches align well with those around the original images/patches. In this paper, we adapt an expansion-based dimensionality measure called Local Intrinsic Dimensionality (LID) to characterize the dimensional property of the local data submanifold (in deep feature space) around a reference image/patch. We further exploit LID to measure the submanifold alignment from a perspective of either images or local patches. This allows us to develop two different levels of submanifold alignment regularizations to improve GAN-based inpainting models: 1) an image-level submanifold alignment (iLID) for structural consistency, and 2) a patch-level submanifold alignment (pLID) for improving textural details. In summary, our main contributions are:

• We propose and generalize the use of Local Intrinsic Dimensionality (LID) in image inpainting to measure how closely submanifolds around restored images/patches align with those around the original images/patches.
• With the generalized LID measure, we develop two submanifold alignment regularizations for GAN-based inpainting models to improve structural consistency and textural details.
• Experiments show that our model can effectively reduce structural/textural inconsistency and achieve more accurate inpainting compared with state-of-the-art models.

2 RelatedWork

Traditional image inpainting models are mostly based on patch matching [Barnes et al., 2009; Bertalmio et al., 2003] or texture synthesis [Efros and Leung, 1999; Efros and Freeman, 2001]. They often suffer from low generation quality, especially when dealing with large arbitrary missing regions. Recently, deep learning and GAN-based approaches have been employed to produce more promising inpainting results [Pathak et al., 2016; Yang et al., 2017; Yeh et al., 2017; Iizuka et al., 2017; Yu et al., 2018; Liu et al., 2018; Song et al., 2018; Zhang et al., 2018; Wang et al., 2018]. Pathak et al.[2016] first propose the Context Encoder (CE) model that has an encoder-decoder CNN structure. They train CE with the combination of a reconstruction loss and an adversarial loss [Goodfellow et al., 2014]. Later models apply post-processing on images produced by encoder-decoder models to further improve their quality. Yang et al. [2017] propose one such model that adopts CE at its first stage, while at a second stage, they refine the inpainting results by propagating surrounding texture information. Iizuka et al. [2017] propose to use both global and local discriminators at the first stage, and Poisson blending at the second stage. Yu et al. [2018] propose a refinement network for post-processing based on contextual attention. These models mainly focus on enhancing the resolution of inpainted images, while ignoring the structural or textural consistency between the restored contents and their surrounding contexts. These models are all based on an encoderdecoder model for generating initial results, which can easily suffer from the ill-posed one-to-many ambiguity. This may produce suboptimal intermediate results that limit the effectiveness of post-processing at a later stage. Lately, Wang et al. [2018] propose a generative multi-column neural network (GMCNN) for better restoration of global structures, however, as we show in Figure 1c, it still suffers from the structural or textural inconsistency problem.

3 Dimensional Characterization of Local Data Submanifolds

We start with a brief introduction of the Local Intrinsic Dimensionality (LID) [Houle, 2017] measure for assessing the dimensional properties of local data submanifolds/subspaces.

3.1 Local Intrinsic Dimensionality

LID measures the rate of growth (e.g. expansion rate) in the number of data points encountered as the distance from a reference point increases. Intuitively, in Euclidean space, the volume of an D-dimensional ball grows proportionally to rD when its size is scaled by a factor of r. From the above rate of volume growth with distance, the dimension D can be deduced from two volume measurements as:

Transferring the concept of expansion dimension from the Euclidean space to the statistical setting of local distance distributions, the notion of r becomes the distance from a reference sample x to its neighbours and the ball volume is replaced by the cumulative distribution function of the distance. This leads to the formal definition of LID [Houle, 2017].

Definition 1 (Local Intrinsic Dimensionality)

Given a data sample x ∈ X, let r > 0 be a random variable denoting the distance from x to other data samples. If the cumulative distribution function F (r) is positive and continuously differentiable at distance r > 0, the LID of x at distance r is given by:

whenever the limit exists. The LID at x is in turn defined as the limit when the radius r → 0:

The last equality of Equation (2) follows by applying L'Hˆopital's rule to the limits. LIDF describes the relative rate at which the cumulative distance function F (r) increases as the distance r increases. In the ideal case where the data in the vicinity of x are distributed uniformly within a local submanifold, LIDF equals the dimension of the submanifold. In more general cases, LID provides a rough indication of the dimension of the submanifold containing x that would best fit the data distribution in the vicinity of x.

3.2 Estimation of LID

There already exists several estimators of LID [Amsaleg et al., 2015; Amsaleg et al., 2018; Levina and Bickel, 2005], among which the Maximum Likelihood Estimator (MLE) shows the best trade-off between statistical efficiency and complexity:

where k is the neighborhood size, ri(x;X) is the distance from x to its i-th nearest neighbor in X \{x}, and rmax(x; X) denotes the maximum distance within the neighborhood (which by convention can be rk(x; X)).

LID has recently been used in adversarial detection [Ma et al., 2018a] to characterize adversarial subspaces of deep networks. It has also been applied to investigate the dimensionality of data subspaces in the presence of noisy labels [Ma et al., 2018b]. In a recent work [Barua et al., 2019], LID was exploited to measure the quality of GANs in terms of the degree to which manifolds of real data distribution and generated data distribution coincide with each other. Inspired by these studies, in this paper, we develop the use of LID in image inpainting to encourage submanifold alignment for GAN-based generative inpainting models.

4 Proposed InpaintingModel with Submanifold Alignment

Given an input (corrupted) image x ∈ X with missing regions, an image inpainting model aims to restore the missing regions so that the output (restored) image z ∈ Z matches the ground truth (original) image y ∈ Y . Following previous works, we adopt GANs to generate the output image z. To address the structural and textural inconsistencies exist in current GAN-based inpainting models, we propose to enforce the restored images or local patches to be drawn from data submanifolds that align closely with submanifolds of the original images or patches. To achieve this, we develop the LID characterization of local data submanifolds to derive two levels of submanifold alignments: image-level and patch-level.

4.1 Image-level Submanifold Alignment

We propose Image-level Local Intrinsic Dimensionality (iLID) to measure how close data submanifolds around restored images are to those around the original images. For an original image, we define the iLID of its local submanifold by its neighbors found in GAN generated samples:

where, transformation φ represents the Lth layer of a pretrained network mapping raw images into deep feature space, ri(φ(y), φ(Z)) is the L2 distance of φ(y) to its i-th nearest neighbor φ(zi) in φ(Z), and kI is the neighborhood size.

Figure 2 illustrates a toy example showing how iLID(y; Z) works. In the left scenario, GAN learns a data submanifold (with blue points indicating samples Z drawn from this submanifold) that is closely surrounding the original image y and the iLID(y;Z) score is low, while in the right scenario, as the GAN-learned data submanifold drifts away from y the iLID(y; Z) score increases from 1.53 to 3.78. Note that, from the original LID perspective, the right scenario can be interpreted as a sample y lies in a higher dimensional space that is off the normal data submanifold. Revisit Eq. (5), assuming rmax > ri, as the distance between y and its nearest neighbors in Z increases by d > 0, then the term inside of the log function becomes (ri + d)/(rmax + d). Since,

iLID(y; Z) will increase with the increase of d, reflecting the alignment/closeness drift of local data submanifold from y.

Taking the expectation of iLID(y;Z) over all the original samples in Y gives us the regularization for image-level submanifold alignment:

Low LiLID(Y ; Z) scores indicate low average spatial distance between the elements of Y and their neighbor sets in Z. LiLID(Y ; Z) provides a local view of how well data submanifolds learned by a GAN model align with the original data submanifolds. By optimizing LiLID(Y ;Z), we can encourage a GAN generator to draw restorations from data submanifolds that are closely surrounding the original images, thus can avoid structural inconsistency between the restored image and the original image.

As computing neighborhoods for each y ∈ Y with respect to large sets Z can be prohibitively expensive, we use a batchbased estimation of LID: for one batch during training, we use the GAN generated samples as the Z for that batch to compute iLID score for each original sample in the batch. In other words, the LiLID(Y ;Z) is efficiently defined over a batch of samples (original or restored).

4.2 Patch-level Submanifold Alignment

To avoid local textural inconsistency, we further propose the Patch-level Local Intrinsic Dimensionality (pLID) based on feature patches in the deep feature space (rather than the raw pixel space). Feature patches were extracted on the feature map obtained by the same transformation φ as used for image-level submanifold alignment. Specifically, for an original image y ∈ Y and its corresponding restored image z ∈ Z with restored region zo , we extract one set P of 3 × 3 feature patches from the entire feature map φ(y), and one set Q of 3 × 3 feature patches only from part of the feature map φ(z) that is associated with region zo . Following similar formulation of iLID, we define pLID as:

where, p ∈ P is a patch in P and kP is the neighborhood size. Then, the patch-level submanifold alignment regularization can be defined as:

Optimizing LpLID(Y ; Z) enforces the GAN model to draw patches from data submanifolds that are closely surrounding the original patches, which helps improve texture consistency. Specifically, LpLID(p;Q) encourages p to receive reference from different neural patches in Q, thus can avoid repetitive patterns and maintain variety in local texture details. Similar to LiLID(Y ;Z) , we define the LpLID(Y ;Z) over a batch of original or restored samples during training.

4.3 The Overall Training Loss

The overall training loss of our model is a combination of an adversarial loss, a reconstruction loss and the two proposed regularizations:

where Ladv and Lrec denote the adversarial loss and reconstruction loss respectively, and parameters λI, λP and λA control the tradeoff between different components of the loss. Following recent works [Liu et al., 2018; Yu et al., 2018], we adopt the improved Wasserstein GAN [Gulrajani et al., 2017] as our adversarial loss:

where G and D denote the generator and the discriminator of our model respectively, x ∈ X is an input (corrupted) sample and z = G(x) is the GAN generated sample. The restored image for x is ˆx = tx+(1-t)G(x) with the mask t ∈ {0, 1} having zeros at the missing pixels and ones at the non-missing pixels, and we use the L2 reconstruction loss:

5 Experiments

We compare with four state-of-the-art inpainting models: CE [Pathak et al., 2016], GL [Iizuka et al., 2017], GntIpt [Yu et al., 2018] and GMCNN [Wang et al., 2018]. We apply the proposed training strategy on both CE and GL GANarchitectures, and denote their improved versions as CE+LID and GL+LID respectively. We evaluate all models on four benchmark datasets: Paris StreetView [Doersch et al., 2012], CelebA [Liu et al., 2015], Places2 [Zhou et al., 2017] and ImageNet [Russakovsky et al., 2015]. No pre-processing or post-processing is applied in all experiments.

We set λA to 0.01 as in [Yu et al., 2018; Iizuka et al., 2017]. Based on our analysis in Section 5.3, we empirically set λI = 0.01, λP = 0.1, kI = 8 and kP = 5 for our models. All training images are resized and cropped to 256 × 256. We choose conv4 2 of VGG19 [Simonyan and Zisserman, 2014] as the transformation φ. For all the tested models, we mask an image with a rectangular region that has a random location and a random size (ranging from 40 × 40 to 160 × 160). The size of a training batch is 64. The training takes 16 hours on Paris StreetView, 20 hours on CelebA and one day on both Places2 and ImageNet using an Nvidia GTX 1080Ti GPU.

5.1 Quantitative Comparisons

We compute the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity (SSIM) between a restored image and its ground truth for quantitative comparison as in [Yu et al., 2018; Wang et al., 2018]. As shown in Table 1, both of our proposed models CE+LID and GL+LID achieve better or comparable results to existing models across the four datasets, which verifies the effectiveness of submanifold alignment on different inpainting models. Since GL+LID demonstrates more promising results than CE+LID, we only focus on the GL+LID model in the following experiments.

5.2 Qualitative Comparisons

We show sample images in this section to highlight the performance difference between our models and the four state-ofthe-art models on the selected four datasets. As shown in Figure 3, CE [Pathak et al., 2016] can generate overall reasonable contents though the contents may be coarse and semantically inconsistent with surrounding contexts. For example, the input image of the first row shows an building that has a part of the window missing. CE can only generate coarse content with incomplete structures of the window and blurry details. This is due to the limitation of the original GAN model that only provides overall high-level guidance (whether the inpaining output is generally similar to the ground truth image) and lacks specific regularization. With the help of global and local discriminators, GL [Iizuka et al., 2017] can generate better details than CE but it still suffers from generating corrupted structures and blurry texture details. Although GntIpt [Yu et al., 2018] can restore sharp details, it can lead to structural or textural inconsistency so that the restored contents may be contradictory with the surrounding contexts. Since the first stage of GntIpt resembles the overall structure of GL, the coarse outputs from the first stage of GntIpt also have similar problems with GL. Moreover, the second stage of GntIpt (finding the closest original patch from the surroundings with cosine similarity for the restored content) tends to easily attach simple and repeating patterns onto the restored content, thus even worsens the above mentioned problem. Specifically in the first image, it fills the missing region with wall pattern ignoring the remaining window part at the bottom. GMCNN [Wang et al., 2018] proposes an improved similarity measure to encourage diversity when finding the patch match, however it only insists on single patch diversity and thus causes unexpected artifacts and noises in the restored content. In contrast, our model can generate more realistic results with better structural/textural consistency compared with the four state-of-the-art models.

5.3 Ablation Study

Parameter Analysis on iLID and pLID Regularizations

We further conduct ablation experiments for our model on Places2. Specifically, we investigate different combinations of iLID regularization and pLID regularization in the loss function by assigning λI and λP with different values in [0.001, 1]. An example of qualitative comparisons with different combinations of λI and λP is shown in Figure 4. When λI and λP are both set to 0, the restored image encounters both structural and textural inconsistency problem with part of the building missing and filled with background (sky) pattern. If we assign λI with non-zero values and remain λP as 0, the model becomes able to restore the overall structure of the building even if the texture details are noisy and blurry. When we assign λP with non-zero values and remain λI as 0, the restored results have sharper and more diverse texture details on local patch level, but they fail to be consistent with the structure of original image on overall image level. When we both assign λI and λP with non-zero values, we notice that the restored images are consistent on not only overall structure but also texture details with the surrounding context. From the visual results, we see that iLID regularization can increase structural consistency of restored contents, while pLID regularization focuses on patch-level alignment and thus benefits textural consistency. Quantitative comparisons with 4 groups of parameter setting are given in Table 2. We see that both iLID regularization and pLID regularization contribute to the model performance when using them alone. Moreover, the collaboration of these two different regularizations can achieve complementary effects to improve inpainting quality. We therefore empirically set λI = 0.01 and λP = 0.1 based on the ablation experiments.

The Effect of Different Neighborhood Sizes

We also investigate the effect of different neighborhood sizes on iLID (kI) and pLID (kP ) regularizations. For iLID regularization, different values of kI achieve similar results and empirically kI = 5 ∼ 10 strikes a better balance. For pLID regularization, we find that assigning kP with different values can bring variable texture diversity for results. Figure 5 shows that kP = 5 leads to more accurate texture details.

6 Conclusion

We studied the structural/textural inconsistency problem in image inpainting. To address this limitation, we propose to enforce the alignment between the local data submanifolds around restored images and those around the original images. Our proposed inpainting model utilizes a combination of two LID-based regularizations: an image-level alignment regularization (with iLID) and a patch-level alignment regularization (with pLID) during the training process. The experimental results confirm that our model achieves more accurate inpainting compared with the state-of-the-art models on multiple datasets. We note that our enhancement can be applied to any GAN-based inpainting model, and further analysis in this direction can be interesting future work.

Acknowledgements

The work is partially supported by the ARC grant DP170103174 and by the China Scholarship Council.