Eliminating the Cross-Domain Misalignment in Text-guided Image Inpainting

1 Introduction

Image inpainting aims to restore damaged images to make them appear realistic and harmonious [Criminisi et al., 2003; Barnes et al., 2009]. The advent of deep learning [Liu et al., 2018; Cao and Fu, 2021; Vaswani et al., 2017; Li et al., 2022a; Suvorov et al., 2022] has brought significant advancements in related fields. Nevertheless, image inpainting, being an ill-posed task, yields unpredictable outcomes that may not align with user expectations. As a response, text-guided image inpainting has emerged [Zhang et al., 2020b; Zhang et al., 2020a], empowering users with control over restoration through textual descriptions, resulting in customized outcomes (Figure 1). Recently, Denoising Diffusion Models [Ho et al., 2020; Rombach et al., 2022; Zhang et al., 2023b], especially the Stable Diffusion (SD)

∗Corresponding author

family, have demonstrated outstanding performance across tasks involving image generation, editing, and inpainting.

However, we find that most text-guided image inpainting methods primarily focus on the semantic alignment between the generated results and the provided textual prompts, yet pay little attention to the authenticity and correctness of the generated images. Therefore, when the task of text-guided image inpainting necessitates the dual objectives of adhering to semantic constraints while also ensuring the creation of visually cohesive images, the majority of current algorithmic models encounter considerable challenges. To a significant extent, these challenges arise from the substantial disparity and misalignment that exists between the guidance provided by textual semantic cues and the fine-grained pixel-level information. These challenges manifest concretely as follows: (a) Structural Inconsistency: Despite the semantics of the inpainted image being roughly correct, there could be artifacts at the boundaries of masks, as well as fragmentation of object structures. This problem results from the emphasis on synthesizing content based on text prompts while disregarding the preserved structural information present in the original image. (b) Texture Detail Disparity: The reconstructed areas may lack critical high-frequency information, leading to distinct clarity disparities when compared to adjacent retained regions. These issues stem from the underlying misalignment between the high-level semantic information conveyed through textual descriptions and the inherent low-level pixel information in the images.

In this paper, we propose a Structure-Aware Inpainting Learning scheme and an Asymmetric Cross Domain Attention to address the aforementioned misalignment. First, we leverage a pre-trained ControlNet branch as a teacher to add structure guidance during the training process. Specifically, we leverage the edge features extracted from the training image serving as an intermediate modality connecting semantic features of the text prompt with the low-level information of the image. The pre-trained ControlNet branch guides the training process of the inpainting network by supervising the weight updates of the decoding stage, which directs the network's focus toward capturing the overarching semantic structure implicitly, yielding results that exhibit integrated structural coherence. Within this architecture, we introduce an Asymmetric Cross Domain Attention module, where the masked input is mapped to the frequency domain through Fourier transformation. Subsequently, high-pass filtering is employed to extract high-frequency components before computing the similarities of the objective in the image domain with features in the frequency domain and the textual domain, respectively. This process facilitates the fusion of highfrequency image information with text features, which dynamically influences the generated results to intricately restore texture details while ensuring semantic correctness. The contributions of this work are summarized as follows:

• We devise a Structure-Aware Inpainting Learning mechanism, integrating a structure-guided pre-trained controlnet branch, to direct the network to consciously restore comprehensive semantic information.
• We introduce an Asymmetric Cross Domain Attention module that employs attention on high-frequency features and textual prompts, enabling the model to emphasize and incorporate the texture information.
• Experimental results conducted on two prominent datasets MS-COCO and Open Images demonstrate the superiority of our method over the state-of-the-art textguided inpainting techniques.

2 RelatedWork

2.1 Image Inpainting

A multitude of deep learning-based methodologies have been explored for tackling image inpainting tasks, frequently adopting frameworks such as Generative Adversarial Networks (GANs) [Goodfellow et al., 2020] or encoder-decoder architectures. Certain approaches have decomposed the task into multiple stages [Yu et al., 2019; Huang and Zhang, 2022], employing sequential networks to gradually restore coarse elements and intricate textures within images. To ensure the coherence of the whole image, attention mechanisms have gained prominence to align the generated content with the surrounding context. RFR [Li et al., 2020] adopts an iterative strategy to propagate inpainting and calculate attention score from mask edges inward. LBAM [Sun et al., 2021] introduces bidirectional attention maps for enhanced longrange contextual modeling. However, the filled content often lacks control and exhibits significant randomness, leading to uncertain quality of reconstructed images that might not fulfill user expectations in practical applications.

2.2 Text-guided Image Inpainting

To offer more controlled restoration options, text-guided image inpainting methods enable users to input both masks and accompanying text, facilitating the reconstruction of missing content based on textual cues. LSAI [Xie et al., 2022] employs semantic alignment to repair patches of the damaged image. ALMR [Wu et al., 2021] employs adversarial learning and introduces a dual-attention module for both coarseand fine-grained stages, spanning from semantic to textural image recovery. MMFL [Lin et al., 2020] introduces a multimodal fusion approach that leverages both textual and image information. However, these methods often fall short in terms of image diversity, struggling to provide ample choices when confronted with low-quality generations.

2.3 DiffusionModel in Image Inpainting

The diffusion model has emerged as a prominent framework for generating high-quality images in a controlled manner. Originating from DDPM [Ho et al., 2020], this paradigm involves iteratively diffusing noise levels to transform latent representations into images. By modulating noise diffusion, this approach elegantly balances the trade-off between image generation and control. The inherent attributes of diffusion models also find resonance in the domain of image restoration. SDM [Li et al., 2022b] introduces a spatial diffusion model to restore images with large holes, while RePaint [Lugmayr et al., 2022] employs fine-tuning through resampling based on pre-trained DDPM, enhancing image realism. Furthermore, EditBench [Wang et al., 2023], Blended Diffusion [Avrahami et al., 2022], GLIDE [Nichol et al., 2022], and SmartBrush [Xie et al., 2023] propose diffusionbased text-guided image editing methods, producing controlled high-quality images that are also applicable to inpainting tasks. To maintain the high performance and variability of DDPM under limited computational resources, LDM (Stable Diffusion) [Rombach et al., 2022] incorporates a pretrained autoencoder, enabling the diffusion model to operate on smaller latent feature maps. Building upon this, ControlNet [Zhang et al., 2023b] introduces additional control branches, exclusively training control branches while keeping the weights of Stable Diffusion branches fixed, offering a more flexible approach to image generation.

These methods inherit the iterative nature of diffusion, enabling them to adaptively restore missing regions while adhering to textual cues. However, while augmenting the image restoration with guidance has its merits, it also presents new challenges. Due to gaps between different modalities, there can be misalignment or even conflicts between the textual information and the preserved image details.

3 Preliminary: DiffusionModel

In this paper, we employ the diffusion model as our generator. The Denoising Diffusion Probabilistic Models (DDPM) consist of a forward process and a reverse process, with the former being also referred to as the diffusion process.

The diffusion process involves incrementally adding Gaussian noise to the data until it becomes random noise. For the original data x0 ∼ q (x0), each step of the diffusion process generates a sample xt ∼ q (xt|x0) by iteratively adding Gaussian noise to the previous step's data over T steps, ultimately resulting in a random noisy image xt ∼ q (xt|x0) that completely loses the original information:

where {βt}Tt=1 represents the variance at each noise addition step.

Given αt = 1 - βt and ¯αt = ∏ti=1 αi, by reparameterization we have:

The variable xt can be regarded as a linear combination of the original data x0 and random noise ϵ, where √ ¯αt and √1- ¯αt act as the blending coefficients, with their squares summing up to 1.

The reverse process, on the other hand, is a denoising process. It starts with a noisy image xT ∼ N (0, I) at step T and gradually denoises it using known real distributions q (xt-1|xt) for each step:

Here, p (xT ) = N (xT ; 0, I), and pθ (xt-1|xt) represents a parameterized Gaussian distribution, with its mean and variance determined by the trained networks µθ (xt, t) and Σθ (xt, t). After reparameterizing , µθ can be expressed as:

where ϵ ∼ N (0, I), and ϵθ is a fitting function based on neural networks, implying the use of predicted noise ϵ instead of predicted mean. The simplified optimization objective is:

4 Method

Given an original image Igt, a binary mask m, and a text prompt c, the masked image Iinput is obtained by overlaying the mask on the original image.

The corresponding latent feature z0 is derived through encoding. Passing through the overall network yields the latent feature map ˜ z, which is subsequently decoded into the final output Ioutput. The generation of ˜ zt-1 and the composition with restored region in each denoising step is as follows:

In this work, we introduce a novel Structure-Aware Inpainting Learning scheme that offers semantic guidance to the inpainting network, enabling the generation of structurally coherent images. Furthermore, we devise an Asymmetric Cross Domain Attention module that blends high-frequency texture details from the preserved region of the image with text embeddings, which dynamically constrains the generation of texture details.

4.1 Structure-Aware Inpainting Learning Scheme

To narrow the disparity between the semantic content of the text and the pixel-level information in the image, we introduce the use of image structure as an intermediary between the two domains. This strategy associates high-level textual features with low-level image features, aiding the inpainting network to implicitly focus on global image semantics during training, thus leading to the generation of cohesive images in accordance with textual descriptions.

The training strategy is shown as Figure 2. (a) depicts a simplified diagram of the latent diffusion model, where images are encoded into the latent space through an AutoEncoder, reducing the feature map dimensions. Training Cross Attention of the diffusion model then occurs within the latent feature space, utilizing a U-net architecture for the generator network. The output of the diffusion model is finally mapped back into image space through a decoder. Figure 2(b) illustrates ControlNet with the LDM. It introduces a control branch between the input x and output blocks of the diffusion model. The condition signal added with noise z is propagated through input blocks, and each input block is processed with a zero-initialized convolution, effectively connecting with the main LDM. This enables ControlNet to accept various types of conditional signals, such as Canny edge and segmentation maps, thereby flexibly guiding the generation process of the latent diffusion model without compromising the performance of the pre-trained Stable Diffusion model.

Figure 3: The Asymmetric Cross Domain Attention mechanism in trainable input blocks.

Figure 2(c) showcases our training scheme. We employ a pre-trained stable diffusion model as the backbone of the generation process. The inpainting branch receives the concatenation of the masked image and mask as inputs, which are then passed through trainable input blocks and zero convolutions before being fed back into the main network. Notably, a pre-trained ControlNet, employing structure information as a condition, is used as supervision during network training. This is achieved by computing the feature differences of each output block in the inpainting branch and the ControlNet branch using a soft loss mechanism, implicitly guiding the network to focus more on image structural information. "Zero Conv" refers to 1×1 convolution initialized with zero, while "Structure Conv" stands for the zero-initialized convolution that has already been activated. The computation of the soft loss involves employing an intuitive mean error loss:

where fcontrol is the feature map during the structure convolution block of the ControlNet, and finpaint is that of the inpainting network branch.

4.2 Asymmetric Cross Domain Attention

In the context of inconsistencies between regions of image reconstruction, beyond discrepancies in content and structure, differences in image clarity and resolution also exist. Many existing text-guided image inpainting algorithms predominantly focus on adhering to textual prompts while disregarding the coherence between the reconstructed objects and the preserved regions. To address this issue, we further propose a network architecture based on Asymmetric Cross Domain Attention, facilitating the interaction between information from the image's spatial domain, frequency domain, and the textual domain, to integrate the impact of high-frequency image texture information into text-guided object generation. The Asymmetric Cross Domain Attention calculation is as follows:

where Attn denotes Attention(q, k, v, k′ , v′ ), d represents the dimension of the input vector, and b is the bias.

The overall framework of the network is illustrated in Figure 3. The text prompt is encoded through CLIP and successively introduced into the transformer blocks within the U-net structure of the diffusion model. This process involves the integration of Asymmetric Cross Domain Attention with the high-frequency feature map of the image encoded into the latent feature space, as depicted on the right side of Figure 3. The high-frequency feature map Ih is obtained by transforming the masked image into the frequency domain using Fourier transformation, followed by extraction via a high-pass filter with a radius of 4. Then we get the feature ffreq in frequency domain by:

A very green hillside under the golden sunset.

A cat crouched over sitting by a red brick wall.

tude and phase, respectively. We concatenate the amplitude spectrum and phase spectrum of the high-frequency component along the channel dimension, presenting them in a frequency domain form for input into the Asymmetric Cross Domain Attention module.

The overall optimization objective is

where λ is a hyper-parameter controlling the weight of the soft loss and is set to 0.1 in our experiments.

5 Experiments

5.1 Experiment Settings

We employ our proposed Structure-Aware Inpainting Learning (SAIL) approach for image inpainting under the architecture of ControlNet and it is finetuned from Controlnet - v1.1 - InPaint Version. The learning rate is set at 5e-5 , and the batch size is configured to be 4. Each experiment necessitate the utilization of one A100 GPU. After a series of systematic experiments, it has been shown that choosing Canny edge as the structure information of the auxiliary ControlNet branch during training can effectively correlate high-level semantic information with low-level pixel details. Therefore, all subsequent experimental setups utilize Canny edge as the condition for the ControlNet branch.

Datasets. We fine-tune our model on the standard MSCOCO dataset [Lin et al., 2014], which comprises over 100k images in the training set. For testing, we utilize 5k imagetext pairs from the MS-COCO validation set. To assess the robustness of our model to diverse data, we further validate its performance on 1.5k images from the Open Images dataset [Kuznetsova et al., 2020].

Baselines. We select six state-of-the-art approaches based on the diffusion model as our baselines: GLIDE [Nichol et al., 2022], Blended Diffusion [Avrahami et al., 2022], Stable Diffusion inpainting [Rombach et al., 2022], ControlNet inpainting [Zhang et al., 2023b], Uni-paint [Yang et al., 2023], and CoPaint [Zhang et al., 2023a].

Metrics. We compare the proposed method with other approaches on four widely-adopted metrics, namely PSNR, SSIM, Mean l1, and LPIPS, to evaluate the clarity, structure similarity, and diversity of the results comprehensively.

5.2 Qualitative Comparison

We conduct image inpainting tests with various mask ratios on both the MS-COCO dataset and the Open Images dataset. As depicted in Figure 4, for custom masks and prompt inputs, our proposed method generates high-quality images that align both internally and externally with the textual descriptions. In contrast, GLIDE and Blended Diffusion often fail to generate content aligned with textual descriptions. While the generated results of Stable Diffusion, ControlNet, and Unipaint incorporate textual descriptions, the overall semantic coherence of the images may be compromised, resulting in artifacts or noticeable boundaries along the mask edges. CoPaint is the only comparative method that performs inpainting without text guidance. It can produce reasonable results but falls short in meeting user-customized content generation demands. Figure 5 showcases the comparative results of our approach against other methods on the MS-COCO dataset using randomly generated masks. It is evident that our method consistently produces coherent and semantically sound clear images, both for wide and narrow masks. In contrast, other methods struggle to maintain the structural consistency when dealing with objects that are partially occluded, such as the cat in the first row of the figure. Note that Uni-paint, being a task-specific training-free approach, relies solely on the generative capacity inherited from the pre-trained stable diffusion model. While it performs well with regularly shaped masks, it exhibits a lack of robustness in situations involving free-form masks or cases where the mask does not entirely cover the object.

5.3 Quantitative Comparison

We conduct quantitative evaluations on both the MS-COCO and Open Images validation sets, selecting 5k and 1.5k images respectively. We apply same irregular masks to different methods. Table 1 presents numerical outcomes with various mask ratios. As observed, the images generated by our proposed approach outperform other SOTA methods across all metrics, objectively showcasing our method's superiority in terms of image structure, clarity, and diversity.

5.4 Ablation Study

In this section, we meticulously validate the efficacy of the proposed Asymmetric Cross Domain Attention module (ACDA) and the Structure-Aware Inpainting Learning (SAIL) scheme. The baseline is the naive fine-tuning scheme as ControlNet with masked image as condition. The notation "w/ ACDA" signifies the adoption of solely the ACDA module without utilizing the pre-trained ControlNet model with Canny edge as structural guidance, while "w/ SAIL" denotes the exclusive use of the SAIL training mode, substituting the ACDA module with a standard cross attention mechanism.

As discerned from Figure 6, employing only the ACDA module yields images with distinct texture details, but semantic structural inaccuracies are observed. Conversely, employing solely the SAIL strategy enhances the overall structural coherence of the images, but at the expense of losing highfrequency information in the reconstructed missing regions. Table 2 objectively quantifies the effectiveness of our proposed network architecture and training methodologies.

5.5 User Study

To investigate human perception of generated results, we conducted user study with a random selection of 29 participants. A total of 348 answers for several image-text pairs were presented. They were asked two questions for each pair, focusing on the generated images from GLIDE, Blended Diff., Stable Diff., ControlNet, Uni-paint, CoPaint and our method: 1) Which image best matches the textual description? 2) Which result appears the most realistic and natural?

The results of the study, as shown in Figure 7, demonstrate significant advantages of our method in terms of textual fidelity and image realism.

6 Conclusion

In this paper, we propose Structure-Aware Inpainting Learning scheme for text-guided image inpainting, aiming at enhancing the efficacy of conditioned image synthesis model based on LDM for inpainting tasks. By leveraging Canny edge as intermediate modality, our method offers crucial structural guidance to the inpainting network training procedure. Moreover, we introduce an Asymmetric Cross Domain Attention mechanism within the diffusion model to achieve harmonious alignment between and high-frequency image textures. This strategic fusion ensures both semantic consistency in composition and retention of fine-grained texture details. Experimental results validate the capability of our approach to reconstruct globally semantically coherent high-quality images across different datasets, achieving remarkable performance compared to state-of-the-art diffusion model-based methods.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 62122060 and the Special Fund of Hubei Luojia Laboratory under Grant 220100014.