Stabilizing Holistic Semantics in Diffusion Bridge for Image Inpainting

1 Introduction

Image inpainting refers to reconstructing a high-quality image from an incomplete one, and it has a wide range of applications in many fields such as image editing [Li et al., 2021], artifact restoration [Quan et al., 2024], and object removal. It is an inverse problem with an ill-posed nature. To resolve this challenging issue, conventional algorithms [Criminisi et al., 2004; Komodakis and Tziritas, 2007;

∗Corresponding author.

Barnes et al., 2009] mainly utilize low-level visual assumptions to heuristically reconstruct the damaged regions or search and copy similar image patches from of the undamaged source image to fill the target region. However, due to limited feature representation, these methods typically struggle to generate accurate semantics within the hole. To this end, later works [Krizhevsky et al., 2012; Liu et al., 2021; Liu et al., 2024b; Suvorov et al., 2022; Zeng et al., 2022; Yu et al., 2018; Ko and Kim, 2023; Li et al., 2022; Yao et al., 2024; Peng et al., 2023] have attempted to design advanced components to enhance feature representations or introduce self-attention mechanisms into GAN-based [Goodfellow et al., 2014] conditional generative models, achieving better performance. Nonetheless, such strategies often lead to weak semantics correlation among various patches within the masked regions. To address this, some approaches [Nazeri et al., 2019; Dong et al., 2022; Liu et al., 2022; Wang et al., 2023] explore the utilization of additional sparse structure priors as a means to strengthen the correlations between the inpainted and masked regions. However, due to their limited semantic generation capabilities and heavy reliance on the semantic consistency between structure priors and texture, these methods inevitably suffer from meaningless artifacts.

Recently, diffusion models [Ho et al., 2020; Song et al., 2021] have shown state-of-the-art performance in generative tasks, exhibiting excellent semantic generation capability and pattern convergence, thus effectively addressing poor semantic generation in image inpainting [Lugmayr et al., 2022; Luo et al., 2023; Liu et al., 2024a; Yue et al., 2024; Wang et al., 2022]. Within these models, a notable development is the diffusion bridges [Liu et al., 2023; Shi et al., 2024; Zhou et al., 2024; Han et al., 2025], which skillfully integrate the end-to-end training paradigm of CNN-based models with the denoising concept of diffusion models, creating a point-to-point diffusion process between original and damaged images. Among these developments, GOUB [Yue et al., 2024] proposes applying Doob's h-transform to the Generalized Ornstein-Uhlenbeck (GOU) process, achieving superiority over other diffusion bridge models. However, due to the absence of richer contextual semantics in the denoising process, all of the above diffusion-based methods tend to either exhibit poor semantics consistency or fill the masked area with inferior context-stability (see Figures 1 (a) and (b)).

Early stage Late stage

(b) StrDiff Early stage Late stage

(c) GSGDiff (Ours)

Figure 1: Illustration of the denoised results for existing arts on PSV [Doersch et al., 2012], e.g., GOUB [Yue et al., 2024] (a) fails to generate consistent semantics between the masked and unmasked regions due to missing structure guidance; equipped with the guidance of sparse consistent structure, StrDiff [Liu et al., 2024a] (b) tackles the semantic discrepancy with consistent structure guidance but introduces blurring and artifacts. Our GSGDiff (c) achieves coherent denoised results via auxiliary global and consistent structure guidance.

This, in practice, limits their performance capabilities, especially in complex degradation scenarios.

In this paper, we are committed to stabilizing the holistic semantics by exploiting additional semantic priors in texture denoising and propose an intuitive and effective inpainting architecture named the Global Structure-Guided Diffusion Bridge framework (GSGDiff). Specifically, we employ a guidance scheme that diffuses from grayscale images to masked edge maps within the structure denoising network to produce auxiliary structural information. To obtain global semantic prior over time, we propose a posterior sampling approach for the generalized ornstein-uhlenbeck bridge model. This approach enables the structure denoising network to capture global and consistent semantics at each timestep. Moreover, given the unique characteristics of the diffusion models, where higher values of timesteps correspond to lower levels of denoising, the guidance semantics often contain more irrelevant noise. To mitigate this issue, we propose a semantic fusion schedule that reduces the weight of ineffective semantics in the early stages to improve the effectiveness of guided information. With the assistance of structures in the texture denoise process, GSGDiff can generate meaningful results.

The main contributions of the paper are as follows:

• This paper proposes a novel Global Structure-Guided Diffusion Bridge framework that leverages a pre-trained auxiliary network to acquire structure semantics and efficiently injects them into texture generation, achieving superior restoration in texture and structure.
• To obtain time-dependent global semantics guidance, this paper proposes a posterior sampling approach tailored to the generalized ornstein-uhlenbeck bridge model, allowing the structure denoising network to capture global and consistent semantics at each timestep.
• Considering the lower denoising capability in the initial stages of denoising, this paper proposes a semantic fusion schedule to improve guided semantics more effectively while reducing the impact of irrelevant noise on texture generation.

2 Methodology

2.1 Preliminaries: Generalize Ornstein-Uhlenbeck Bridge for Image inapinting

Given a ground-truth image Igt ∈ R3×H×W and a binary mask M ∈ R1×H×W that indicates the region to be inpainted, the goal of image inpainting is to reconstruct the masked image Im = Igt ⊙M into a fully inpainted image. Typical diffusion bridge models for inpainting consist of two main processes: the forward texture diffusion process and the reverse texture denoising process.

For the forward diffusion process, given an initial texture state x0 representing the ground truth image Igt and a final state xT representing the corresponding masked image Im, with both ends of the bridge (i.e., x0 and xT ) known, the diffusion process {xt}Tt=0 for any time t ∈ [0, T] can be expressed as:

where θt and gt refer to the drift and diffusion coefficients, respectively. They are positive time-dependent parameters that satisfy 2λ2 = g2 t /θt. λ2 is a given constant scalar and wt is a standard Brownian motion that introduces randomness to the differential equation. The stochastic differential equation (SDE) defined above will necessarily pass through the final state xT when t = T , creating a bridge connecting the points x0 and xT , hence this type of model is referred to as a diffusion bridge model. The forward process at any given moment t can be defined as follows:

Masked

Image Inpainted result Masked Image Inpainted result For the reverse texture denoising process, we can reverse the diffusion SDE from Eq. (1) to denoise starting from the final state xT and obtain the restored image x0, this process can be defined as:

Since the conditional score function is unknown, we can train a conditional time-dependent neural network by parameterizing the noise as ϵθ(xt, xT , t) to estimate the score. Thus, the ultimate training objective can be expressed as:

where a and b is positive weight and the conditional score ∇xt log p(xt | xT ) ≈ ∇xt log pθ(xt | xT ) = -ϵθ(xt, xT , t)/¯σ′t . Therefore, starting from masked image xT , we can recover x0 by utilizing Eq. (3) to perform reverse iteration.

2.2 How to obtain Time-dependent Global Semantics for Texture Denoising?

Recent work [Liu et al., 2024a] indicates that progressively incorporating the semantically sparse structure into texture generation over time can encourage consistent semantics in the inpainted regions. However, in more complex degradation scenarios, merely relying on consistent semantics established at an early stage often fails to align the generated content with the global semantics well.

As shown in Figure 1, both GOUB [Yue et al., 2024] (a), which lacks additional semantic prior, and StrDiff [Liu et al., 2024a] (b), which relies on consistency-based semantics structure guidance, fail to achieve optimal inpainted result. To this end, an intuitive idea is to introduce global semantics, building upon the consistent sparse structure guidance provided in the early stage.

To achieve this, this paper proposes a posterior sampling approach for GOUB to generate richer global semantic prior from the posterior distribution p (xt-1 | xt, xT ) conditioned on x0. Specifically, given an initial state x0 and a finite random diffusion state xt at time t ∈ [0, T ], we can prove that the posterior of GOUB is tractable, and this posterior distribution is given by:

From Bayes' formula, we can infer that:

Since p(xt-1 | x0, xT ) and p(xt | x0, xT ) are Gaussian distributions (2), by employing the reparameterization technique, we can obtain p(xt | xt-1, xT ) = N(axt-1 + bxT , ( ¯σ′2 t -a2 ¯σ′2 t-1 ) I). Thus, this posterior distribution with mean and variance is given by:

2.3 Structure-Guided Denoising Process and Semantic Fusion Schedule

Considering that the semantic discrepancy between the masked and unmasked regions gradually increases as the timestep increases (see Figure 3 (a)), our aim is to alleviate this discrepancy in the early denoising through the aid of structures. This led to the introduction of the following:

How does the Structure Guide the Texture Denoising Process?

For the choice of guidance approaches, previous works [Liu et al., 2024a; Dong et al., 2022] used designed feature fusion strategies or modules to incorporate guidance information. However, these approaches typically require retraining the entire model, resulting in both limited flexibility and high computational costs. To address this, we introduce some new guidance modules inspired by ControlNet [Zhang et al., 2023], which guide the model's generation without modifying the original pre-trained diffusion model, thus fully leveraging its capabilities.

Specifically, as illustrated in Figure 2, the global and consistent semantics ˆy0 and ˆyt-1 are first encoded by their corresponding semantic blocks. These extracted features, combined with features of available regions, serve as inputs to the Conditional Guidance Module (CGM). The CGM is specifically designed to process and integrate the guidance information. Its left side aligns with and is initialized from the downsampling module of the pre-trained texture denoising U-Net, while its right side comprises zero-initialized convolutional layers. Due to the zero initialization, the initial influence of the conditional guidance on the denoising process is zero, which allows the model to maintain stability during the early stages of training. As the model learns, it gradually incorporates the guidance information, enabling it to approach optimal parameters more stably.

How to avoid noise interference within the obtained structure priors?

Due to the limited denoising capabilities in the early stages, the obtained structure semantics usually lack effective information. As depicted in Figure 3 (b), as t decreases from

where,

Moreover, thanks to the reparameterization technique (xt = ¯m′t + ¯σ′t ϵt), we can combine it with the noise prediction network ˜ ϵϕ(xt, xT , t) to estimate the variable x0 at time t:

where ¯m′t and ¯σ′t are the forward mean and variance in Eq. (2). Then we can iteratively sample reverse states by combining Eq. (9) with Eq. (5) to construct the sampling process.

By applying this sampling strategy to the structure denoising U-Net ϵφ(yt, t), as depicted in Figure 2, we can obtain the global semantics ˆy0 and the consistent semantics ˆyt-1 at any time t to assist the texture generation.

100 to 60, the masked regions within the auxiliary consistent structure predominantly contain invalid noise, thus it is difficult to perform any guidance function. To minimize the influence of invalid information during training, we propose a specific Semantic Fusion Schedule for the extracted global and consistent structure features, aiming to assign appropriate weights to incorporate them into the fusion process. Specifically, for the weight function of the consistent features, we desire it to smoothly approximate zero in the early stages and eventually converge to one. Given the natural properties of the exponential function, it can be set as follows:

where the parameter a is set to 5 for all experiments. Compared to the sparse consistent structure, the global structure contains richer semantics (see Figure 3 (b) and (c)). Noisy and ineffective information primarily appears in the earlier stages (e.g., when t decreases from 100 to 90), as illustrated in Figure 3 (d). Therefore, our idea is to initially assign higher weights to the global features, ensuring robust guidance from global semantics throughout training. The weight function for global features is given by:

The function curves of the two weight functions are shown in Figure 4. Thus, the final fusion strategy is defined as follows:

As a result, each semantic block is updated only when the guidance features become more distinctive, effectively avoiding the impact of irrelevant information.

2.4 Sampling Strategy for Texture Denoising Process

Generally, we can utilize the trained network to generate high-quality images by sampling a state xT and then iteratively solving the Eq. (3) with a numerical scheme. However, recent studies [Zhang et al., 2024; Luo et al., 2024] have shown that iterative sampling using the posterior sampling method during the reverse process can accelerate the convergence of the generative process and improve sample efficiency. Therefore, this paper applies the proposed posterior sampling strategy to texture denoising networks as well, aiming to achieve more excellent inpainting results. However, as the denoising proceeds, the correlation between the semantics generated by the auxiliary structure branch (ˆy0 and ˆyt-1) and the texture (ˆxt-1) gradually weakens. Continually injecting the guidance semantics throughout the entire texture denoising process may result in color distortion and blurring in the masked regions [Liu et al., 2024a].

To address this, we adopt a simple yet effective stage-wise semantic injection strategy, utilizing the stage point α to control the guided timesteps. Specifically, during the early denoising stages (i.e., when t ∈ [T, α]), we leverage sparse consistent and global structures to generate coherent contents align well with the overall semantics. In the later stage (i.e., when t ∈ (α, 0]), we rely solely on dense textures to generate meaningful semantic details. By doing so, we avoid semantic discrepancy between the generated textures and structures in the later stage, ultimately eliminating color distortion and achieving excellent results.

2.5 Overall Architecture

The pipeline of proposed GSGDiff is illustrated in Figure 2. Specifically, we first employ a pre-trained structure denoising U-Net ϵφ(yt, t), which models the transformation from gray image to masked edge image, to acquire structural semantic priors at each timestep. Then, these structural priors, combined with the masked image, are encoded by their respective feature extract blocks. Each block consists of four convolutional layers followed by a zero-initialized convolution layer and utilizes the SiLU [Elfwing et al., 2018] activation function. Afterward, the guidance module takes these encoded features to guide the texture generation. We initialize the texture denoising network with a pre-trained model and freeze its parameters, preserving its base denoising capability while enabling guidance learning and reducing training overhead.

3 Experiments

3.1 Experimental Settings

We validate our method and various baselines on three typical datasets, including Paris Street View (PSV) [Doersch et al., 2012], which consists of street photos taken in Paris and contains 14,900 training images and 100 validation images; CelebA-HQ [Karras et al., 2018] is a dataset containing 30,000 high-quality human face images, divided into 28k training images and 2k validation images; Places2 [Zhou et al., 2017] is a collection of more than 1.8 million natural images in multiple scenes. Following previous research, we use PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) [Wang et al., 2004] to measure pixel and structural similarity. FID (Fr´echet Inception Distance) [Heusel et al., 2017] and LPIPS (Learned Perceptual Image Patch Similarity) [Zhang et al., 2018] are used to measure perceptual disparity and visual effect.

3.2 Comparison with the State of the Arts

To validate GSGDiff's effectiveness, various typical inpainting models are compared, including LaMa [Suvorov et al., 2022] which employs Fourier convolution to enlarge the receptive field; MAT [Li et al., 2022], CMT [Ko and Kim, 2023] which uses a self-attention mechanism to model the long-distance dependencies between masked and unmasked regions; CTSDG [Guo et al., 2021], DGTS [Liu et al., 2022], ZITS [Dong et al., 2022] which focuses on using the structure to assist texture generation; Repaint [Lugmayr et al., 2022] and IR-SDE [Luo et al., 2023], which benefit from DDPM but overlook the semantic consistency; StrDiff [Liu et al., 2024a], which recently exploits consistent structure to assist texture generation, and GOUB [Yue et al., 2024], which performs well as a diffusion bridge model in inpainting.

Quantitative Results Table 1 shows that GSGDiff achieves near-optimal performance across all three benchmark datasets as masking ratios increase. On the Places2 dataset, we achieve PSNR improvements of 1.6%, 2.6%, and 4.8% for three different masking ratios compared to GOUB. Additionally, for street-view and natural image scenarios, our method outperforms StrDiff in all metrics. This superiority is attributed to StrDiff's lack of holistic semantics in its denoising process, suggesting that our method is capable of generating more stable and semantically consistent contexts. For texture-rich face inpainting, GSGDiff demonstrates superiority over other methods in terms of perceptual performance, as evidenced by lower FID and LPIPS indices.

Qualitative Results In Figures 5 and 6, some visualization results are presented for our method and some representative methods. For complex structural degradation, our method avoids structural inconsistencies and blurring, yielding sharper results. Examples include the wall texture in the first row of Figure 5 and the second row of Figure 6. In addition, our method excels in restoring natural scenery, such as the wheat field and the chair depicted in Figure 5, demonstrating that our method can obtain more contextually semantic restoration results. For face restoration, GSGDiff also shows excellent texture generation ability, and compared with other methods, our method generates more natural face features with more realistic expressions.

3.3 Ablation and analysis

In this section, several ablated methods are conducted to assess the efficacy of the proposed algorithm and its components. We also analyze the selection of stage points α in texture denoising. Experiments use the PSV [Doersch et al., 2012] dataset with mix mask ratios of 20%-60%, and the baseline model is obtained by removing the proposed prior guidance and the semantic fusion schedule from our model.

Semantic prior guidance Table 2 shows that both semantic priors yield improvements over the baseline model. The global prior notably enhances the performance, with model B achieving a gain of 0.7 dB in PSNR over the baseline. Notably, by combining consistent and global semantics, model C exhibits better FID and SSIM, but performs slightly worse on the other metrics. The reason for this is that the consistent semantics are too sparse in the early denoising stage and provide limited effective guidance to the texture denoising network. Hence, the performance improvement of the model C is not as significant as that of the model B.

Semantic fusion schedule Table 2 shows that both model D and model E show better performance than model C when the respective weighting functions are applied to the semantic priors. At the beginning of the denoising process, the consistent semantics are sparse compared to the global semantics. Thus, after adjusting the weights to reduce the interference of invalid noise, the model D exhibits higher performance metrics than the model E. Ultimately, our GSGDiff exhibits optimal performance when equipped with a corresponding fusion schedule for each semantic prior. In contrast, model F, which utilizes a direct concatenation of semantic priors, is shown to be inferior to our proposed semantic fusion schedule.

The analysis of sampling strategies in texture denoising To explore the practical performance of the proposed posterior sampling in texture denoising, we compare sampling strategies across models. The results, shown in Table 3, demonstrate that using our posterior sampling during inference significantly improves the model's performance, yielding better PSNR, SSIM, and LPIPS metrics.

The choices for stage point α In Figure 7, we set α to take values at [0.2T, 0.4T, 0.6T, 0.8T] to evaluate its effect on the final inpainted results. From the figure, it can be seen that the values of distortion metrics (PSNR and SSIM) tend to increase as the value of α decreases. Notably, the FID and LPIPS metrics show a significant advantage at α = 0.6T; however, as the guided timestep is prolonged, the perceptual performance (FID and LPIPS) deteriorates. This is due to the correlation between the semantics generated by the auxiliary structure branch and the texture gradually weakens. Continually injecting the guidance semantics throughout the entire texture denoising process may result in color distortion and blurring. Considering the limitations of the traditional metrics (i.e., PSNR, SSIM) which tend to assign higher values to smoothed results and do not reflect human perception well [Zhang et al., 2018], we finally choose α = 0.6T , as the stage point for phased injection of guidance information.

4 Conclusion

We propose a novel diffusion bridge for inpainting, which aims to stabilize the inpainted result by integrating timedependent holistic semantics in texture denoising. A posterior sampling approach is tailored to the Generalized OrnsteinUhlenbeck bridge, which acquires semantically global and consistent structure priors. Considering the limited guidance caused by noise in the early denoising, a semantic fusion schedule is designed to reduce the weight of ineffective guided semantics. Experimental results demonstrate that our method achieves superior performance compared to previous methods.

Acknowledgments

This work was jointly supported by the National Key Research and Development Program of China Grant 2024YFB4710800; Liaoning Provincial Natural Science Foundation Grant 2024-MS-012; Dalian Science and Technology Talent Innovation Support Plan, Grant 2024RY010; Basic Research Project of Shijiazhuang Municipal Universities in Hebei Province (241791387A); Interdisciplinary Research Program of Hebei University (DXK202404).