Semantic-Guided Diffusion Model for Single-Step Image Super-Resolution

1 Introduction

Image super-resolution (SR) is a fundamental yet challenging problem in low-level vision, aiming to reconstruct a highresolution (HR) image from a given low-resolution (LR) input [Wang et al., 2020]. The task is inherently ill-posed due to the complexity and unknown nature of degradation models in real-world scenarios. Recently, diffusion models, as an emerging generative paradigm, have achieved unprecedented success in image generation and demonstrated remarkable Figure 1: A comparison between the most recent SOTA one-step SR model and our SAMSR model. Different from recent works with simple noise distribution, the proposed method incorporates semantic segmentation information into the noise distribution and gause diffusion process.

∗Hao Tang is the corresponding author. This work was done while Zihang Liu was visiting Peking University. Our code is released at https://github.com/Liu-Zihang/SAMSR.

potential in various low-level vision tasks, including image editing, inpainting, and colorization.

Currently, strategies for employing diffusion models in image SR can be broadly categorized into two approaches: (i) inserting the LR image as input to the denoiser and retraining the model from scratch, [Rombach et al., 2022; Saharia et al., 2022] and (ii) utilizing an unconditional pre-trained diffusion model as a prior and modifying its reverse path to generate the desired HR image [Choi et al., 2021; Chung et al., 2022; Wang et al., 2021]. However, both strategies face significant computational efficiency challenges. Conventional methods typically initiate from pure Gaussian noise, failing to leverage the prior knowledge embedded in the LR image, consequently requiring a substantial number of inference steps to achieve satisfactory performance and severely constraining the practical application of diffusion-based SR techniques.

Although various acceleration techniques have been proposed to accelerate diffusion model sampling [Lu et al., 2022; Lugmayr et al., 2020; Song et al., 2020], these methods often compromise performance in low-level vision domains that demand high fidelity. Recent innovative research has begun to reformulate the diffusion process in image restoration tasks, attempting to model the initial step as a combination of LR images and random noise [Yue et al., 2024]. However, the inference speed remains limited. Some subsequent works have explored deterministic sampling strategies for image SR, learning bidirectional deterministic mappings between noise and HR image generation to improve inference speed [Wang et al., 2024], which is shown in Fig. 1a. However, these models frequently suffer from limited authenticity and reduced capability in processing complex semantic images due to constrained inference steps.

To address these challenges, we propose a semantic segmentation-based pixel-wise sampling framework, SAMSR as shown in Fig. 1b, a semantic segmentation guided framework to address the limitations of deterministic sampling in diffusion-based image SR. Existing methods often apply uniform noise addition and global parameters, making it challenging to recover fine details in semantically complex regions. To overcome this, we introduce the SAM-Noise Module, which leverages segmentation masks generated by the Segment Anything Model (SAM) to perform spatially adaptive noise sampling, preserving both semantic and spatial features. Additionally, we propose a semantic-guided forward process that dynamically adjusts the residual transfer rate and noise strength at the pixel level based on semantic weights, enabling prioritized recovery of semantically rich regions. To enhance training, a semantic consistency loss is introduced to align the pixel-wise semantic weights between the prediction and the ground truth. These innovations allow SAMSR to effectively utilize semantic information, achieving superior performance in both real-world and synthetic datasets, particularly in recovering fine details and textures.

Our main contributions are summarized as follows: (i) We introduce, for the first time, a segmentation-mask-based random noise sampling method. This approach performs singledistribution noise sampling separately within each masked region and normalizes and combines them, allowing the Gaussian noise to retain both the spatial and semantic features of the original image. (ii) We leverage the segmentation masks obtained from SAM to derive pixel-level sampling hyperparameters, differentiating the noise addition speed for pixels with varying levels of semantic richness. This ensures that semantically rich regions are distinctly recovered within a single sampling step. (iii) We propose a novel consistency semantic loss that utilizes ground-truth images during training to enhance the model's understanding and application of region segmentation masks, leading to improved performance.

2 RelatedWork

Advances in Super-Resolution Techniques. Superresolution (SR) has undergone significant evolution, transitioning from early methods based on handcrafted priors to deep learning-based approaches. Early SR algorithms leveraged priors such as non-local similarity [Sun et al., 2008], sparse coding [Yang et al., 2010; Cai et al., 2019a], and lowrankness [Milanfar, 2012; Cai et al., 2019a]. These handcrafted approaches were effective for basic degradations but lacked flexibility and generalization for complex real-world scenarios [Zhang et al., 2021].

Deep learning revolutionized SR with the introduction of convolutional neural networks (CNNs) in SRCNN [Dong et al., 2014], which marked the beginning of a series of innovations. Residual learning [Zhang et al., 2018b], attention mechanisms and transformers further improved SR performance in terms of fidelity and perceptual quality [Cao et al., 2023]. Generative adversarial networks (GANs) like ESRGAN [Wang et al., 2018] pushed SR towards generating more realistic textures but often introduced perceptual artifacts and training instability [Ledig et al., 2017].

Recently, diffusion models, initially developed for generative tasks [Saharia et al., 2022], have emerged as promising tools for SR. Unlike GANs, diffusion models iteratively refine Gaussian noise into structured HR images, offering superior theoretical guarantees and perceptual quality. Methods like SR3 [Saharia et al., 2022] and SRDiff [Song et al., 2020] adapt diffusion models for SR by modifying their reverse processes or combining LR images with noise. However, these methods are often computationally expensive, requiring hundreds or thousands of iterative steps [Ho et al., 2020; Lu et al., 2022; Lugmayr et al., 2020].

Acceleration of Diffusion Models for SR. The inefficiency of diffusion models has motivated extensive research into acceleration techniques. DDIM introduced deterministic sampling paths, which reduced inference steps but often compromised image fidelity in SR tasks [Ho et al., 2020; Lu et al., 2022]. Progressive sampling methods, like those used in Latent Diffusion Models (LDM), offered more efficient sampling but still required tens of steps to achieve satisfactory results [Rombach et al., 2022].

To address these challenges, ResShift [Yue et al., 2024] proposed embedding LR information directly into the Markov chain, significantly reducing sampling steps while preserving fidelity. Similarly, SinSR [Wang et al., 2024] distilled the mapping between Gaussian noise and HR images into a lightweight student network, achieving singlestep SR with up to a tenfold speedup. However, these methods face limitations in semantically complex regions, as they rely on uniform noise priors rather than adaptive spatial information [Yue et al., 2024; Wang et al., 2024]. Recent advancements such as HoliSDiP [Tsao et al., 2024] integrate semantic segmentation with diffusion frameworks, providing global and localized semantic information for better spatial fidelity. This approach demonstrates the potential to further reduce sampling complexity while enhancing image quality by leveraging holistic semantic priors [Cao et al., 2023; Tsao et al., 2024].

Applications of SAM in Various Domains. Segment Anything Model (SAM) [Kirillov et al., 2023] and its successor SAM2 [Ravi et al., 2024] have been adapted to diverse fields, including medical imaging, video analysis, and super-resolution. Below, we summarize its applications in these areas and their relevance to this work. (i) SAM in Medical Image Segmentation: SAM has been widely adopted in medical imaging. Models like Med-SAM-Adapter and SAMed leverage parameter-efficient fine-tuning strategies, such as adapters and low-rank adaptations, to tailor SAM for domain-specific applications [Wu et al., 2023; Zhang and Liu, 2023; Leng et al., 2024]. For instance, SAM-based segmentation has been explored in retinal vessel segmentation [Zhang et al., 2024] and polyp segmentation [Xiong et al., 2024]. Additionally, MedSAM-2 [Zhu et al., 2024] demonstrates how memory mechanisms can adapt SAM for 3D segmentation tasks, allowing it to handle unordered medical image slices. These works emphasize SAM's ability to address challenges like data scarcity and fine-grained segmentation in medical domains [Shen et al., 2024; Yao et al., 2023]. (ii) SAM in Video Understanding and Analysis: In video segmentation and tracking, SAM has been adapted to dynamic contexts by incorporating temporal modeling and memory management. For example, SAMURAI introduces motion-aware memory mechanisms to enhance object tracking under occlusion and rapid motion [Yang et al., 2024]. Similarly, SAM2Long uses tree-based memory architectures to improve segmentation consistency across long video sequences [Ding et al., 2024]. In surgical video segmentation, Surgical SAM2 achieves real-time segmentation by employing efficient frame pruning, reducing computational demands while maintaining accuracy [Liu et al., 2024b]. These adaptations enable SAM to excel in spatiotemporal tasks requiring consistent object identity across frames. (iii) SAM in Image and Video SR: SAM's semantic capabilities have recently been extended to super-resolution tasks. SAM Boost utilizes semantic priors to improve alignment and fusion in video super-resolution, enabling better handling of large motions and occlusions [Lu et al., 2023; Liu et al., 2024a]. HoliSDiP combines SAM-derived segmentation maps with diffusion models, providing spatial guidance for improved image super-resolution in real-world scenarios [Tsao et al., 2024]. By integrating SAM's zero-shot segmentation and spatial adaptability, these works demonstrate the potential of SAM in enhancing detail reconstruction and semantic consistency in SR tasks.

Figure 2: The overall framework of the proposed SAMSR method. The SAM-Noise Module (SNM) generates semantically-guided noise maps ϵ′ for the forward process while computing pixel-wise semantic weights. These weights are utilized to adaptively adjust the residual transfer rate and noise strength in both forward and reverse processes, enabling fine-grained control over semantic region reconstruction.

3 Methodology

3.1 Overview

The SinSR model and the ResShift model differ primarily in their ability to reduce the number of inference steps from 15 to a single step through a deterministic sampling strategy. In the original SinSR, the forward diffusion process begins by combining a low-resolution (LR) image y with Gaussian noise ϵ, scaled by a residual transfer rate ηt and noise strength κ. This is formulated as:

while the reverse process is represented by a deterministic mapping:

where kt, mt, and jt are coefficients derived from ηt. Although SinSR achieves single-step sampling efficiency, its reliance on global Gaussian noise ϵ and uniform diffusion parameters ηt and κ limits its flexibility in handling semantically complex regions.

To address this, we propose the SAMSR framework, which introduces semantic guidance via the SAM-Noise Module and SAM-based Forward Process. The SAM-Noise Module refines the global Gaussian noise ϵ into a spatially adaptive noise map ϵ′ , as described in Sec. 3.2. The SAM-based Forward Process dynamically adjusts the residual transfer rate ηt and noise strength κ based on semantic weights derived from SAM masks, as detailed in Sec. 3.3. Furthermore, to enhance the model's understanding of semantic information during training, we introduce a semantic consistency loss that aligns semantic features between predictions and ground truth, which will be elaborated in Sec. 3.4.

The overall framework of SAMSR is illustrated in Fig. 2. As shown in Fig. 2(a), the forward process combines the SAM-Noise Module and pixel-wise weight computation to generate semantically-guided noise. In the reverse process (Fig. 2(b)), these semantic cues are utilized to dynamically adjust the sampling hyperparameters, enabling region-aware image reconstruction. This semantic-guided framework allows SAMSR to better preserve details in semantically rich regions while maintaining global consistency.

3.2 SAM-Noise Module

The SAM-Noise Module aims to integrate the spatial and semantic features of the original image into Gaussian noise, thereby enhancing the diffusion model's ability to handle images with complex semantic information. As shown in Fig. 3, to improve the accuracy of semantic segmentation, the given LR image is first passed through a bicubic interpolation process as input to the SAM. The resulting mask information is then processed through global average pooling and thresholding operations to obtain a binary tensor mask of the original LR image. The specific computation formula is as follows:

where BIC is the bicubic interpolation, the SAM is the segment anything model, and the GAP is the global average pooling. To obtain the final binary mask, a thresholding operation is applied to Fu, where values greater than a predefined threshold T are set to 1, and the rest are set to 0:

We set the threshold T to 0.5. To generate refined noise for the forward diffusion process, we incorporate the binary mask Fa into the noise sampling procedure. Specifically, we first sample M independent noise maps Zm ∈ R3×H×W from a standard normal distribution N(0, 1), where m = 1, 2, . . . ,M. These noise maps are then multiplied by the binary mask in the element Fa, ensuring that the noise is applied only within the regions covered by the mask. Mathematically, the masked noise Nm is defined as:

where ⊙ denotes the element-wise multiplication. Next, the M masked noise maps are summed to produce a combined noise map Nsum:

To ensure the noise map is normalized for the diffusion process, we standardize Nsum to have zero mean and unit variance. The final noise map ϵ′ is computed as follows:

where µNsum and σNsum represent the mean and standard deviation of Nsum. The resulting noise ϵ′ is used as the input to the forward diffusion process, ensuring that noise is spatially restricted to the mask-covered regions while maintaining a normalized distribution.

3.3 SAM-based Forward and Reverse Process

To refine the residual transfer rate and noise strength based on semantic regions, we introduce a pixel-wise weight matrix W(x, y), which is derived from the binary masks Fa. Specifically, for each pixel location (x, y), W(x, y) is defined as the normalized coverage across all M masks, where normalization is performed using the maximum pixel coverage:

where F m a (x, y) ∈ {0, 1} represents the binary value of the m-th mask at pixel location (x, y), and M denotes the total number of masks. The denominator represents the maximum coverage among all pixels in the image.

Using the weight matrix W(x, y), the residual transfer rate ηt and noise strength κ are adjusted as follows:

where m is the hyper-parameter that controls the noise addition speed and intensity for pixels with different levels of semantic richness during the forward diffusion process.

Utilizing the Pixel-wise Weight Matrix, we update the forward and reverse process to introduce semantic adaptiveness into the deterministic sampling framework. Specifically, the adjustments are applied to the residual transfer rate ηt and noise strength κ, enabling pixel-wise control based on semantic guidance.

Therefore, using the noise map ϵ′ , we can update the forward process of the diffusion model starting from an initial state from the LR image y as below:

The updated reverse process at time step t is given as:

Algorithm 1 Training the Pixel-wise Sampling Framework

• Require: Pre-trained teacher diffusion model fθ Require: Paired training set (X, Y ) 1: Init fˆ θ from the pre-trained model, i.e., ˆ θ ← θ 2: while not converged do 3: Sample x0, y ∼ (X, Y ) 4: Compute Wy(x, y) using Equation 8 5: Compute κnew , ηnew T using Equation 9 6: Compute knew t ,mnew t , j new t using Equation 12 7: Sample ϵ ∼N(0, (κnew )2 ηnew T I) 8: xT = y + ϵ 9: for t = T, T - 1, . . . , 1 do 10: if t = 1 then 11: ˆx0 = fθ(x1, y, 1) 12: else 13: xt-1 = knew t fθ(xt, y, t) +mnew t xt + j new t y 14: end if 15: end for 16: Ldistill = LMSE(fˆ θ (xT , y, T ), ˆx0) 17: Linverse = LMSE(fˆ θ (ˆx0, y, 0), xT ) 18: ˆxT = fˆ θ (x0, y, 0) 19: Lgt = LMSE(fˆ θ (detach(ˆxT ), y, T ), x0) 20: Compute Wˆx0 (x, y),Wx0 (x, y) using Equation 8 21: LSC = LMSE(Wˆx0 (x, y),Wx0 (x, y)) 22: L = Ldistill +Linverse +Lgt +λLSC 23: Perform a gradient descent step on ∇ˆ θ L 24: end while 25: return The student model fˆ θ

where knew t , mnew t , and j new t are the updated coefficients derived from the pixel-wise adjusted residual transfer rate ηnew t . These parameters are defined as:

where ηnew t represents the dynamically adjusted residual transfer rate at each pixel location (x, y), which is computed in Eq. (9).

3.4 Semantic Consistency Loss

To further incorporate semantic guidance into the training process, we introduce a Semantic Consistency Loss LSC, which aligns the semantic weights between the predicted output ˆx0 and the ground truth x0. Using the Pixel-wise Weight Matrix W(x, y) defined in Sec. 3.3, we compute the normalized semantic weights Wˆx0 (x, y) for the predicted image and Wx0 (x, y) for the ground truth. The loss is formulated as:

This additional loss is integrated into the original training objective, which consists of the distillation loss Ldistill, reverse loss Lreverse, and ground truth loss Lgt. The updated training objective is defined as:

where λ is a hyper-parameter that controls the contribution of the semantic consistency loss.

By explicitly enforcing alignment between the semantic weights of the prediction and ground truth, the proposed LSC improves the model's ability to utilize semantic segmentation masks effectively, leading to better semantic understanding during training. The overall of the proposed method is summarized in Algorithm 1.

4 Experiments

4.1 Experimental Setup

Compared Methods. We compare our method with several representative SR models, including RealSR-JPEG [Ji et al., 2020], ESRGAN [Wang et al., 2018], BSRGAN [Zhang et al., 2021], SwinIR [Liang et al., 2021], RealESRGAN [Wang et al., 2021], DASR [Liang et al., 2022], LDM [Rombach et al., 2022], ResShift [Yue et al., 2024] and SinSR [Wang et al., 2024].

Metrics. To evaluate the fidelity of our method on synthetic datasets with reference images, we used PSNR, SSIM, and LPIPS [Zhang et al., 2018a]. Additionally, two recent nonreference metrics were employed to assess the realism of the generated images: CLIPIQA [Wang et al., 2023], which leverages a pretrained CLIP [Radford et al., 2021] model on a large-scale dataset, and MUSIQ [Ke et al., 2021].

Training Details. To ensure a fair comparison, we adopted the same experimental configuration and backbone architecture as described in prior work. However, our approach introduces modifications to the forward diffusion process and corresponding loss function. These adjustments enable a significant reduction in the number of training iterations compared to existing models. Specifically, our model achieves convergence in only 10,000-15,000 iterations. We attribute this improvement to the integration of a semantic consistency loss, which accelerates the convergence of the student model and further optimizes the training efficiency.

4.2 Experimental Results

Evaluation on Real-world Datasets. We comprehensively evaluate SAMSR on both real-world and synthetic datasets to demonstrate its robustness and effectiveness across diverse scenarios. For real-world evaluation, we utilize RealSR [Cai et al., 2019b] and RealSet65 [Yue et al., 2024]. Both datasets exhibit diverse degradation patterns and lack ground truth. SAMSR is compared against SOTA SR methods, using nonreference metrics CLIPIQA and MUSIQ [Wang et al., 2023; Ke et al., 2021]. As shown in Table 1, SAMSR achieves superior performance in both metrics, benefiting from its semantic-guided noise sampling and region-aware diffusion dynamics.

Evaluation on Synthetic Datasets. For synthetic datasets, we follow the standard pipeline to create LR inputs from 3000 HR images randomly selected from ImageNet [Wang et al., 2024]. Evaluation metrics include fidelity measures (PSNR, SSIM, LPIPS) [Zhang et al., 2018a] and perceptual quality metrics (CLIPIQA, MUSIQ) [Wang et al., 2023; Ke et al., 2021]. Results in Table 2 indicate that SAMSR attains comparable fidelity metrics to existing diffusion-based models while significantly improving perceptual quality. The introduction of semantic masks enables pixel-wise adjustment of residual transfer rates and noise strengths, enhancing detail preservation in semantically rich regions and allowing SAMSR to outperform SinSR in balancing detail recovery and perceptual realism. These comprehensive evaluations demonstrate SAMSR's effectiveness and versatility in both real-world and controlled synthetic environments.

4.3 Model Analysis

Hyper-parameter m. The hyper-parameter m controls the noise addition speed and intensity for pixels with different levels of semantic richness during the forward diffusion process. Table 3 summarizes the performance of SAMSR on the Realset65 and RealSR dataset under different values of m. We observe that both excessively large and small values of m degrade the model's authenticity. Experiments show that when m is within the range of [1/5, 1/8], our method achieves outstanding results on both the Realset65 and RealSR datasets. Therefore, in this paper, we set m to 1/5.

Hyper-parameters k and ηt. ηt is the residual transition ratio defined during the diffusion process, which controls the Table 4: A comparison of the SAMSR method with different hyperparameters (evaluation on RealSet65 datasets).

gradual transition speed from the HR image to the LR image in the Markov chain. κ is the control parameter for noise intensity during the diffusion process, affecting the strength of noise introduced at each diffusion step. In this paper, we modify ηt and κ using the hyper-parameter m, incorporating dense semantic guidance. This ensures that semantically significant regions are prioritized for finer recovery during a single step of reverse diffusion, while background regions maintain higher global consistency. Table 4 further explores the effects of individually modifying ηt and κ.

Effective of Pixel-wise Sampling. Previous research has shown that learning the deterministic mapping between xt and x0 is hindered by the non-causal nature of the generative process. However, our experiments demonstrate that modifying the local noise intensity across different masked regions of an image effectively mitigates this issue. This allows the student network in the knowledge distillation process to better solve the ODE process in a single step, while maintaining the same model size.

Effective of Consistency Semantic Loss. Our consistency semantic loss aims to improve the model's ability to understand and apply semantic information. Specifically, we establish a loss function based on the semantic weight differences between the ground-truth image and the predicted image x0, enhancing the model's ability to recover semantic details. From Table 5, we observe that the consistency semantic loss not only significantly accelerates the model's convergence speed but also effectively improves its performance, demonstrating its effectiveness during the training process.

5 Conclusion

In this paper, we propose a semantic segmentation-guided diffusion model named SAMSR. Specifically, we introduce a pixel-wise sampling framework based on semantic segmentation, where noise is sampled within masked regions to retain the spatial and semantic characteristics of the original image. Additionally, we leverage segmentation masks to derive pixel-level sampling hyperparameters, enabling differentiated noise schedules for pixels with varying semantic richness. This ensures that semantically rich regions achieve significant recovery within a single sampling step. Furthermore, we propose a semantic consistency loss to accelerate the convergence of the model. Experimental results show that our approach achieves significant performance improvements, particularly for SR tasks on semantically complex images.