A Timestep-Adaptive Frequency-Enhancement Framework for Diffusion-based Image Super-Resolution

1 Introduction

Image super-resolution (ISR) is a fundamental task in lowlevel vision that aims to reconstruct high-resolution (HR) images from their low-resolution (LR) counterparts. It has widespread applications in areas such as medical imaging [Li et al., 2024], satellite imagery [Liu et al., 2017], and surveillance systems [Shermeyer and Etten, 2019], where obtaining high-quality images can naturally be subject to hardware limitations and transmission losses. Early ISR methods [Dong et al., 2016a; Tai et al., 2017] attempt to construct synthetic image pairs through simple handcrafted degradation operations (e.g., bicubic downsampling). However, they fail to generalize well in realistic scenarios since real-world LR images typically involve complex and unknown degradation patterns.

∗Corresponding author.

To address this problem, some studies employ diverse network architectures, such as CNN-based [Dong et al., 2016a] and GAN-based methods [Zhang et al., 2021; Wang et al., 2021]. Among them, recent diffusion model-based ISR methods [Wang et al., 2024b; Lin et al., 2023; Yu et al., 2024; Wu et al., 2024; Yang et al., 2024] have gained great attention, which exhibit superior performance levels in image quality assessment (IQA) metrics while maintaining competitive full-reference results. Another mainline of ISR researches [Guan et al., 2024; Li et al., 2023a; Xu et al., 2024; Xie et al., 2021] approaches image restoration from a frequency-domain perspective, aiming to model the relationship with degradation processes for targeted image recovery. But, current frequency-domain enhancement algorithms are predominantly applied to GANs [Fu et al., 2021; Xu et al., 2024] and traditional CNN models [Yu et al., 2022; Huang et al., 2022; Guo et al., 2022]. Only a few methods [Wang et al., 2024c; Moser et al., 2024; Zhao et al., 2024; Luo et al., 2023] explore the potential of diffusion-based models in the frequency domain. Overall, these methods either rely solely on a single frequency decomposition approach [Wang et al., 2024c], or apply frequency information merely during image processing [Moser et al., 2024; Zhao et al., 2024]. For instance, FourierDiff [Lv et al., 2024] merely leverages the amplitude component of the generative prior to ensure enhanced brightness aligns with natural image distributions. WF-Diff [Zhao et al., 2024] employs a wavelet-based dual-branch diffusion framework to refine high-frequency and low-frequency components of the initial enhanced images. To date, the frequency-domain property of the current diffusion methods remains unexplored.

In this paper, we investigate the frequency-domain behavior of diffusion-based ISR models from a novel sampling timestep perspective. More concretely, we are interested in the impact of different frequency-domain components obtained via frequency spectral and band decomposition, on the restoration process as low-resolution images are progressively recovered through diffusion model timesteps. Through our preliminary experiments, we have two main findings: (1) Impact of Amplitude and Phase Components. To investigate the variation of frequency spectral components across timesteps, we decompose the amplitude and phase components. By fusing one of the components at the current timestep with the corresponding component of the HR images (see top two rows, Figure 2), we observe that in the early timesteps, containing the phase of current timesteps significantly reduces MSE, while in the later timesteps, containing the amplitude drive more MSE change. This indicates that during the restoration process, the diffusion model first generates the phase component (structural information), and later generates the amplitude component (overall visual information). (2) Impact of High- and Low-Frequency Components. Next, to investigate the variation of frequency band components, we decompose the images into high- and lowfrequency components and perform a similar fusion procedure (see bottom two rows, Figure 2). The experimental results show that in the early timesteps, containing the lowfrequency component significantly reduces the MSE, while in the later timesteps, containing the high-frequency has a greater impact. This indicates that the diffusion model initially reconstructs the low-frequency component with global structures, and later generates high-frequency components with fine details, consistent with its generation mechanism.

Based on this, we believe that enhancing the diffusion model's frequency-domain perception at the appropriate time can improve ISR performance effectively. Therefore, we propose a Timestep-adaptive Frequency-aware Diffusion framework for Super-Resolution (dubbed TFDSR), which enhances the diffusion ISR models's different frequency components at adaptive timesteps. It comprises three core components: (1) Timestep Division Controller (TDC): This module dynamically determines the enhancement frequency components across different timesteps, strategically selecting phase and low-frequency components for the early stage, while emphasizing amplitude and high-frequency components for later. (2) Amplitude-Phase Enhancement Module (APEM): This module adaptively enhances the missing amplitude and phase components through a channel attention mechanism, thereby optimizing the representation of each frequency component. (3) High-Low Frequency Enhancement Module (HLEM): This module operates on skip connection and adaptively enhances high- and low-frequency information in the skip features using a frequency band modulation. In terms of sampling strategy, TFDSR adaptively enhances different frequency components at different timesteps, requiring only minimal computational resources and time for efficient fine-tuning. Extensive experiments demonstrate that TFDSR significantly outperforms state-of-the-art diffusion models, especially on no-reference metrics, e.g., average +5.40% in CLIP-IQA. Our source code and appendix are available at https://github.com/liyueying233/TFDSR.

Figure 2: Denoising process of each frequency component. Top two rows: amplitude (A) and phase (P) combined with the corresponding frequency components of the HR image. Bottom rows: low- (L) and high-frequency (H) components with their relative HR components after the inverse Fourier Transform, aligned with each timestep. It demonstrates that early timesteps show greater variation in P and L, while A and H change more in later timesteps.

2 RelatedWork

Image Super-Resolution (ISR). Although deep learningbased ISR techniques have gained widespread adoption, most CNN-based methods [Dong et al., 2016a; Lim et al., 2017; Kim et al., 2016; Dong et al., 2016b; Shi et al., 2016] still suffer from the issue of excessive detail smoothing. To better enhance visual perception, some advances [Zhang et al., 2021; Wang et al., 2021; Liang et al., 2021; Chen et al., 2022; Liang et al., 2022; Wang et al., 2024a] using the GAN-based models in Real-ISR have explored more complex degradation models. For instance, BSRGAN [Zhang et al., 2021] and Real-ESRGAN [Wang et al., 2021] employ realistic degradation modeling techniques. Despite progress in generating perceptually realistic details, GAN-based ISR methods often suffer from unstable training and produce unnatural artifacts. In recent years, the powerful Stable Diffusion (SD) [Rombach et al., 2022] model has been applied to ISR tasks [Wang et al., 2024b; Lin et al., 2023; Yu et al., 2024; Wu et al., 2024; Yang et al., 2024]. For instance, SeeSR [Wu et al., 2024] proposes a semantic-aware approach that better preserves semantic fidelity in reconstructing real-world images. While achieving remarkable performance in Real-ISR tasks, these methods are limited to the spatial domain, failing to fully exploit frequency domain characteristics. In contrast, we discuss the degradation processes of various frequency components and design a timestep-adaptive method to enhance them.

Frequency-based Image Reconstruction. Frequency analysis of image processing has been widely used in computer vision [Yu et al., 2022; Huang et al., 2024; Yang and Soatto, 2020; Cai et al., 2021; Si et al., 2024; Yu et al., 2022]. For ISR tasks, many studies improve images reconstruction quality by applying frequency domain to comprehensively extract features from low-resolution images [Guan et al., 2024; Li et al., 2023a; Xu et al., 2024; Xie et al., 2021]. Among these approaches, most frequencydomain enhancement algorithms are primarily applied to CNN-based and GAN-based models [Yu et al., 2022; Huang et al., 2022; Guo et al., 2022]. Some methods improve reconstruction quality by separating specific components (such as high-frequency or amplitude components) in the frequency domain [Guan et al., 2024; Li et al., 2023a; Xu et al., 2024; Xie et al., 2021; Dai et al., 2024]. Appendix A lists the effects of different frequency components on image quality for other computer vision tasks. Although these existing frequency domainbased ISR methods significantly improve performance, their integration with the increasingly popular diffusion models remains largely unexplored. Only a few methods [Wang et al., 2024c; Moser et al., 2024; Zhao et al., 2024; Luo et al., 2023] explore the potential of diffusion-based models in the frequency domain. For instance, WF-Diff [Zhao et al., 2024] employs a wavelet-based dual-branch diffusion framework to refine frequency components of the initial input images. FourierDiff [Lv et al., 2024] decomposes frequency-domain samples, using generative prior amplitudes to enhance brightness. These methods lack deep integration of frequency components with crucial denoising timesteps of the diffusion. In contrast, our TFDSR framework introduces frequency-domain enhancement through a novel timestep-adaptive approach that leverages the inherent characteristics of diffusion models.

3 Background and Preliminaries

3.1 DiffusionModels for Image Super-Resolution

Diffusion models, like DDPM [Ho et al., 2020] and LDM [Rombach et al., 2022], are latent variable models primarily consisting of a diffusion and denoising process. In the diffusion process, Gaussian noise is gradually added at each timestep t via a Markov chain, using a variance schedule β1, ..., βt, resulting in a random noise distribution

In the denoising process, given the noisy input xt, the model outputs the clean data xt-1 without noise, represented as

Here, µθ and Σθ are set by the denoising model. Current diffusion models [Ho et al., 2020; Rombach et al., 2022] are implemented using a U-Net [Ronneberger et al., 2015] to remove noise from data samples, which consists of downsampling and upsampling paths. Each upsampling block concats both the backbone and skip features in the skip connections.

To improve ISR performance, existing diffusion-based methods typically encode LR images and fuse them with the U-Net through cross-attention or a ControlNet module for HR image generation. Through iterative diffusion and reverse processes, these models effectively recover realistic details.

3.2 Fourier Frequency Domain Transformation

The Fast Fourier Transform (FFT) is widely applied in lowlevel vision tasks, transforming images from the spatial domain to the Fourier frequency domain, denoted as

Its inverse function (IFFT) is formulated as

where j is the imaginary unit, and ejθ = cos θ+j sin θ. F(·) and G(·) denote 2D Fourier transform and its inverse. The frequency features F(x) in Eq. (3) and f in Eq. (4) are both complex domain tensors, expressed as F(x) = R(x) + jI(x), with R(x), I(x) being the real and imaginary parts.

In this paper, we explore two frequency-domain decomposition methods (see Appendix A). The first is the frequency spectral decomposition, separating the frequency into the amplitude A and phase P, which is represented as

The second is the frequency band decomposition, which divides the frequency domain into high- H and low-frequency L parts based on their distance from the frequency center.

4 The Propose Framework

In our preliminary experimental explorations, we observe that in the early denoising stages, diffusion-based ISR models tend to reconstruct the overall structural phase and lowfrequency components while generating amplitude and highfrequency details in the later stages. To leverage this dynamic sampling characteristic, we propose a Timestep Division Controller (Section 4.1) to determine which frequency component at the current timestep should be enhanced. For the enhancement modules, shown in Figure 3, we introduce a channel attention-based module (Section 4.2) for frequency spectral enhancement, and an adaptive modulation module (Section 4.3) for frequency band enhancement.

4.1 Timestep Division Controller

To determine the optimal demarcation points for dividing the sampling timesteps into two distinct groups, we propose a computationally efficient timestep division optimization strategy. Recall our frequency-domain analysis (as illustrated in Figure 1a), which reveals rapid phase and lowfrequency components changing in early stages, significant amplitude and high-frequency components variations in later. Leveraging this temporal divergence, our strategy identifies the demarcation points TAP, THL through three steps.

a) MSE Arrays of Frequency Domain Components. To quantify the reconstruction dynamics of different frequency domain components, in the diffusion model-based ISR baseline (SeeSR), we save the sampling results xt every 5 steps. Then we decomposed them into low-frequency Lt and highfrequency components Ht, as well as amplitude At and phase components Pt through FFT. Subsequently, frequencydomain fusion and iFFT are performed, expressed as

where AHR, PHR, HHR and LHR represent the frequency components of HR images, respectively. The fused results are used to calculate the Mean Squared Error (MSE) with the HR images, yielding a similarity array that changes across sampling timesteps for four different frequency-domain components. The visualization curves are shown in the Figure 1a.

1. Gradient Difference Calculation. For high- and lowfrequency components, we calculate the frequency band gradient difference δAP = dIA(t) dt - dIP(t) dt . For amplitude and phase, we calculate the spectral gradient difference δHL = dIH(t) dt - dIL(t) dt to capture the dynamic relationship between different frequency components during the denoising process.

c) Timestep Optimization Search. Notably, the gradient difference δcomp follows a positive-to-negative or negative-topositive pattern, which indicates that L/P changes rapidly in the early timestep group [0, t), while H/A changes dominate in the later (t, T - 1]. To identify the point where the sign reverses, we sum the cancellable signs, expressed as

Through searching across all sampling timesteps, TAP and THL with maxt S t comp are selected as the demarcation points. Then, our controller divides the whole sampling process into two groups [0, TAP), [TAP, T - 1] for frequency spectral decomposition, and [0, THL), [THL, T - 1] for frequency bands, which is applied to determine the enhanced components across diffusion sampling timesteps.

4.2 Frequency Spectral Enhancement

As stated previously, we observe that the amplitude and phase features of images are progressively restored during the ISR reconstruction process. Therefore, we propose that diffusionbased ISR models should actively learn these two types of features during training. Inspired by prior studies [Hu et al., 2018; Zhao et al., 2019], which enhance model performance by adjusting the importance of different channel features in convolutional neural networks (CNNs), we extend this concept to amplitude and phase features. To this end, we design a channel attention module based on CNNs, termed the Amplitude-Phase Enhancement Module (APEM), to enhance these two frequency-domain components. Through this, the diffusion model is able to perceive the phase features that better capture structural information at the early stage, and later amplitude features to improve the visual quality of the image. Technically, this process can be divided into two steps:

1. Initial Feature Extraction. First, we extract the amplitude and phase components of the features from the upsampling blocks of the U-Net. This is implemented through a combination of convolution and ReLU operations, which serve as the initial feature extraction, represented as

where C is the controller, Conv(·) denotes convolution operation, := means the replacement of variables. We activate A when the timestep t ≥ TAP, and activate P when t < TAP, hence achieving adaptive control to better complete the frequency perceptron ability of diffusion-based ISR models.

1. Channel Attention Map Generation and Application. Next, we generate channel attention maps for both the amplitude and phase components, which are denoted as

where Mattn C represents the attention maps of amplitude and phase. σ(·) and AvgPool(·) denote the sigmoid and average pooling operations. Then we calculate the enhanced amplitude and phase components Aout and Pout by applying the Mattn C , which is denoted as

The attention map allows the model to better emphasize channel features that contribute significantly to the ISR task.

To apply the enhanced amplitude and phase components, we combine Aout and Pout into the frequency domain by F ′ = Aout ·ejPout , and further transfer to the spatial domain by iFFT, which is denoted as x′skip = G(F(xskip)′ ).

4.3 Frequency Band Enhancement

Next, for frequency band, we explore enhancing the high- and low-frequency components at different sampling timesteps. As discussed in FreeU [Si et al., 2024], the skip connections in U-Net blocks can transmit high-frequency, informationrich features to deeper layers of the network, thereby preserving more comprehensive image information. Note that FreeU is designed for text-to-image tasks which only applies two constant scaling transformations to low-frequency features on all layers. However, for diffusion-based ISR problems, the features on U-Net layers with various resolutions convey various semantic information. Therefore, considering the varying richness of information, we propose a high- and low-frequency enhancement module (HLEM) with adaptive masking to enhance different frequency bands across sampling timesteps. It can be divided into the following two steps.

1. Adaptive Mask Construction. To accurately filter and dynamically enhance the frequency components in the skip features, we construct the adaptive mask MH and ML, for high- and low-frequency respectively. Considering that lower-level and smaller-scale features often contain less detailed information, the mask adjusts the enhancement factor

based on scale adaptively, to better adapt to the frequency structure of features at different levels, formulated as

Here r and rthresh are the radius and the radius threshold relative to the frequency center; C is the controller. S is the scale of skip features; PC is the enhancement factor. Similarly, we activate L when the timestep t < THL, and activate H when t ≥ THL, obtaining the high- and low-frequency masks.

1. High- and Low-Frequency Component Enhancement. We then perform an element-wise multiplication of the adaptive mask MC from Equation 14 with the skip features xskip in the frequency domain, to amplify and enhance frequency band components, which is represented as

where ⊙ denotes element-wise multiplication. Finally, the inverse Fourier transformation transfers the enhanced skip features to the spatial domain, denoted as x′skip = G(F(xskip)′ ).

4.4 Overall Training and Sampling Details

Notably, diffusion-based ISR models typically consist of two key stages - the training and sampling process. In practice, our modules are integrated into the whole process as follows:

• For frequency spectral enhancement (APEM), we focus on learning frequency-domain feature parameters without requiring a timestep division controller during the training process. During sampling, we apply the demarcation point TAP for adaptive enhancement, prioritizing the phase component in the initial stage with t ∈ [0, TAP), and focusing on the amplitude component in the later stage with t ∈ [TAP, T - 1].
• For frequency band enhancement (HLEM), which does not contain any trainable layers, we set it as a trainingfree module and apply it exclusively during the sampling process. Similarly, we enhance the low-frequency component when the timestep t ∈ [0, THL), and regulate the high-frequency when t ∈ [THL, T - 1].

Overall, we make only minor modifications to the network structure, which have minimal impact on the entire training process. Consequently, in practice, the overall training cost of our method is comparable to that of our diffusion-based ISR baseline model (SeeSR); see Appendix E. These two modules are applied to the skip connection features from the U-Net down-blocks (see Figure 3). For more ablation results of the training strategies and places, please refer to Appendix C.

5 Experiments

5.1 Experimental Settings

Training Datasets. We train TFDSR on the first 10K realworld images from LSDIR [Li et al., 2023b] and first 10k face images from FFHQ [Karras et al., 2021], which are cropped into 512 × 512 patches. And we use the degradation model with the setting of Real-ESRGAN [Wang et al., 2021].

Testing Datasets. We employ the StableSR [Wang et al., 2024b] test datasets and evaluate our approach on the following datasets. (1) For the synthetic dataset, we use 3,000 generated pairs of LR-HR images from the DIV2K validation set [Agustsson and Timofte, 2017], where the LR images have a resolution of 128 × 128, and the HR images have a resolution of 512 × 512. (2) For the real-world datasets, we utilize the DRealSR [Wei et al., 2020] and RealSR [Ji et al., 2020] datasets center-cropping the LR images to 128 × 128.

Evaluation Metrics. We adopt a series of full-reference and no-reference metrics to assess the performance of different methods. The full-reference metrics include PSNR, SSIM (evaluated on the Y channel in the YCbCr color space), and LPIPS [Zhang et al., 2018]. For quality evaluation, we employ no-reference image quality assessment (IQA) metrics, including CLIP-IQA [Wang et al., 2023], MUSIQ [Ke et al., 2021], and MANIQA [Yang et al., 2022].

Implementation Details. We employ the SeeSR [Wu et al., 2024], a controlled T2I (Text-to-Image) diffusion-based model, as our pre-trained baseline. Then we train the APEM for 600 iterations with a batch size of 32, a learning rate of 5 × 10-5 , and 512 × 512 resolution on a single A100 GPU. During sampling, we utilize the adaptive frequency sampling strategy using the TDC module, which dynamically selects enhanced frequency components based on the current sampling timestep, with a total sampling step of 50. Hyperparameters TAP = 400, THL = 500, PH = 0.05, and PL = 0.9 are tuned using a validation set composed of 100 randomly selected images from the training set (LSDIR+FFHQ), which are uniformly applied to the three datasets of Table 1, including varying samples. For the ablation results of hyperparameters tuning and the full reproducibility information, please refer to Appendix B and the source code.

Compared Methods. We select several state-of-the-art (SOTA) ISR models, which can be divided into two groups. The first group consists of GAN-based methods, including BSRGAN [Zhang et al., 2021], Real-ESRGAN [Wang et al., 2021], FeMaSR [Chen et al., 2022], DASR [Liang et al., 2022], SwinIR-GAN [Liang et al., 2021]. The second group is diffusion-based methods, including StableSR [Wang et al., 2024b], SS-MoE [Luo et al., 2023], SUPIR [Yu et al., 2024], PASD [Yang et al., 2024], SeeSR [Wu et al., 2024].

5.2 Comparison with ExistingModels

Quantitative Comparisons. As shown in Table 1, we first conduct a quantitative comparison between the proposed method and the current state-of-the-art (SOTA) methods on both synthetic and real-world datasets. The results demonstrate that our method achieves the best scores on almost all no-reference metrics. Specifically, on the real-world benchmark RealSR, our TFDSR achieves a CLIP-IQA score of 0.7245, representing a 6.75% improvement over our baseline (the second-best method) SeeSR, which fully validates the superiority of TFDSR. Notably, the experimental results indicate that GAN-based methods outperform almost all based on diffusion models in terms of full-reference metrics (average PSNR/SSIMmetrics, Diffusion: 23.21/0.5791 vs. GAN: 24.06/0.6176). This discrepancy can be primarily attributed to a potential limitation inherent to the training strategies of diffusion models. For more detailed discussions, please refer to the Appendix F. Overall, it can be shown that our TFDSR does indeed largely outperform current diffusion-based ISR methods in terms of no-reference metric scores while maintaining very competitive full-reference metrics.

Qualitative Comparisons. To demonstrate the effectiveness of TFDSR, Figure 4 presents a comparison between the existing methods and our TFDSR. It can be observed that our method enhances the quality and fidelity of the image generated by diffusion-based ISR methods, particularly in detailed textures and general visual effects. Specifically, GAN-based approaches tend to produce over-smoothed outputs, whereas diffusion models frequently exhibit unnatural details, particularly manifesting in biological features (e.g., mammalian fur patterns) and complex textures. For instance, the third case in Figure 4 demonstrates some diffusion-based methods will generate incorrect eyes and blurring results. More examples with varying orientations of stripes are in Appendix D.

User Study. We also conduct a user study comparing our method on a set of test datasets, instructing 20 participants to choose the result with high quality and fidelity among these test methods. The results are shown in Figure 1b, revealing superior performance of our approach in perceptual quality.

5.3 Ablation Study

To further demonstrate the effectiveness of our TFDSR method, we conduct ablation results on three key frequencybased modules. We validate the superior performance of APEM shown in Row 2 of Table 2. Next, we evaluate the effectiveness of HLEM shown in Row 3 of Table 2. And we also show the significant potential in ISR of TDC (see Row 4 of Table 2). By applying three modules, we achieve obvious improvement in no-reference metrics over baseline (see Row 1, Table 2), while maintaining competitive full-reference metrics. More ablation results are in Appendix C.

6 Conclusion

In this work, we propose a timestep-adaptive framework TFDSR for enhancing diffusion-based ISR models from a frequency perspective. To achieve this, we first propose a novel channel attention mechanism for enhancing the frequency spectral components (APEM). Also, we develop a new semantic-aware mask that adaptively determines the thresholds by feature inputs for regulating the frequency band components (HLEM). As shown in the extensive experimental evaluation, we demonstrate the effectiveness of the TFDSR. We also hope our work will draw more attention from the community toward a broader view of addressing diffusion for low-level vision from a frequency perspective.

Acknowledgments

This paper is supported by NSFC under Grants (No. 62402424). Haobo Wang is also partially supported by (No. U24A201401). Hanbin Zhao is supported by NSFC under Grant 62402430, Zhejiang Provincial Natural Science Foundation of China under Grant LQN25F020008.