FreqFormer: Frequency-aware Transformer for Lightweight Image Super-resolution

1 Introduction

Single image super-resolution (SR), aiming at reconstructing high-resolution (HR) images from their low-resolution (LR) counterpart, has received much attention in computer vision and has a variety of potential applications [Wang et al., 2020; Zhang et al., 2023; Cui et al., 2024; Li et al., 2023b; Guo et al., 2024], such as medical imaging and video surveillance [Ren et al., 2019]. Recently, various CNN-based [Dai et al., 2023] and Transformer-based SR methods [Dai et al., 2019; Liang et al., 2021] have been developed and achieved superior performance in image super-resolution.

In particular, vision Transformer with window selfattention has received much attention due to its superior performance in image restoration tasks. Among them, SwinIR

∗Corresponding author: Jinbao Wang and Zexuan Zhu

[Liang et al., 2021] obtains remarkable performance by adopting self-attention on a local window. ELAN [Zhang et al., 2022] improves SwinIR by computing self-attention with different window sizes. Recently, SRFormer [Zhou et al., 2023] develops permuted self-attention with fewer parameters and computations.

Despite the great success of Transformer-based SR methods, these methods usually contains several drawbacks. First, these methods are usually computational intensive, due to the huge model parameters, thus hindering the practical use of SR models in real applications. For example, the advanced SwinIR contains more than 11M parameters. Second, it is found that Transformer-based SR methods prefer to retain low-frequency information (see Fig.1) while ignoring the reconstruction of rich high-frequency information, hence hindering the representational power of Transformers.

As shown in Fig.1, we compute the power spectrum of the feature maps of different layer depths produced from SwinIR. We can see that SwinIR shows a significant amplitude drop in the high-frequency phase as the layer goes deep, which indicates that the vanilla attention prefers to focus on low-frequency information while neglecting highfrequency information which is important to SR. Thus, these observations raise a natural question: How to reduce model size while restoring high-frequency image details for Transformer-based SR methods?

Inspired by the above observations, we propose FreqFormer, a Frequency-aware Transformer for lightweight image super-resolution. As shown in Fig. 2, our FreqFormer mainly contains Frequency Division Module (FDM) and Frequency-aware Transformer Block (FTB), which integrates spatial, channel, and frequency information to reconstruct the high-frequency representation. Specifically, FDM is used to separately handle high- and low-frequency information, and recover high-frequency details. Then, Frequency-aware Cascade Attention (FCA) in FTB is introduced to mix the highfrequency information from the shallow feature extraction with the channel information, and perform high-frequency recovery of the shallow extracted information. To obtain better feature representation, we further incorporate spatial frequency attention and channel transposed attention in FCA. In this way, our FreqFormer can capture spatial, frequency, and channel contexts, facilitating inter-block feature aggregation across dimensions and compensating for the loss of long-distance detail information in attention. Moreover, for the further fusion of feature representations, we design a Dual Frequency Aggregation Feed-Forward Network (DFFN), introducing a frequency gate in the middle of the fully connected layer to aggregate frequency information. Overall, our FreqFormer enhances high-frequency features within attention blocks, supplements high-frequency details externally, and achieves a comprehensive enhancement of global information from attention mechanism outputs.

Our main contributions are summarized as follows:

• We design a novel Frequency-aware Transformer, FreqFormer, which integrates spatial, channel, and frequency information to reconstruct the high-frequency representation in vanilla transformers.
• We propose the Frequency Division Module (FDM) to process high- and low-frequency information in a divide-and-conquer manner, and introduce Frequencyaware Transformer Block (FTB) to accomplish highfrequency restoration and multi-feature aggregation.
• Extensive experiments demonstrate that our FreqFormer outperforms existing state-of-the-art methods while maintaining low computational complexity and model size.

2 RelatedWork

2.1 CNN-based SRMethods

Since the introduction of CNNs by SRCNN [Dong et al., 2016a] into the field of image super-resolution, significant success has been achieved. SRCNN is a groundbreaking work that not only utilizes CNNs but also surpasses traditional methods. Subsequently, numerous works and methods [Tai et al., 2017; Kim et al., 2016b] have delved deeper into network layers, exploring improved performance and structures. Some methods [Lim et al., 2017; Zhang et al., 2018; Dong et al., 2016a; Tai et al., 2017] employ abundant skip connections to accelerate network convergence and enhance reconstruction quality. For expediting SR inference, FSRCNN [Dong et al., 2016b] extracts features at the LR scale and performs upsampling operations at the network's end. This pixel-shuffling [Sun et al., 2023; Shi et al., 2016] upsampling framework has been widely adopted in subsequent models. To augment the representational capacity of SR models, some models have introduced channel attention [Zhang et al., 2018]. Recently, some works [Chen et al., 2023] have explored the aggregation of spatial and channel features to further enhance SR performance.

2.2 Vision Transformer BasedMethods

In recent years, Transformers have demonstrated significant potential in both natural language processing and computer vision tasks, such as image classification [Dosovitskiy et al., 2021; Liu et al., 2021; Li et al., 2021] and segmentation [Wang et al., 2021; Wu et al., 2020]. Notably, the Vision Transformer (ViT) [Dosovitskiy et al., 2021] has demonstrated the advantages of Transformers in feature encoding over CNNs. While ViT excels in modeling long-range dependencies in features, a body of work has established that CNNs and Transformers can be complementary. Due to their impressive performance, Transformers have been introduced for low-level visual tasks, including image SR [Chen et al., 2024; Wang et al., 2022]. For example, SwinIR performs selfattention on a local window for image restoration, while SRformer [Zhou et al., 2023] develops permuted self-attention to enlarge receptive field.

Unlike previous works that extract features in spatial domain, we propose a novel frequency-aware Transformer, called FreqFormer, which integrates spatial, channel, and frequency information to reconstruct the high-frequency representation.

3 Methodology

As shown in Fig. 2a, our proposed FreqFormer comprises three components: shallow feature extraction, deep feature extraction, and high-resolution image reconstruction. Formally, given a LR image ILR, a 3×3 convolution layer is used to extract the shallow feature FS ∈ RH×W×C . Then FS will go through the proposed Frequency Division Module (FDM) to process high- and low- frequency features in a divide-andconquer manner. After that, several stacked Frequency-aware Transformer Block (FTB) is employed followed by several convolutions and a skip connection to obtain the deep features FD. Finally, Pixel-Shuffle is used to up-sample the resolution of FD and generate the super-resolved images ISR. More details are described in the following sections.

3.1 Frequency DivisionModule

As shown in the previous work [Park and Kim, 2022], highfrequency information in the original image would be easily lost when extracting feature in spatial domain. For this reason, we develop Frequency Division Module to separately handle high- and low- frequency in the FS, and recover the high-frequency details before entering deeper layers.

Inspired by the work [Patro and Agneeswaran, 2023], we first use Dual-Tree Complex Wavelet Transform (DTCWT) to separate the high- and low-frequency information of FS.

Specifically, the real part of the DTCWT results can be formulated as

where Xϕ denotes the low-frequency scaling component and Xψ denotes the high-frequency wavelet component. M refers to resolution, and k refers to directional selectivity.

Similarly, we can also obtain the imaginary part of the DTCWT and finally obtain Lin and Hin. Lin ∈ RC×H×W represents the low-frequency component of the image. We then use 1 × 1 and depth-wise convolution to capture both global low-frequency information and local detailed information, resulting in Lout ∈ RC×H×W .

However, obtaining useful features from the highfrequency components Hin ∈ Rk×C×H×W×2 is a significant challenge. This is because the presence of multiple angles increases the dimensions by a factor of k, and due to the inclusion of real and imaginary values, the actual parameter count increases by 2k times. We aggregate channel and highfrequency information using fully connected layers. For computational simplification, we divide channels into Cb blocks, each with cd channels, i.e., Hin ∈ R2×k×H×W×Cb×Cd . Along the last dimension, two linear transformations of highfrequency information provide fine-grained details such as edges, patterns, and small features. After that, we perform the inverse transform to bring the spectrum back to the physical domain, obtaining FFDM ∈ RH×W×C . Additionally, we employ residual connections to transfer the high-frequency information extracted by FFDM through DTCWT to different depths of the network.

3.2 Frequency-aware Transformer Block

As shown in previous work [Park and Kim, 2022], selfattention can be considered as a low-pass filter, and thus cannot work well in reconstructing high frequency details. For this reason, we propose the Frequency-aware Transformer Block (FTB) with Frequency-aware Cascade Attention (FCA) as well as Dual Frequency Aggregation FeedForward Network (DFFN) to help reconstruct the highfrequency details. As seen in Fig. see Fig. 2b, FCA mainly consists of two cascaded attention modules, i.e., spatialchannel attention and the channel transposed attention.

Spatial-Frequency Attention. To give the high-frequency components more focus, we propose the Spatial-Frequency Attention (SFA) shown in Fig. 2d, which combines highfrequency and channel information into self-attention to enhance recovery of high-frequency details.

In detail, we first partition the input into rectangular windows of dimensions H×W wh×ww , where wh and ww denote the height and width of the rectangular window. For the i-th rectangular window feature Xi ∈ R(wh×ww)×C , we set the queries, keys, and values as QiSF , KiSF , and V i SF , generated by linear projection matrices without bias. The self-attention computation process is expressed as

where B represents relative position encoding. Additionally, following the Swin Transformer [Liu et al., 2021], we use a shifting window operation to capture more spatial information. Finally, reshaping and concatenating all Y i SF-A results in the output spatial self-attention feature YSF-A.

To extract spatial, frequency, and channel features, we perform multi-feature extraction using YSF -A, X, and VSF . This involves Spatial-Projection (SP), Channel-Projection (CP), and Frequency-Projection (FP) modules. Initially, we employ a series of 1 × 1 convolutions and depth-wise convolution on VSF for global and local feature fusion, obtaining preliminary channel features YC. Subsequently, we perform multidimensional mapping on the features separately for spatial, channel, and frequency dimensions as

where YS represents the output after spatial mapping. We extract spatial information from YSF -A using a set of 1×1 convolutions and depth-wise convolution, followed by channelwise division. To learn more complex spatial feature representations, we introduce non-linear relationships for one-half of the channels. Finally, aggregation along the channel dimension completes the spatial feature mapping.

On the other hand, YF represents the extraction and transformation of frequency features. We initially map X to the frequency feature space using a fully connected layer. Global high-frequency information is extracted through a 1 × 1 convolution , followed by 3 × 3 max-pooling and a series of 1 × 1 convolutions and activation layers to aggregate global and local high-frequency information, resulting in the highfrequency enhanced YF . We simulate a 19 × 19 convolution using depth-wise dilation convolution (DWD-Conv) to increase the receptive field, achieving significant information extraction within channels. This convolution combination considers both spatial and channel information, greatly reducing parameter count.

Two types of outputs are provided, YC1 and YC2. The former further enhances YC, while the latter focuses on feature extraction and mapping in the channel dimension. Next, we aggregate high-frequency and low-frequency information. In addition to fusing high-frequency information, we aim to preserve low-frequency information as much as possible. Therefore, no additional fusion is applied to the self-attention output YSF -A. For the high-frequency feature YF , we add YSF -A and YC2 to it to ensure that low-frequency information still dominates. For the extracted channel information YC2 and frequency information YF , to reduce computational complexity, we pass their global information to YSF-A through average pooling layers.

Finally, we perform feature fusion through cross-branch weighting. The feature fusion process is as follows:

where f(·) represents the sigmoid function. Through the cross-branch weight fusion approach, we attempt to integrate the missing channel and frequency information in selfattention and the globally focused low-frequency information.

Channel Transposed Attention. Channel Transposed Attention (CTA) adopts a different strategy compared to SFA, conducting self-attention computation along the channel dimension. We employ a similar approach by dividing the channels into multiple heads, applying channel attention as illustrated in Fig. 2e. Let the input be YSF , the calculation of queries, keys, and values (QC, KC, and VC) in the self-attention process is expressed as

where α is a learnable temperature parameter used to adjust the dot product. Finally, we obtain the channel attention feature YCA by reshaping the connections of different attention heads. Unlike SFA, to reduce computational overhead, we only aggregate spatial and channel features in channel attention. However, for other feature dimensions, we use the same projection approach to obtain YC1, YC2, and YS. We then proceed with information extraction and cross-weight transfer as

where f(·) represents the sigmoid function. Finally, we weight the channel attention output with spatial features, parallelly cross-weighting the extraction of channel features with spatial information. This complementary approach aggregates spatial and channel information.

Dual Frequency Aggregation Feed-Forward Network. To better integrate the advantages of Transformer and CNN for dual-frequency aggregation, we present the Dual Frequency Aggregation Feed-Forward Network (DFFN) (see Fig. 2(c)). Specifically, the input YSF or YCA is first projected onto Xin through a fully connected layer, followed by a GELU activation function. Then, a frequency gate fg is used to extract information in the frequency domain. The frequency gate is a submodule comprising two convolution layers that separate the input feature into low-frequency and high-frequency information, respectively. Then the low-frequency information remains unchanged, while the high-frequency information undergoes a series of 1 × 1 convolutions and depth-wise convolution(DW-Conv) to enhance details as

Xfg represents the multiplication of the information from the two parts, resulting in a feature map that combines both low and high-frequency information. Finally, another fully connected layer maps the fused feature map back to the original feature space XDFFN . Through this structure, DFFN can extract frequency domain information while preserving spatial domain information.

At the end of DFFN, to compensate for the loss of longdistance detail information resulting from the transition from DFB to different layers, we incorporate a series of 1×1 convolutions and depth-wise convolution at the end of each FTB group, based on the depth. This allows us to recover highfrequency details after the self-attention mechanism using the High-Frequency Recovery Block (HFRB).

3.3 Loss Function

Following the previous work, we use the L1 loss to minimize the distance between the model predictions ISR and the ground truth IHR, expressed as

4 Experiments

4.1 Experimental Setup

Datasets and Metrics. We followed the methodologies established in previous studies to train and test our model. Specifically, experiments were conducted for upscaling factors of ×2, ×3, and ×4. Two training datasets, DIV2K [Lim et al., 2017] and Flickr2K [Radu Timofte and Zhang, 2017], were used for model training. Additionally, five benchmark testing datasets-Set5 [Bevilacqua et al., 2012], Set14 [Zeyde et al., 2010], B100 [D. et al., 2002], Urban100 [Huang et al., 2015], and Manga109 [Matsui et al., 2016]-were used to evaluate the model. Image quality was evaluated using PSNR and SSIM metrics, computed on the Y-channel (brightness) in the YCbCr color space. The lowresolution (LR) images were created by downscaling the corresponding high-resolution (HR) images using bicubic interpolation.

Implementation Details. In our training setup, the model was configured with a patch size of 64×64, and a batch size of 32. The training process comprised 500,000 iterations, with an initial learning rate of 2 × 10-4 . The learning rate was halved at specific milestones: [250K, 400K, 450K, 475K]. Data augmentation techniques, including random horizontal flipping, and rotations at 90°, 180°, and 270°, were applied to the training set. For optimization, the Adam optimizer was employed with β1 = 0.9 and β2 = 0.99 to minimize the L1 loss. Additionally, the model was trained using the PyTorch toolkit on 4 NVIDIA 3090 GPUs.

Table 2: Ablation studies. The models are trained on DIV2K and Flickr2K, and tested on Set5(×2)

4.2 ComparisonWith State-of-the-Arts

We compare the effectiveness of the proposed FreqFormer with several state-of-the-art Single Image Super-Resolution (SISR) methods, including VDSR [Kim et al., 2016a], EDSR [Lim et al., 2017], IMDN [Hui et al., 2019], LatticeNet [Luo et al., 2020], ESRT [Lu et al., 2022], SwinIR [Liang et al., 2021] , SwinIR-NG [Choi et al., 2023], CRAFT [Li et al., 2023a], and SRFormer [Zhou et al., 2023]. Quantitative Results. The quantitative results on five benchmark datasets are presented in Table 1. The FreqFormer, proposed in this study, outperforms all comparative methods on the benchmark datasets, showcasing significant advantages across all test cases. Compared to traditional CNN-based methods, such as EDSR, the FreqFormer exhibits notable performance improvements on the Manga109 dataset at scaling factors of ×2, ×3, and ×4, achieving improvements of 1.11

dB, 1.35 dB, and 1.24 dB, respectively. Notably, when compared to other Transformer architectures with similar parameter counts, like SwinIR and ESRT, FreqFormer consistently achieves superior performance. On Urban100 and Manga109 datasets at scaling factor of ×2, our model also demonstrates performance gains of 0.34 dB and 0.26 dB. These quantitative results collectively underscore the effectiveness and necessity of the self-attention modules in restoring high-frequency information.

Qualitative Comparison. In Fig. 3, we present qualitative comparisons at ×4 magnification. For distant high-frequency details, our model demonstrates state-of-the-art results. In img 024, we are able to restore high-frequency details that other models cannot, while alternative methods may result in blurriness or artifacts in these complex areas. Similar observations can be found in img 059 and img 092, suggesting that our approach effectively reduces artifacts, preserves more structures, and captures surprising details.

4.3 Real-world Image Super-Resolution

In Fig. 4, we present the visual results generated by different methods on real-world low-resolution images. In the first row of visual comparisons, the wall texture created by our FreqFormer appears more natural, clear, and accurate, eliminating the edge distortions caused by traditional Transformers. In the second row, the glass curtain produced by FreqFormer is sharper and more cohesive. It is evident that our approach can produce and restore sharper edges, especially noticeable in visual effects, demonstrating the significant potential applicability of our method in real-world scenarios.

4.4 LAMVisualization

As seen in Fig. 5, the pixels restored by FreqFormer extend across almost the entire image, in contrast to SwinIR, which only aggregates within a limited range. These results indicate that our approach can activate more pixels to reconstruct low-resolution input images. These observations confirm the effectiveness of our method and demonstrate the superiority of our approach from an interpretability perspective.

4.5 Ablation Study

For a fair comparison, we trained models on the DIV2K dataset and the Flickr2K dataset, and tested them on the Set5 dataset in ablation studies. All models share the same implementation details as FreqFormer, except for the iteration count, which is set to 300K. Additionally, we set the output size to 3×256×256 for calculating FLOPs.

Alternate Strategy. To showcase the efficacy of FDM, SFA, and CTA, we carried out several sets of experiments, as depicted in the structure shown in Fig. 2a. Firstly, we compared the performance of DFFN with and without FDM, as indicated in the first and second rows of the table. We replaced all attention blocks with SFA and CTA, respectively. The third row represents the concatenation of SFA and CTA, alternately used in the FCA. By comparison, we can observe that the model using only SFA performs significantly better than the one using only CTA. However, the best performance (38.27 dB) is achieved when alternating between CFA and CTA. Furthermore, the combined use of CFA and CTA results in a decrease in Params by 68K and FLOPs by 4.4G compared to using SFA alone. This suggests that CTA serves as a complementary block to SFA and plays a crucial role in image restoration. We also performed an ablation study on FDM, and the results indicate that combining high-frequency information and channel information after shallow feature extraction effectively restores high-frequency details.

Different Projection. We verify the effectiveness of projection parts in Table 2b. Firstly, we conducted an ablation study on Spatial-Projection, providing a PSNR of 38.26dB at 30K iterations. In the second row, we tested the most critical ablation experiment on Frequency-Projection. Despite having 719K parameters, the model achieved only 38.25dB PSNR, indicating the effectiveness and necessity of restoring high-frequency information after the self-attention mechanism. The last row, Channel-Projection, demonstrates that feature restoration solely in spatial dimensions is insufficient. To make better use of information for image restoration, channel information needs to be incorporated.

5 Conclusion

In this paper, we propose FreqFormer, a novel Hybrid Attention Transformer. Our model integrates spatial, frequency, and channel information, activating more pixels for reconstruction through multi-branch aggregation. Additionally, by blending frequency and channel information, along with various frequency aggregations, it effectively restores intricate details in images, such as edges, patterns, and small features. Extensive experiments have demonstrated that our FreqFormer outperforms several state-of-the-art methods in various image super resolution tasks.

Acknowledgements

This work is supported in part by the National Key Research and Development Program of China, under Grant 2022YFF1202104, National Natural Science Foundation of China, under Grant (62302309,12326619), Shenzhen Science and Technology Program (Grant No.JCYJ20220818101014030), Open Fund of National Engineering Laboratory for Big Data System Computing Technology (Grant No.SZUBDSC-OF2024-23).