Lightweight Bimodal Network for Single-Image Super-Resolution via Symmetric CNN and Recursive Transformer

1 Introduction

Single image super-resolution (SISR) aims to recover the corresponding high-resolution (HR) image with rich details and better visual quality from its degraded low-resolution (LR) one. Recently, convolutional neural networks (CNN) based SISR methods have achieved remarkable performance than traditional methods due to their powerful feature extraction ability. For example, Dong et al. [Dong et al., 2014] pioneered the Super-Resolution Convolutional Neural Network (SRCNN). Later, with the emergence of ResNet [He et al., 2016] and DenseNet [Huang et al., 2017], plenty of CNNbased SISR models have been proposed, like VDSR [Kim et al., 2016a], EDSR [Lim et al., 2017], and RCAN [Zhang et al., 2018]. All these methods show that the deeper the network, the better the performance. However, these methods are difficult to be used in real-life scenarios with limited storage and computing capabilities. Therefore, a model that can achieve better performance while keeping the lightweight of the network has become attractive research. One of the most widely used strategies is to introduce the recursive mechanism, such as DRCN [Kim et al., 2016b] and DRRN [Tai et al., 2017]. The other one is to explore the lightweight structure, including CARN [Ahn et al., 2018], FDIWN [Gao et al., 2022], and PFFN [Zhang et al., 2021a]. Although these models reduce the number of model parameters to a certain extent through various strategies and structures, they also lead to a degradation in performance, thus it is difficult to reconstruct high-quality images with rich details.

∗Co-first authors

†Corresponding author

Recently, with the continuous progress of Transformer in Natural Language Processing (NLP), how to apply it to computer vision tasks has become a hot topic. Transformer can model the long-term dependence in the image, and this powerful representation ability can help to restore the texture details of the image. However, most methods blindly use Transformer to replace all original CNN structures, this is unreasonable since the ability of CNN to extract local features is irreplaceable. These features can maintain their own stability under different viewing angles, also called local invariance, which is helpful for image understanding and reconstruction. Therefore, we recommend fusing CNN and Transformer to take full use of the advantages of both to achieve efficient SR image reconstruction.

To achieve this, we propose a Lightweight Bimodal Network (LBNet) for SISR. In LBNet, we use both CNN and Transformer to achieve dual-mode coordination reconstruction. As for the CNN part, we focus on local features extraction. Specifically, we propose a novel Local Feature Fusion Module (LFFM), which consists of a series of Feature Refinement Dual-Attention Block (FRDAB). FRDAB uses the channel reduction strategy to reduce the parameters of the model and introduces channel attention and spatial attention mechanisms to reweight the feature information extracted from different branches. Meanwhile, to balance the performance and size of the model, we introduce the parameter sharing strategy to construct the symmetric-like network, the output of the corresponding shared module of the previous stage will be integrated through the channel attention module as the input of the current module. This method can maximize the use of feed-forward features to restore texture details. As for the Transformer part, we propose a Recursive Transformer to learn the long-term dependence of images, thus the texture details can be further refined with global information. In summary, the main contributions are as follows

• We propose an effective Symmetric CNN for local feature extraction and coarse image reconstruction. Among them, Local Feature Fusion Module (LFFM) and Feature Refinement Dual-Attention Block (FRDAB) are specially designed for feature extraction and utilization.
• We propose a Recursive Transformer to learn the longterm dependence of images. This is the first attempt of the recursive mechanism in Transformer, which can refine the texture details by global information with few parameters and GPU memory consumption.
• We propose a novel Lightweight Bimodal Network (LBNet) for SISR. LBNet elegantly integrates CNN and Transformer, enabling it to achieve a better balance between the performance, size, execution time, and GPU memory consumption of the model.

2 RelatedWorks

2.1 CNN-based SISR

Thanks to the powerful feature representation and learning capabilities of CNN, CNN-based SISR methods have made great progress in recent years [Li et al., 2021a]. For example, SRCNN [Dong et al., 2014] applied CNN to SISR for the first time and achieved competitive performance at the time. EDSR [Lim et al., 2017] greatly improved the model performance by using the residual blocks [He et al., 2016]. RCAN [Zhang et al., 2018] introduced the channel attention mechanism and built an 800-layer network. Apart from these deep networks, many lightweight SISR models also have been proposed in recent years. For instance, Ahn et al. [Ahn et al., 2018] proposed a lightweight Cascaded Residual Network (CARN) by using the cascade mechanism. Hui et al. [Hui et al., 2019] proposed an Information MultiDistillation Network (IMDN) by using the distillation and selective fusion strategy. MADNet [Lan et al., 2020] used a dense lightweight network to enhance multi-scale feature representation and learning. Xiao et al. [Xiao et al., 2021] proposed a simple but effective deep lightweight model for SISR, which can adaptively generate convolutional kernels based on the local information of each position. However, the performance of these lightweight models is not ideal since they disable obtaining larger receptive fields and global information.

2.2 Transformer-based SISR

In order to model the long-term dependence of images, more and more researchers pay attention to Transformer, which was first used in the field of NLP. Recently, many Transformer-based methods have been proposed for computer vision tasks, which also promote the development of SISR. For example, Chen et al. [Chen et al., 2021] proposed a pre-trained Image Processing Transformer for image restoration. Liang et al. [Liang et al., 2021] proposed a SwinIR by directly migrating the Swin Transformer to the image restoration task and achieved excellent results. Lu et al. [Lu et al., 2022] proposed an Effective Super-resolution Transformer (ESRT) for SISR, which reduces GPU memory consumption through a lightweight Transformer and feature separation strategy. However, all these models do not fully consider the fusion of CNN and Transformer, thus difficult to achieve the best balance between model size and performance.

3 Lightweight Bimodal Network (LBNet)

3.1 Network Architecture

As shown in Figure 1, Lightweight Bimodal Network (LBNet) is mainly composed of Symmetric CNN, Recursive Transformer, and reconstruction module. Specifically, Symmetric CNN is proposed for local feature extraction and Recursive Transformer is designed to learn the long-term dependence of images. We define ILR, ISR, and IHR as the input LR image, the reconstructed SR image, and the corresponding HR image, respectively. At the head of the model, a 3×3 convolutional layer is applied for shallow feature extraction

where fsf (·) represents the convolutional layer, Fsf is the extracted shallow features. Then, the extracted shallow features will be sent to Symmetric CNN for local feature extraction

where fCNN(·) represents the Symmetric CNN and FCNN represents the extracted local features. Symmetric CNN is one of the most important components in LBNet, which consists of several pairs of parameter-sharing Local Feature Fusion Modules (LFFMs) and channel attention modules. All of these modules will be introduced in the next section.

After that, all these features will be sent to the Recursive Transformer for long-term dependence learning

where fRT (·) is the Recursive Transformer and FRT is the feature enhanced by global information. Finally, the refined features FRT and the shallow features Fsf are added and sent to the reconstruction module for SR image reconstruction

where fbuild(·) is the reconstruction module, which composed of a 3×3 convolutional layer and a pixel-shuffle layer.

During training, LBNet is optimized with L1 loss function.

Given a training dataset {I iLR, I iHR}Ni=1 , we solve

where θ denotes the parameter set of our proposed LBNet, F(ILR) = ISR is the reconstruct SR image, and N is the number of the training images.

3.2 Symmetric CNN

Symmetric CNN is specially designed for local feature extraction, which mainly consists of some paired parametersharing Local Feature Fusion Modules (LFFMs) and Channel Attention (CA) modules. The parameter-sharing of every two symmetrical modules can better balance the parameters and performance. In addition, each pair of parameter-sharing modules will be fused through the channel attention module, so that the extracted features can be fully utilized.

As shown in Figure 1, Symmetric CNN is a dual-branch network. The shallow feature Fsf will first be sent to the top branch and the outputs of each LFFM in the top branch will serve as part of the input of the corresponding LFFM in the down branch. The complete operation can be defined as

where f T,i LFFM (·) and f D,i LFFM (·) represent the i-th LFFM in the top and down-branch, respectively. f iCA(·) denotes the ith channel attention module. It is worth noting that when i = 1, F D,1 LFFM = f D,1 LF FM (F T ,n LF FM + f 1CA(F T ,1 LFFM )). Moreover, the weight sharing strategy is applied on the paired modules, thus f D,i LFFM (·) = f T,i LFFM (·). Finally, the outputs of all these LFFMs are concatenated and a 1 × 1 convolutional layer is used for feature fusion and compression. Therefore, the most effective features extracted at different levels will be sent to the next part to learn the long-term dependence of images.

Local Feature Fusion Module (LFFM). LFFM is the core component of Symmetric CNN. As shown in Figure 2 (a), LFFM is essentially an improved version of DenseBlock [Huang et al., 2017]. Different from DenseBlock, (1) we use FRDAB to replace the original convolutional layer to make it have a stronger feature extraction ability; (2) we introduce a 1 × 1 group convolutional layer before each FRDAB for dimensionality reduction; (3) local residual learning is introduced to further promote the transmission of information. The complete operation of LFFM can be defined as

where F iFRB represents the output of the i-th (i = 1, 2, 3) FRDAB module in LFFM. f j gc (·) means the j-th (j = 1, 2) group convolutional layer followed by FRDAB. F m-1 LFFM and F m LFFM represent the input and output of the m-th LFFM module, respectively.

Feature Refinement Dual-Attention Block (FRDAB). As shown in Figure 2 (b), FRDAB is a dual-attention block, which specially designed for feature refinement. Specifically, the multi-branch structure is designed for feature extraction and utilization. In this part, the feature will be sent to two branches and each branch uses a different number of convolutional layers to change the size of the receptive field to obtain different scales features. c/2 indicates the operations of halving the outputs. After that, channel attention is used to extract channel statistics for re-weighting in the channel dimension and spatial attention is used to re-weighted the pixel according to the spatial context relationship of the feature map. Finally, the output of these two attention operations is fused by the addition operation. With the help of this method, the finally obtained features will show a stronger suppression of the smooth areas of the input image.

3.3 Recursive Transformer

As we mentioned before, Symmetric CNN is designed for local feature extraction. However, this is far from enough to reconstruct high-quality images since the depth of the lightweight network makes it difficult to have a large enough receptive field to obtain global information. To solve this problem, we introduce Transformer to learn the long-term dependence of images and propose a Recursive Transformer (RT). Different from previous methods, we introduced the recursive mechanism to allow the Transformer to be fully trained without greatly increasing GPU memory consumption and model parameters. As shown in Figure 1, RT is located before the reconstruction module, which consists of two Transformer Modules (TM) and two convolutional layers. The completed operation of RT can be defined as

where f3×3(·) and fTM(·) represent the convolutional layer and the TM, respectively. ⟳ denotes the recurrent connection, which means that the output of TMwill be served as its new input and looped S times. As for the TM, we only use the encoding part of the standard Transformer structure inspired by ESRT [Lu et al., 2022]. As shown in Figure 3, TM is mainly composed of two layer normalization layers, one Multi-Head Attention (MHA), and one Multi-Layer Perception (MLP). Define the input embeddings as Fin, the output embeddings Fout can be obtained by

where fnorm(·) represents the layer normalization operation. fMHA(·) and fMLP (·) represent the MHA and MLP modules, respectively. Like ESRT, we project the input feature map of MHA into Q, K, and V through a linear layer to reduce GPU memory consumption. Meanwhile, feature reduction strategy is also used to further reduce the memory consumption of the Transformer. Following [Vaswani et al., 2017], each head of the MHA must perform a scaled dot product attention, and then concatenate all the outputs and perform a linear transformation to obtain the output. Where the scaled dot product attention can be expressed as

This is the first attempt of the recursive mechanism in Transformer. With the help of this strategy, we can fully train and utilize Transformer without increasing the parameters and GPU memory consumption of the model. We will further discuss its effectiveness in the next section.

4 Experiments

4.1 Datasets and EvaluationMetrics

Following previous works, we use DIV2K as training set. For evaluation, we use five benchmark test datasets to validate the effectiveness of LBNet, including Set5 [Bevilacqua et al., 2012], Set14 [Zeyde et al., 2010], BSDS100 [Martin et al., 2001], Urban100 [Huang et al., 2015], and Manga109 [Matsui et al., 2017]. In addition, PSNR and SSIM are used as evaluation indicators to evaluate the performance of SR images on the Y channel of the YCbCr color space.

4.2 Implementation Details

During training, we randomly crop 48 × 48 patches from the training set as the inputs, and enhance the images with random rotation and horizontal flipping. The initial learning rate is 2×10-4 and finally dropped to 6.25×10-6 by cosine annealing. Meanwhile, the model is trained by Adam optimizer under the PyTorch framework with a NVIDIA RTX 2080Ti GPU. In the final model, the input and output channels of each part are set to 32, three (n = 3) LFFMs are used, and Transformer module recurses 2 times (S = 2). Moreover, we also built a tiny version, named LBNet-T, which only consists of two (n = 2) LFFMs. Meanwhile, the number of channels in LBNet-T has also been reduced to 18.

4.3 Comparison with Lightweight SISRModels

In Table 1, we compare our LBNet with 11 advanced lightweight SISR models. Most of them achieve the best results at the time in the lightweight SISR task. According to the table, we can clearly observe that our LBNet achieves the best results, and our tiny version LBNet-T also achieves the best results under the same parameter level. In addition, the number of parameters and Mult-Adds of LBNet and LBNet-T are also very low, which proves the efficiency of our models. Meanwhile, we also provide the visual comparison between LBNet and other lightweight SISR models in Figure 5. Obviously, SR images reconstructed by our LBNet have richer detailed textures with better visual effects. This further validates the effectiveness of our proposed LBNet.

4.4 Model Complexity Studies

As can be seen from Table 1, our model achieves an excellent balance betweenmodel size, performance, andMult-Adds. In addition, the execution time of the model is also an important indicator to measure the complexity of the model. To make a more intuitive comparison with other models, we provide the trade-offs between model performance, parameter quantity, and execution time in Figure 4. Obviously, our LBNet achieved the best PSNR results under the premise of comparable execution time and parameters. This further illustrates that LBNet is an efficient and lightweight SISR model.

4.5 Ablation Study

Symmetric CNN Investigations. In Symmetric CNN, each pair of LFFMs will be fused through a channel attention module. In order to verify the effectiveness of this fusion method, we conduct a series of experiments to study the different feature interaction methods in Symmetric CNN. It is worth noting that we removed the Transformer part of the model to speed up the training time in this ablation study. Table 2 shows the results of these different feature interaction methods. Among them, FF means the ordinary concatenate operation without attention module, SA and CA indicate the use of spatial attention and channel attention for fusion, respectively. Obviously, our method achieves the best results with the same parameters and Multi-Adds level. This fully demonstrates the effectiveness of our method.

To verify the effectiveness of our FRDAB, we replace FRDAB with some commonly used feature extraction modules in lightweight SISRmodels, like IMDB [Hui et al., 2019] and RCAB [Zhang et al., 2018]. It can be seen from Table 3 that although FRDAB will bring a little increase of parameters and Mult-Adds, its performance has been significantly improved. This gain is considerable, which benefit for lightweight models construction. This fully proves that FRDAB is an effective feature extraction module.

Recursive Transformer Investigations. In order to learn the long-term dependence of images, we introduced the Transformer and proposed a Recursive Transformer (RT). To verify the effectiveness of the proposed RT, we remove RT and provide the results in Table 5. According to the table, we can clearly observe that the introduced RT will increase the number of parameters. However, the increase in Multi-add and execution time is insignificant. In addition, the experiment also shows that the PSNR/SSIM results of the model with RT have been significantly improved. It shows that the introduced Transformers can improve the learning capabilities of the model thus improving the model performance.

Different from previous Transformers, we introduced the recursive mechanism in the Transformer Module (TM). This design can make the Transformer more fully utilized without increasing the number of model parameters. To verify the effectiveness of the recursive mechanism in Transformer, we conduct a series of experiments with different recursion times. In Table 6, TM-N denotes TM recursed N times and TM-0 represents the flat model that no recursive mechanism is used. Obviously, as the number of recursion times increases, the performance of the model will further improve. Meanwhile, we also notice that when the number of recursion times is 3 (TM-3), the improvement of model performance is not obvious. Therefore, we set the recursion times as 2 (S = 2) in the final model to achieve a better balance between mode performance, Multi-Adds, and execution time.

Comparison with Other Transformer-based Methods. Recently, some Transformer-based methods have been proposed for SISR. In Table 4, we provide a detailed comparison with SwinIR [Liang et al., 2021] and ESRT [Lu et al., 2022]. According to the results, we can clearly observe that our LBNet achieved competitive results with fewer parameters and Multi-Adds. Although the PSNR result of LBNet is 0.14dB worse than SwinIR, it is worth noting that SwinIR uses an additional dataset (Flickr2K) for training. This is one of the key factors to further improve model performance. All these results further verify the effectiveness of LBNet.

5 Conclusions

In this paper, we proposed a Lightweight Bimodal Network (LBNet) for SISR via Symmetric CNN and Recursive Transformer. Specifically, we proposed an effective Symmetric CNN for local feature extraction and proposed a Recursive Transformer to learn the long-term dependence of images. In Symmetric CNN, Local Feature Fusion Module (LFFM) and Feature Refinement Dual-Attention Block (FRDAB) are designed to ensure sufficient feature extraction and utilization. In Recursive Transformer, the recursive mechanism is introduced to fully train the Transformer, so the global information learned by the Transformer can further refine the features. In summary, LBNet elegantly integrates CNN and Transformer, achieving a better balance between the performance, size, execution time, and GPU memory consumption of the model.

Acknowledgments

This work was supported in part by the National Key R&D Program of China (No.2021YFE0203700), the National Natural Science Foundation of China (No.61972212), the Natural Science Foundation of Jiangsu Province (No.BK20190089), the Six Talent Peaks Project in Jiangsu Province (No.RJFW011), and the CRF (No.8730063).