A Novel Learnable Interpolation Approach for Scale-Arbitrary Image Super-Resolution

1 Introduction

The single image super-resolution (SISR) task aims to recover high-resolution images from degraded low-resolution images. To address the SISR task, there are mainly two categories of SISR methods, namely traditional methods [Aghighi, 2015; Akhtar et al., 2015] and deep learning based methods [Dong et al., 2015; Lim et al., 2017; Zhang et al., 2018b; Zhang et al., 2018a; Liang et al., 2021; Liu et al., 2021]. In the last decades, CNN based methods [Lim et al., 2017; Zhang et al., 2018a; Zhang et al., 2018b] have gradually become the mainstream approach for SISR problems. Nevertheless, most of them are designed as scale-specific networks. For real-world applications, it is not uncommon to zoom the image to a customized scale factor instead of a specific scale factor. Thus, it is vital to design scale-arbitrary networks to enable a wider application.

∗Corresponding author.

As we know, traditional image interpolation approaches such as nearest-neighbor interpolation, bilinear interpolation, lanczos interpolation and bicubic interpolation [Lin et al., 2008; De Boor, 1962] are naturally capable of addressing the scale-arbitrary super-resolution problem. They map the neighboring pixels to the corresponding target pixels in HR via the selected kernel functions. However, these kernel functions are fixed, leading to mediocre performance.

Several works attempt to address scale-arbitrary super resolution (SR) using deep learning methods. The multi-path learning strategy is widely employed to address super resolution with different scale factors [Lim et al., 2017; Ahn et al., 2018; Wang et al., 2018b]. To be concrete, the principal components of the model (i.e., the feature extraction components) are shared and then scale-specific pre-processing paths and upsampling paths are attached. Meta-SR [Hu et al., 2019] applies meta-learning to tackle the super-resolution task of different non-integer scale factors in a single network. ArbSR [Wang et al., 2021] shares most of the parameters for different scales and handle the SR tasks with asymmetric scale factors. It is observed that ArbSR includes the bilinear interpolation on the scale-specific paths which limits the capability of dynamic scale-aware filters. The multi-path learning strategy requires high computational and memory costs and bears the disadvantage of being complicated and limited when it comes to continuous scale factors. The scale factor is used for selecting the path and adjusting the weight according to the paths. The alternative is to treat the scale factor as the parameter of the function. LIIF [Chen et al., 2021] first tries to represent an image as a continuous function that learns the continuous representation of images that follows the idea of implicit neural representations. However, LIIF does not make full use of the scale factor and the LR images, thus resulting in relatively low performance on ×2, ×3, ×4 scale SR tasks.

It is a tipping point to combine the simplicity of the interpolation and the capability of neural networks. Concretely, the kernel weights are learned by a neural network which can be seen as a function. And the scale factor is treated as one of the inputs of the function. In this way, the kernel weights are varied with different situations of the scale factor without the heavy computation and memory cost required by multi-path learning strategies.

In this paper, we propose a learnable interpolation module that performs better than traditional methods while preserving its simplicity. In the learnable interpolation module, a neural network is introduced to replace the typically fixed interpolation kernel function. To make full use of the scale factor, a scale-aware channel attention module is also proposed for more powerful feature extraction and feature correlation learning. It adjusts the weights of different feature channels with regard to the scale factor to extract informative features and boosts the performance of the model. Besides, the scale-aware channel module is carefully designed to integrate with the widely used residual module for wider application. These two modules are designed as plug-ins to be easily embedded into the mainstream SR models. Baseline networks equipped with our network achieve competitive performance compared with state-of-the-art SISR approaches on symmetric and asymmetric scale factors with almost negligible additional computation and memory costs.

Our contributions can be summarized as follows:

• We propose a learnable interpolation method that amplifies the image to any size in a unified network. It retains the simplicity of commonly-used interpolation methods and surpasses the performance of existing methods on scale-arbitrary SR tasks.
• We propose a scale-aware channel attention module. It dynamically adjusts the weights of the feature channels according to the input scale factor, contributing to better performance for scale-arbitrary SR tasks.
• We conduct extensive experiments on five benchmark datasets. The results show that the network equipped with our plug-in module improves the PSNR value by 0.13dB on average and up to 0.27dB over the existing models with only a slight increase in the number of parameters and computational cost.

2 RelatedWork

2.1 Learning-based Upsampling

To enhance the performance of traditional interpolation methods, several learning-based upsampling methods have been proposed. The standard learning-based upsampling methods are transposed convolution and sub-pixel convolution. The transposed convolution [Zeyde et al., 2010] is essentially the opposite of a vanilla convolution. However, it tends to produce crosshatch artifacts due to zero padding. Also, the upsampled feature values are fixed and redundant. The subpixel convolution [Shi et al., 2016] was proposed to circumvent this problem. It generates features with a number of extra channels by convolution and then reshapes the features to obtain the output image. In practice, it is difficult for the deeper networks to ignore the repeating artifacts produced by the sub-pixel convolution layer.

However, these methods can only handle fixed scale factor SR problems. To address this, some upsampling methods that can handle arbitrary scale factors have been proposed in recent years. [Hu et al., 2019] first proposed a Meta-Upsample module to solve the scale-arbitrary SR problem based on meta-learning. [Wang et al., 2021] proposed a scale-aware upsampling module, which is composed of bilinear sampling and conditional convolution and realizes image super-resolution of asymmetric scale factors. [Chen et al., 2021] proposed the LIIF to learn a continuous image representation with a local implicit image function. LIIF achieves state-of-the-art performance on larger-scale tasks but has no superiority on smaller-scale tasks. In addition, several methods have been proposed for learning dynamic interpolation under certain circumstances. [Guo et al., 2021] proposed dynamic interpolation that dynamically learns weights from input views. In contrast to these works, our learnable interpolation method replaces the fixed kernel function with a neural network to solve the scale-arbitrary SR problem.

2.2 Single Image Super-Resolution

Due to the rapid development of deep neural networks, CNNbased SISR approaches have a distinct advantage over traditional approaches. [Dong et al., 2015] first proposed SRCNN for SISR, which uses three convolution layers for superresolution. [Lim et al., 2017] proposed a particularly deep network called EDSR, which removed the batch normalization layers and used residual scaling to stabilize the training process. [Zhang et al., 2018b] proposed a residual dense network (RDN) that combined the advantages of residual blocks and dense connection blocks to improve SR performance. [Zhang et al., 2018a] introduced the attention mechanism to the super-resolution task, and achieved significant improvement. Recently, [Liang et al., 2021] proposed SwinIR, which used Swin Transformer [Liu et al., 2021] and achieved stateof-the-art performance.

To deal with the problem of scale-arbitrary superresolution, [Lim et al., 2017] proposed a Multi-Scale Deep Super-Resolution (MDSR) to integrate multiple modules of different integer scale factors. However, MDSR cannot handle image super-resolution with non-integer scales. [Hu et al., 2019] proposed Meta-SR, which utilizes the meta-upsample module to realize scale-arbitrary super-resolution. However, Meta-SR is also limited in that the horizontal and vertical scale factors must be consistent. Subsequently, [Wang et al., 2021] proposed the ArbSR, which includes feature adaptation modules for scale perception and a scale-aware upsampling module. In ArbSR, the height and width are decoupled, and different features for SR are extracted from different scale factors in the feature extraction stage. In our work, we devise a scale-aware channel attention module that uses the mechanism of co-adaption [Kong et al., 2021] and attention to extract suitable features with regard to different scale factors. The extracted features are fed into our learnable interpolation module, which computes the interpolation kernel and uses the interpolation kernel to interpolate the features to obtain the high-resolution counterparts.

3 Methods

3.1 Learnable Interpolation

Let f be the learnable interpolation function, and c denotes the number of feature channels, Fx and Fy denote the features before interpolation and after interpolation, of which sizes are c×h×w and c×h′×w′ , respectively. Remark that h′ > h and w′ > w. The horizontal and vertical scale factors are rh = w′ w and rv = h′ h . Then the learnable interpolation problem can be formulated as:

Similar to conventional interpolation methods, one may utilize the relative offset to compute the interpolation kernel. The relative offset tensor R with the size 2 × h′ × w′ , can be seen as the mapping from high resolution (HR) space to low resolution (LR) space, and is defined as follows:

where {x} = x - ⌊x⌋, and i, j are the pixel coordinates in HR space. To calculate the exact offset, a translation 0.5 in Equation (2) is used to represent the pixel center.

Rather than fixed kernel functions, in our learnable interpolation module, we utilize a neural network denoted by Hk, to compute the dynamic interpolation kernel,

where ω is interpolation kernel of size k×k×h′ ×w′ and k is the interpolation kernel size. Concretely, Hk consists of three convolution layers, two LeakyReLU activation layers [Maas et al., 2013] and a sigmoid activation layer, whose network structure is depicted in Figure 1. ω(:, :, i′ , j ′ ) denotes that the (i′ , j ′ ) point in the HR space corresponding to the interpolation weight of the k nearest neighbors of the point (i, j) in the LR space.

To achieve better performance, we add a multi-head mechanism [Vaswani et al., 2017] in the learnable interpolation module. Suppose m is the number of heads in the multihead mechanism, and assume that m is a factor of c, i.e., m|c. One can compute m groups of different interpolation kernels ω separately and get the global kernel ˆω of size m × k × k × h′ × w′ by concatenating kernels ω from m groups. ˆ Fx with the size m × cm × h × w is obtained by partitioning the input feature Fx into m equal subsets.

Based on the above discussion, Fy in Equation (1) can be rewritten as:

where Hi is the interpolation operation in our learnable interpolation module. Suppose (i′ , j ′ ) is the coordinate of a pixel in the HR space, let us explain Hi that computes the interpolated feature Fy(i′ , j ′ ). Let ¯ω be the local kernel with the size m× k × k corresponding to the pixel (i′ , j ′ ). And it can be captured by indexing on the global kernel ˆω, i.e.,

For the given scale factor rh, rv, our learnable interpolation produces the corresponding pixel (i, j) in the LR space, i.e.,

To enrich the information for every pixel, we consider the k nearest neighbors of input feature ˆ Fx. The concatenation of these features is defined as:

where Concat refers to the concatenation of a set of vectors. By performing the above operations, the weighted sum of ¯ Fx over ¯ω is reformulated as the interpolated feature of size m× cm ,

where p, q are the traversal of the first two dimensions in ¯ Fx, p ∈ [0,m- 1], q ∈ [0, cm - 1]. Notably, the final output Fy can be easily obtained by reshaping ¯ Fy .

Intuitively, the goal of our learnable interpolation module is to support the interaction and the integration of aspects of CNN-based SISR methods and conventional interpolation methods, i.e., the learnable interpolation module can perform as well as the CNN-based networks on the super-resolution task while preserving the flexibility of conventional interpolation methods. Our learnable interpolation module can provide suitable kernel functions for scale-arbitrary SR tasks by updating the parameters for the interpolation kernel functions through backpropagation. Benefiting from these appealing features, our proposed model can achieve considerable performance.

3.2 Scale-Aware Channel Attention

Since the degradation varies for different scale factors [Simonyan et al., 2014; Kong et al., 2021], it is necessary to take the scale factor into account for better performance. This finding inspires us to take advantage of feature recalibration in terms of different scale factors to improve the reconstruction performance. As demonstrated in Section 4.4, the features exhibit specificity for different scaling factors. To this end, we propose a scale-aware channel attention module by integrating the scale factor with channel attention.

In addition, it is essential to combine the scale-aware channel attention module with the baseline network. It can be noticed that a majority of feature extraction modules of superresolution models contain residual structures for feature extraction. With the goal of wider applicability and better

(c) Scale-Aware Module

Figure 2: Overview of Scale-Aware Channel Attention block. (a) The original residual block. (b) Scale-aware channel attention block. (c) Scale-aware module. For a given scale factor, the scale-aware channel attention module generates adaptive channel weights to extract suitable features in the feature extraction stage.

performance, we combine the scale-aware channel attention module with residual structure.

We denote the input of the residual module as Fi ∈ Rc×h×w , the output as Fo ∈ Rc×h×w and the residual backbone as Hb. The scale-aware channel attention module takes the scale factor into account and recalibrates features:

where Hs is a neural network consisting of two fully connected layers with tanh activation function as shown in Figure 2. Hs can generate suitable channel weights in terms of different scaling factors. Thanks to its simple structure, the introduced parameters and additional computational cost are negligible.

3.3 Integration with SR architectures

The integration of the learnable interpolation module and the scale-aware channel attention into the existing networks is easy since the residual modules are widely used and the upsample module is indispensable for SR architectures. Designed as plug-ins, the learnable interpolation module replaces the original upsample module. Also, the scale-aware channel attention is integrated and jointly trained with the residual module.

As shown in Figure 3, the inference process is divided into two stages. In the feature extraction stage, the scale-aware channel attention module perceives the input scale factor to dynamically adjust the weights of different channels to extract features suitable for the super-resolution task of the current scale factor. In the upsampling stage, the learnable interpolation module uses the dynamically calculated kernel to interpolate the features extracted in the previous stage and obtain high-resolution images. In this way, we are capable of interpolating the features and obtaining the HR image output of the target size.

4 Experiments

4.1 Datasets andMetrics

Following Meta-SR [Hu et al., 2019], we use the DIV2K [Timofte et al., 2017] dataset as our training dataset. For testing, we evaluate our model on five standard benchmark datasets, i.e., Set5 [Bevilacqua et al., 2012], Set14 [Zeyde et al., 2010], B100 [Martin et al., 2001], Urban100 [Huang et al., 2015] and Manga109 [Huang et al., 2015]. The PSNR value [Wang et al., 2004] on the Y channel of the transformed YCbCr color space is utilized to evaluate the performance of our method. Following the settings of previous works, we do the same border crop operation before calculating the evaluation metrics for a fair comparison.

4.2 Implementation Details

Our network uses the BasicSR [Wang et al., 2018a] framework. In order to improve the efficiency of the learnable interpolation function, we use CUDA [NVIDIA et al., 2020; Okuta et al., 2017] Programming language to implement the above logic. The experiment is implemented on an NVIDIA RTX 3090 GPU with PyTorch [Paszke et al., 2019].

We adopt two training strategies for the pretrain phase and the finetune phase [Park et al., 2020]. During the pretrain phase, we specify several specific symmetric scale factor pairs (i.e., ×2, ×3, ×4) and generate LR training images to pretrain the model based on the selected scale factors. During the finetune phase, we select asymmetric scale factors uniformly and randomly in [1.5, 4.5] to generate LR training images to finetune the model based on the pretrained model.

To stabilize the memory consumption during training, we fix the resolution of LR input patches since the extracted features are of the same size as LR input patches before upsampling. A pair of horizontal/vertical scale factors (rh, rv) is randomly selected for each batch during training. We randomly extract 16 ground-truth (GT) patches with the size of 48rv×48rh as a batch input. Then, the GT patches are downsampled into LR patches with the size of 48 × 48 by bicubic downsampling for training. In addition, random flipping is utilized for data augmentation.

We use EDSR [Lim et al., 2017], RDN [Zhang et al., 2018b] and RCAN [Zhang et al., 2018a] as the baseline network and our plug-in module is embedded in the baseline networks to generate three scale-arbitrary networks, namely LIEDSR, LIRDN and LIRCAN. For each model, it is pretrained for 300K iterations and finetuned for 300K iterations. We set L1 loss between SR results and HR images as the loss function. For optimization, we use Adam [Kingma and Ba, 2015] with β1 = 0.9 and β2 = 0.999. In order to stabilize the training process, we use the exponential moving average (EMA) strategy. The initial learning rate is set to 1 × 10-4 and halved at 200K iterations for both the pretrain and finetune phases.

4.3 Experimental Results

We compare our LISR with state-of-the-art methods including Meta-SR [Hu et al., 2019], ArbSR [Wang et al., 2021] and LIIF [Chen et al., 2021]. We present the quantitative results including PSNR value, the total number of model parameters and FLOPs in Table 1 and 2, where FLOPs are measured with a 64 × 64 image as input, and the scale factor are consistent with ArbSR. Also, we present some representative qualitative comparison results in Figure 4 and 5. Since ArbSR does not provide the models and codes of ArbEDSR and ArbRDN, their parameters are not shown in the Tables.

Quantitative Results. It can be observed from Table 1 that our LIEDSR, LIRDN and LIRCAN perform significantly better than the baseline networks. For example, our LIEDSR has better results (40.74 vs 40.39 for ×1.6 SR, 41.03 vs 40.71 for ×1.55 SR) than EDSR on Set5.

Compared to Meta-SR, ArbSR and LIIF, our LISR network also generally achieves better PSNR performance. Among the 135 experiments with symmetric scale factors listed in Table 1, we get the highest PSNR metrics on 130 experiments. For example, our LIRCAN exceeds ArbRCAN by 0.35dB on PSNR on Manga109 with a scale factor of 4 and has 1.1M fewer parameters. Our LIRDN exceeds Meta-RDN by 0.35dB on PSNR on Manga109 with a scale factor of 4 and has 0.3M fewer parameters. Table 2 shows the experimental results of asymmetric scale factors. On the Urban100 dataset, the PSNR value of RCAN is 30.72 / 28.81 / 29.98 dB, the PSNR value of Meta-RCAN is 30.73 / 29.03 / 29.67 dB, and the PSNR value of ArbRCAN is 31.13 / 29.36 / 30.04 dB. And our LIRCAN model outperforms all their models with the PSNR value 31.29 / 29.61 / 30.26 dB.

Specifically, our LIRCAN network performs better on the Set14 dataset for ×2.8 SR (31.20 dB), while the PSNR value of Meta-RCAN is 31.00 dB and that of ArbRCAN is 31.01 dB. Moreover, the number of model parameters added by our plug-in module is minimal among the three methods, and the number of model parameters of LIEDSR and LIRDN is even fewer than that of the baseline network.

Qualitative Results. We compare the visual results on image Belmondo from Manga109 dataset and img 032 from Urban100 dataset as shown in Figure 4. From the upper half of Figure 4, we can see that our network can handle complex textures better than RCAN and ArbRCAN. From the bottom part, it can be observed that RCAN and ArbRCAN models introduce unpleasant artifacts. Figure 5 shows the visual results on img 009 and img 037 from the Urban100 dataset. From the upper part of Figure 5, we can observe that our model produces fewer artifacts than other models. The bottom part of Figure 5 shows that the railway details generated by our model are more realistic and natural than that of RCAN and ArbRCAN. This demonstrates that our network achieves better visual quality with fewer artifacts and more realistic and natural textures than state-of-the-art approaches.

4.4 Ablation Studies

Ablation experiments are performed on Set5 and Manga109 to analyze the relative importance of each component in our method. We use EDSR as the baseline network and introduce two variants. In the case of variant 1, we replace the learnable interpolation module with bicubic interpolation and in the case of variant 2, we remove the scale-aware channel attention module. All variants are trained under the same settings.

The PSNR comparison results of LIEDSR and the two variants on Set5 are shown in Table 3. And the curves of PSNR on the Belmondo image from the Manga109 dataset for factors from 1.5 to 4.5 with a step of 0.01 are shown in Figure 6.

Ablation Study on Learnable Interpolation. From Table 3 and 6, we can conclude that in the case of variant 1, which replaces our learnable interpolation module with bicubic interpolation, is of relatively low PSNR value. When our learnable interpolation module is added, the improvement in PSNR results is significant (e.g., 40.74 vs 37.37 for ×1.6 SR, 34.48 vs 30.95 for ×1.5 ×4 SR). This shows that ordinary interpolation methods have poor performance due to their fixed and simple kernel functions. The learnable interpolation modifies the kernel function of ordinary interpolation, which increases the learnability and thus improves the performance.

Channel Importance Analysis. We first sequentiallymask each channel of the feature, then compute their PSNR value. The experiment is conducted on the butterfly images in the Set5 dataset. It can be seen from Figure 7 that there are horizontal gradient stripes in the figure, which indicates that the feature importance value exhibits specificity with respect to different scale factors. Therefore, for a specific scale factor, the scale-aware channel attention module can extract more suitable features.

Ablation Study on Scale-Aware Channel Attention. From Table 3 and Figure 6, we can see that our LIEDSR achieves higher PSNR values (e.g., 35.38 vs 35.23 for ×2.75 SR, 36.12 vs 35.94 for ×1.6 ×3.05 SR) compared with variant 2. As the only difference is that the scale-aware channel attention module is removed in the case of variant 2, we speculate that the scale-aware channel attention module can extract more suitable features with regard to the scale factor to improve the performance of the model.

5 Conclusion

This paper presents a novel plug-in module that includes a learnable interpolation module and scale-aware channel attention modules. Notably, the learnable interpolation module can amplify an image to any size in a unified network. Moreover, the scale-aware channel attention module can extract features suitable for any corresponding scale factor. Our plug-in module can be easily embedded in existing singleimage super-resolution methods. Experiments demonstrate that baseline networks equipped with our plug-in module provide excellent quantitative and qualitative results for scalearbitrary tasks with a relatively small increase in parameters and computational cost.

Acknowledgments

This work was supported in part by "Digital Silk Road" Shanghai International Joint Lab of Trustworthy Intelligent Software (Grant No. 22510750100), and Shanghai Trusted Industry Internet Software Collaborative Innovation Center.