DFVSR: Directional Frequency Video Super-Resolution via Asymmetric and Enhancement Alignment Network

1 Introduction

Video super-resolution (VSR) serves as an essential function in video processing. It solves how to reconstruct a highresolution (HR) video from its corresponding low-resolution (LR) video. VSR has attracted considerable attention in both research and industrial communities because of its great value in video surveillance and autonomous vehicles.

∗Corresponding Author

Deep neural networks [Xue et al., 2019; Wang et al., 2019; Cao et al., 2021; Chan et al., 2021a; Chan et al., 2021a; Liu et al., 2022; Chan et al., 2022], have made a substantial impact on enhancing the performance of VSR tasks. BasicVSR [Chan et al., 2021a] designed a succinct pipeline to untangle the VSR task to four basic functionalities, i.e., Propagation, Alignment, Aggregation, and Upsampling. BasicVSR++ [Chan et al., 2022] redesigned BasicVSR by proposing second-order grid propagation and flow-guided deformable alignment. VSR-T [Cao et al., 2021] proposed a transformer-based method to exploit the locality and spatialtemporal data information through different layers to improve performance. These methods achieve remarkable performance on the VSR task. Nevertheless, they still face challenges in preserving sufficient details and avoiding structural artifacts. Recently, frequency-based methods [Xiao et al., 2021; Qiu et al., 2022; Leng et al., 2022; Liu et al., 2021; Fuoli et al., 2019] have attracted great interest owing to their remarkable ability to recover details and structures in video restoration tasks. Natural images can be decomposed into high spatial frequency components that represent the image rapidly changing details and low spatial frequency components that describe the smoothly changing structure. Therefore, effective utilization of this property offers the possibility to recover details and structures efficiently.

Existing frequency-based methods [Xiao et al., 2021; Qiu et al., 2022; Leng et al., 2022; Liu et al., 2021; Fuoli et al., 2019] developed a number of network structures, emphasizing the processing of frequency features to optimize the details and structure of video restoration. STD [Xiao et al., 2021] derived spatial attention maps that indicated highfrequency video content. These maps were then employed to facilitate the transfer of spatial modeling between the networks, enhancing the overall performance of the process. FTVSR [Qiu et al., 2022] and ICNet [Leng et al., 2022] separated different frequencies to help improve the performance of VSR tasks. They have brought considerable performance gains. However, these methods have not fully explored the role of frequency on VSR tasks for the following reasons. Firstly, previous frequency-based methods mainly divide images into high- and low-frequency or different levels of frequency representations. However, such frequencybased representations only describe the appearance information (including structure and texture details) of the object. But, objects in video restoration tasks are directional in motion. These representations ignore the motion direction information of the object. Secondly, not all frequency information is equally useful for video restoration tasks. Without selecting and enhancing effective information, the frequencybased model may easily attend to many low-fidelity and less informative frequency regions. This may degrade model efficiency. Thirdly, existing frequency-based works commonly adopted down-sampling to enlarge the receptive field and extract global information, resulting in the inevitable loss of spatial information. However, VSR tasks are pixel-wise dense problems. Sharp predictions would not be made without the assistance of sufficient spatial information.

In this paper, we propose a novel VSR network, DFVSR, to cope with the above problems. Firstly, we observe the object motion as the combination of movements in different directions information (horizontal, vertical, and diagonal). This coincides with the fact that natural images (frames) can be decomposed into multiple high and low frequencies in different directions. Inspired by this observation, we reconsider motion from a new perspective and propose to view the motion as representations of frequencies in different directions. This way effectively borrows the property of frequency representation of texture details and structure information, and importantly, it also contains motion information in multiple directions that is extremely significant in videos. Secondly, as shown in Figure 2, based on the above representation, we propose a Directional Frequency-Enhanced Alignment (DFEA) for directing the alignment to pay attention to the regions with high-fidelity frequency representations to generate highquality alignment features. We design DFEA to use double enhancements of task-related information to ensure the retention of valid information and the weakening of invalid information. Meanwhile, DFEA also alleviates the limitation of DCN instability during training. Finally, we design a novel Asymmetrical U-shaped network architecture to progressively fuse the alignment features. This architecture enables the intercommunication of the same level of resolution in the encoder and decoder to achieve the supplement of spatial information. Powered by the above designs, we surpass the state-of-the-art (SOTA) methods in the VSR task.

Our contributions can be summarized as follows:

• We propose a novel network, DFVSR, for improving the performance of VSR tasks. To the best of our knowledge, it is the first attempt to explore the potential of directional frequency in VSR tasks.
• We present a novel implicit alignment, DFEA, to pay attention to the regions with high-fidelity frequency representations to generate high-quality alignment features. DFEA also alleviates the limitation of DCN instability during training.
• We customize a novel Asymmetrical U-shaped architecture to enable the intercommunication of the same level of resolution in the encoder and decoder to achieve the supplement of spatial information.
• Experimental results demonstrate the superiority of our

model over SOTA models on both quantitative and qualitative evaluations.

2 RelatedWork

Frequency-Based Methods. Recently, frequency-based methods [Xiao et al., 2021; Qiu et al., 2022; Leng et al., 2022; Liu et al., 2021; Fuoli et al., 2019] have developed a variety of network architectures that concentrate on processing frequency features to enhance the details and structure of image and video restoration. STD [Xiao et al., 2021] extracted spatial attention maps, which represented high-frequency video content from both networks. These maps were subsequently utilized to streamline the transfer of spatial modeling between the networks, ultimately improving the overall performance of the process. Additionally, methods like FTVSR [Qiu et al., 2022] and ICNet [Leng et al., 2022] separated various frequencies to effectively boost the performance of VSR tasks. These frequency-based methods have resulted in significant performance improvements. However, such frequency-based representations ignore the motion information of the object. In this paper, we propose a more effective representation, DFR, to simultaneously model detail and structure information, as well as motion information.

Alignment Methods. The existing alignment works are mainly divided into two categories: (i) flow-based alignment that exploits optical flow to predict motion fields, and (ii) deformable convolution (DCN) based alignment that employs DCN [Dai et al., 2017] to perform implicit feature learning. The flow-based alignment methods rely heavily on flow estimation, and any errors in the flow computation may introduce artifacts around frame structures [Chan et al., 2021a; Tian et al., 2020]. DCN-based alignment methods [Chan et al., 2021a; Chan et al., 2021b; Zhu et al., 2019] have demonstrated significant improvements over flow-based alignment thanks to the diversity of its offset. Nonetheless, DCN-based alignment can be difficult to train [Chan et al., 2021b]. BasicVSR++ [Chan et al., 2022] proposed a flow-guided deformable alignment to help offset learning by using optical flow field guidance. Unlike the aforementioned methods, we propose to pay attention to the regions with high-fidelity frequency representations to generate more valid features. Then, these features are fed to implicit alignment. This way alleviates the limitation of DCN instability during training.

3 Method

3.1 Overall Architecture

The overall architecture of DFVSR is shown in Figure 1 (a). We propose a novel directional frequency representation in DFVSR, which not only borrows the property of frequency representation of detail and structure information but also contains the direction information of the object motion that is extremely significant in videos. DFVSR consists of multiple units, and each unit is abbreviated as DFunit (as shown in Figure1 (b)). We design the DFunit with a directional frequency guidance encoder, a bottleneck, and an asymmetrical decoder. In the encoder of DFUnit, the initial input is low-frequency features, which are progressively fused with higher-frequency alignment features. We propose a novel implicit alignment, DFEA, to generate these alignment features (as shown in Figure 2). DFEA uses double enhancements of task-related information to ensure the retention of highfidelity frequency regions to generate high-quality alignment features. Then, the output of the DFUnit encoder is fed into an asymmetric decoder through a bottleneck. This decoder further processes, refines, and reconstructs features to output the final HR frame. The above design ensures the final video is a high-quality product.

3.2 Directional Frequency Representation

For high-quality output, the restoration of rich texture details and clear structure is essential. Natural images can be decomposed into high spatial frequency components that represent rapidly changing texture details of images and low spatial frequency components that describe the smoothly changing structure. Thus, efficient exploration of this property opens up possibilities for high-quality restoration of texture details and structure. It brings considerable performance gains for frequency-based methods on video restoration tasks. Previous frequency-based methods mainly divide images into high- and low-frequency maps or different levels of frequency. However, such frequency representations do not contain object motion information, ignoring the directionality of object motion.

To handle this limitation, it is necessary to develop a directional representation for precisely detecting the motion directions of objects while providing detail and structure information. We propose Directional Frequency Representation (DFR), which not only borrows the property of frequency representation of detail and structure information but also contains the direction information of the object motion that is extremely significant in videos. Specifically, the ith frame feature Gi can be represented as:

where F denotes the frequency feature, and j reflects the frequency direction. l represents low frequency. h, v, and d represent the high frequencies in the horizontal, vertical, and diagonal directions, respectively. γ is the discrete wavelet transform operation for generating the directional frequency features.

Different from previous frequency-based methods, this representation incorporates directional information, thus object motion can be viewed as an amalgamation of frequency representations from multiple directions. This will provide more sufficient motion information for subsequent interframe alignment.

Directional Frequency Loss. We argue that the directional information of motion in videos is significant for subsequent inter-frame motion alignment operations. Unlike images, an important property of objects in video is their motion. As analyzed above, in previous frequency-based methods, the motion information of objects may be not well modeled. It will cause great challenges for feature alignment and texture refinement. Therefore, we hope our model could not only focus on the appearance (texture details and structure) information of objects but also could pay more attention to the object motions. As shown in Figure 1(a), we propose a novel Directional Frequency Loss (DFLoss) to ensure the accuracy of motion information in the propagation process. We define the DFLoss:

where E is the proposed DFEA (alignment module) that will be described in detail in section 3.4. φ represents the reconstruction layer. In this way, directional frequency maps act as the appearance and motion guidance for the objects in videos. The proposed DFLoss enforces the ingredients of appearance and motion in deep feature layers for blending. Through propagating the directional frequency features, the motion information will be effectively retained.

3.3 Asymmetric Encoder and Decoder

Directional Frequency Guidance Encoder. Different from the previous methods mainly based on using different branches to separately process multiple frequency features, we propose a novel encoder that progressively incorporates alignment frequency features.

In our design, the computationally intensive alignment operation is assigned to high-frequency features. This strategic allocation of network parameters shifts more attention towards detail-representing high-frequency features, while significantly reducing the computational load associated with low-frequency features. As shown in Figure 1(b), we first input the low-frequency feature of ith frame and progressively feed the alignment directional high-frequency features. Specifically, the low frequency F l i is further extracted and generates the feature E0. E0 is the input of the encoder, which consists of several R-FFNs (as shown in Figure 1(c)). We propose an R-FFN module to improve the feed-forward network (FFN) by adding residual connections. Each encoder block contains two inputs: the output features derived from the downsampling operation applied to the previous encoder block, and the corresponding directional alignment frequency feature generated by our proposed DFEA module. The output Em of mth encoder block is defined as:

where E is the proposed DFEA (alignment module) that will be described in detail in section 3.4. C denotes the R-FFNs. The output of each encoder block will be sent to two places for two different purposes. One is to conduct a downsampling operation before feeding into the next encoder block. The other is performed as an added skip connection to provide supplementary spatial information to the decoder.

After this encoding process, the final output feature E3 of the encoder, incorporates alignment frequency information in multiple directions. E3 then passes through the bottleneck, which consists of an R-FFN module to output the feature D0. Asymmetrical Decoder. We design a novel asymmetrical decoder that is able to complement the lost spatial information. The nth decoder block includes two inputs: 1) the output Dn-1 of the previous decoder block, and 2) the output Em of the corresponding mth encoder block. These two inputs are not simultaneously but sequentially input to the decoder block. Dn-1 is input to the upsampling part and R-FFN, respectively. We design the features generated by R-FFN to be fed back into R-FFN iteratively to refine the encoder feature. We design the iteration parameter as the hyperparameter (set to 3 in this paper). Then, we concatenate the feature Dup n generated by upsampling part, the feature Drefine n generated by the R-FFN iterative refinement part, and the feature Em to generate the output Dn of nth decoder block. The decoder process is defined as:

The output feature D3 of the last decoder block is then fed into the reconstruction layer to output the ith video frame I HR i :

where R is the reconstruction layer.

3.4 Directional Frequency-Enhanced Alignment

The other limitation of previous frequency-based methods is that the learning model may attend to some less informative spatial regions with low-fidelity frequency representations. This may degrade model efficiency. Ideally, the model would be able to pay more attention to spatial regions with higher fidelity information. This goal motivates us to design an alignment module that focuses more on highfidelity regions. Therefore, we propose a novel alignment model, DFEA, which consists of double enhancements of task-related information to ensure the retention of valid information in high-fidelity regions and the weakening of invalid information in low-fidelity regions.

As shown in Figure 2, we map F j i into the query Qji , the key Kj i , and the value V j i . Differently, we map F j i-1 into the key Kj i-1 and the value V j i-1 . We do this to preserve task-relevant information in ith frame, as well as adaptively select alignment information in (i-1)th frame. Specifically, we first utilize Qji , Kj i and Kj i-1 to calculate the correlation matrix (C j i )′ and (C j i-1 )′ . Next, the correlation matrix multiplies its corresponding value to obtain the correlation features (F j i )′ and (F j i-1 )′ . Then, to further enhance the correlation information, we concatenate (F j i )′ and (F j i-1 )′ to calculate the enhanced matrix (C j i )′′ and (C j i-1 )′′ . Subsequently, the enhanced task-relevant features (F j i )′′ and (F j i-1 )′′ are generated by the matrix multiplication between the enhanced matrices and the original frequency value features.

In addition, we argue that the frequencies of different directions for the same frame are not independent but correlated.

Therefore, the offsets obtained by learning different direction frequencies in the same frame should also be related to each other. Based on this analysis, as shown in Figure 2, we design to use previously learned offers Θj-1 offset to help learn offers Θjoffset in the current direction. Here, j -1 does not refer to the mathematical meaning, but the previous frequency direction. Last, the alignment features Aji-1 of the j th direction is generated as follows:

where DC(·) denotes deformable convolution.

Unlike the original DCN-based alignment to directly concatenate two features to learn offsets, our proposed DFEA enhances the valid information of ith and i - 1th frames and ensures that its learning offsets are more accurate and valid. This way makes the alignment process more stable.

4 Experiments

4.1 Datasets and Implementation

Datasets. We adopt two widely used datasets to train: REDS [Nah et al., 2019] and Vimeo-90K [Xue et al., 2019]. Following [Chan et al., 2021a], we apply REDS4 as our test set, and REDSval4 as the validation set. We also use Vid4 [Liu and Sun, 2013], UDM10 [Yi et al., 2019] and Vimeo-T [Xue et al., 2019] as test sets.

Implementation and Training Details. We employ Adam optimizer by setting β1 = 0.9 and β2 = 0.999. The learning rate is initialized as 2.5 × 10-5 . We apply RGB patches of size 64×64 as inputs. We set the mini-batch size to 32. In addition to our proposed DFLoss, we also adopt Charbonnier loss [Lai et al., 2017], and ε is set to 1 × 10-3 . The total number of iterations is 600K. The model is trained under the PyTorch framework with an NVIDIA RTX 2080Ti GPU. To speed up the convergence, we develop shallower networks to initialize deeper network parameters.

4.2 Comparison with State-of-The-Art Methods

We compare our method with 12 SOTA models that are BasicVSR++ [Chan et al., 2022], TTVSR [Liu et al., 2022], IconVSR [Chan et al., 2021a], BasicVSR [Chan et al., 2021a], VSR-T [Cao et al., 2021], TDAN[Tian et al., 2020], EDVR [Wang et al., 2019], TOFlow [Xue et al., 2019], FRVSR [Sajjadi et al., 2018], DUF [Jo et al., 2018], RBPN [Haris et al., 2019] and MuCAN [Huang et al., 2017]. The results are in Table 1. Our DFVSR achieves SOTA performance on all datasets for both BI and BD degradations. Compared to BasicVSR++ [Chan et al., 2022] which is the representative progressive method, our model achieves superior performance on all datasets by using fewer parameters. Furthermore, DFVSR also surpasses the recent transformerbased method VSR-T [Cao et al., 2021] by 1.57 dB, 1.54 dB, and 0.56 dB, respectively. The qualitative comparisons are shown in Figure 3. These visual results show that our method restores clearer details and a more precise structure. Furthermore, our method is also very efficient in the recovery of bright and dark areas. This gain is considered significant for VSR tasks.

4.3 Ablation Study

We conduct the ablation study to explore the contributions of the proposed components. We start with a baseline, which is the cascaded R-FFNs. We denote the model generated by the baseline as A0. A0 uses original non-frequency features to train. Then, we gradually insert the components to validate the necessity of each component. The quantitative results are in Table 2.

Effectiveness of the input strategy of frequency features. There are two strategies for input here. One is to input different frequencies into the model training together, and the other is to gradually input different frequencies into the model training. We train the two strategies separately. The model generated by the former strategy is denoted as A1-1, and the latter strategy applies the proposed model to gradually incorporates frequency features to generate a model, which is denoted as A1-2. The results show that A1-2 is better. This illustrates that progressively fusing the frequencies yields better results.

Effectiveness of the DFLoss module. We test the contribution of our proposed DFLoss based on the A1-2 model (trained only by Charbonnier loss). We design to train two models (A2-1 and A2-2), which use Charbonnier loss LCL

and DFLoss LDF . The total loss is defined as: L = λ1LCL+ λ2LDF . To train A2-1, we set λ1 = 0.7 and λ2 = 0.3. On the contrary, to train A2-2, we set λ1 = 0.3 and λ2 = 0.7. The experimental results show that both A2-1 and A2-2 models are better than A1-2. And these two models gain improvements of 0.54dB and 0.83dB respectively. In addition, the performance of A2-2 is further better. This shows that directional frequencies can be an effective guide for the appearance and motion of objects in video.

Effectiveness of the Asymmetrical U-shaped architecture. To explore the contributions of the proposed Asymmetrical U-shaped architecture. We train two U-shape models based on the A2-2 model. A model trained using a common symmetric U-shape architecture (ie, delete Drefine for Equation (5)), called A3-1. Another model is trained with our proposed asymmetric architecture, called A3-2. As shown in Table 2, the results of both A3-1 and A3-2 are better than A2-2. This shows the importance of supplementary spatial information. In addition, the proposed A3-2 achieves better results. It demonstrates the superiority of the proposed Asymmetric U-shape. It also shows that our proposed architecture can supplement more effective spatial information. This shows the necessity of further refinement of the decoder features.

Effectiveness of the DFEA module. We designed three alignment modules based on A3-2. The first alignment module, that is, the original way of DCN-based alignment, performs a concatenate operation on the frequency features of the input two frames. The second alignment module uses one attention operation on the frequency features of the input two frames. The third alignment module is our proposed DFEA, which designs double enhancements of task-related information to ensure the retention of high-fidelity frequency regions. The models generated by these three modules are respectively denoted as A4-1, A4-2, and A4-3. The experimental results show that A4-3 (DFVSR) has the best performance, and A4-2 is the second best. This reflects the importance of enhancing the weight of high-fidelity regions. In addition, it demonstrates the efficiency of our alignment method.

4.4 Temporal Consistency

In Figure 4, we show the comparisons of the temporal profiles between our DFVSR and three SOTA VSR methods, BasicVSR++ [Chan et al., 2022], BasicVSR [Chan et al., 2021a] and EDVR [Wang et al., 2019]. We collect a column (blue dotted lines) and obtain the temporal profile to compare temporal consistency. We can observe that our DFVSR produces smoother temporally consistent results by comparing all of the methods. Furthermore, in the area indicated by the orange arrow, the comparison results demonstrate that our model preserves more details and produces high-quality products with temporal consistency.

5 Conclusion

In this paper, we propose a novel and effective DFVSR network to improve the performance of VSR tasks. We propose a novel representation, DFR, which not only borrows the property of frequency representation of detail and structure information but also contains the object motion information that is extremely significant in videos. Based on this representation, we propose a new implicit alignment module, DFEA, to ensure the retention of high-fidelity frequency regions to generate the high-quality alignment feature. Furthermore, we design a novel Asymmetrical U-shaped network architecture to progressively fuse these alignment features and output the final output. This architecture enables the intercommunication of the same level of resolution in the encoder and decoder to achieve the supplement of spatial information. The above designs ensure the final video is a high-quality product.

Acknowledgments

This work was supported by the National Key R&D Program of China (2022YFB4701400/4701402), SZSTC Grant (JCYJ20190809172201639, WDZC20200820200655001), Shenzhen Key Laboratory (ZDSYS20210623092001004). This work was also supported by National Natural Sciences Foundation of China (62102207).