Exploring Fourier Prior for Single Image Rain Removal

1 Introduction

Outdoor computer vision systems are increasingly widely used, such as objection tracking, autonomous driving, and pedestrian detection. Still, the performance of these tasks is severely impaired under various rain conditions [Comaniciu et al., 2003; He et al., 2017]. Hence, image deraining is an important pre-processing task in the computer vision community and has drawn much research attention in recent years [Fu et al., 2017a; Jiang et al., 2020; Wang et al., 2020].

∗Corresponding author

Earlier, with the success of the dark channel prior [He et al., 2010] in the dehazing task, exploiting prior knowledge to obtain clean images in the area of image deraining once rose. By exploring the physical properties of the rain layer and background layer, various priors are introduced to separate rain streaks from clean images. The representative methods achieved respectable deraining results proving the advantage of priors via dictionary learning [Kang et al., 2011], discriminative sparse coding [Luo et al., 2015], and Gaussian mixture models [Li et al., 2016]. However, they depend on time-consuming iterative computations and subjective prior in specific scenarios, which lead to efficiency issues and poor adaptability in diverse rainy images from different sources.

Recently, CNNs have achieved significant success in single image rain removal. By end-to-end training, CNNs learn mapping rules from paired rainy/clean images. Researchers mainly focus on the design of complex networks for better deraining performance [Fu et al., 2017a; Ren et al., 2019; Jiang et al., 2020; Jiang et al., 2020; Chen et al., 2021]. For example, the network in [Jiang et al., 2020] consists of a multi-scale structure and Conv-LSTM modules with dense connection between each unit. Despite of the impressive capability to restore clean images from rainy images, these methods neglect the intrinsic prior of the physical properties of rain streaks, making them easily fall into the problem of overfitting on one single synthetic dataset [Wang et al., 2020].

In this paper, we introduce a novel prior for image deraining. The motivation comes from a well-known property of the Fourier transformation: the Fourier phase spectrum preserves high-level semantics, while the amplitude spectrum contains low-level features [Oppenheim et al., 1979]. Fig. 1 shows the results of exchanging the Fourier amplitude and phase spectrum of paired rainy/clean images. (a) and (c) are reconstructed with the phase of rainy image and the amplitude of ground truth, and (b) and (d) are obtained with the amplitude of rainy image and the phase of ground truth. From these images, three interesting phenomena arise. First, rain streaks in (a) and (c) are greatly suppressed, meaning most of the rain streaks information is preserved in the amplitude spectrum of rainy images. Second, comparing the ground truth and (a)/(c), we can see that the fine-grained structures of clean images are well preserved after replacing the phase spectrum with which of the rainy images. This indicates that the phase of rainy images keeps the similar background structures as the ground

Real-world data

Synthetic data

𝑃 : the Fourier phase spectrum (

) 𝐴 : the Fourier amplitude spectrum (

)

Figure 1: Examples of Fourier reconstructions. (a) and (c) are reconstructed with the phase of rainy images and the amplitude of ground truth. In contrast, (b) and (d) are reconstructed with the phase of ground truth and the amplitude of rainy images. There are three observations. First, it is obvious that rain streaks are diminished in (a) and (c), compared with (b) and (d). This means the amplitude of rainy images contains most rain streaks information. Second, similarities between ground truth and (a), (c) indicate that the phase of rainy images keeps background structures similar to ground truth. Third, the homogeneous banded degradation appears over the whole image in (b) and (d). This shows that the amplitude of rainy images shares the same physical characteristic across the synthetic and real-world domain.

truth. Third, (b) and (d) share a homogeneous banded degradation over the whole image, showing that the amplitude of synthetic and real-world rainy images have similar physical characteristics. Naturally, processing the amplitude spectrum of rainy images independently has the potential of effective rain removal and generalization to real-world datasets. Besides, the phase spectrum of rainy images can also be used to refine structural details of the background. In this way, the Fourier prior is achieved by learning the transformation of the amplitude and phase spectrum separately.

Following the Fourier prior, we design a simple but effective two-stage architecture by processing the amplitude and phase spectrum of rainy images sequentially. At the first stage, the network learns the mapping of the amplitude spectrum from the rainy domain to the clean domain. And the second stage learns the phase transformation and refinement of images, with the phase spectrum of the rainy images as part of input. To facilitate the transformation of the Fourier spectrum, we introduce a Fourier amplitude residual block (FARB) and a Fourier phase residual block (FPRB). FARB and FPRB alter the amplitude and phase spectrum respectively in the residual branch, avoiding destruction in the mainstream. Finally, the Fourier-prior-based architecture (FPNet) is built with two UNet-like networks [Ronneberger et al., 2015] and FARB and FPRB as basic units. More sophisticated architecture may bring in larger performance gains, but that is not the priority of this paper. Compared with recent deep-learning-based methods, better performance reflects the superiority of FPNet and Fourier prior. More importantly, our proposed FPNet shows great generalizability on real-world rainy datasets when only trained on synthetic datasets.

The main contributions of this work are as follows:

• We propose a novel prior - the Fourier prior, which contains the physical properties of rainy images across different domains.
• We implement a simple but effective two-stage deraining network, where two effective residual blocks are introduced to facilitate the Fourier amplitude and phase spectrum transformations.
• Experiments demonstrate that our method outperforms the state-of-the-art methods on synthetic datasets and possesses better generalizability to real-world datasets.

2 RelatedWorks

2.1 Single Image Deraining

Generally, there are two kinds of single image deraining: model-driven and data-driven.

Model-driven approaches separate the rain layer from the background layer by exploring physical attributes and prior knowledge of rain streaks. For example, by using dictionary learning to analyze morphology, rain streaks are extracted in [Kang et al., 2011]. to remove rain streaks in high frequency regions. In [Luo et al., 2015], based on image patches, a discriminative sparse coding is proposed to obtain rain-free content. And a Gaussian mixture models based patch prior [Li et al., 2016] is introduced to adapt to rain marks in multiple directions and scales. But efficient issues and poor adaptability limit these method .

With the rise of deep learning and CNNs, data-driven methods have dominated the single image rain removal task in recent years. To use properties of rain layers, a multibranch network using multi-task learning (DerainNet) is proposed [Yang et al., 2017]. In [Fu et al., 2017a], the authors present a three-layer network on high-frequency parts to simplify the learning processing. In [Zhang and Patel, 2018], a density-aware multi-stream CNN (DIDMDN) is introduced, which can automatically recognize the rain-density information. With consideration of the correlations between different stages of rain removal, a recurrent context aggregation network (RESCAN) is proposed in [Li et al., 2018]. Unlike the multi-scale network using cycle spinning (UMRL) [Yasarla and Patel, 2019], a progressive network (PReNet) is wellbehaved with a simple architecture [Ren et al., 2019]. The authors designed a semi-supervised transfer learning framework (SEMI) to tackle the domain adaptation in [Wei et al., 2019]. By digging the correlations of rain streaks in an image, a multi-scale progressive fusion network (MSPFN) is introduced in [Jiang et al., 2020]. In [Zamir et al., 2021], a multistage progressive network (MPRNet) is proposed by further using features across each stage. On the basis of MPRNet, half instance normalization is introduced to be an important component of the network (HINet) in [Chen et al., 2021]. Unlike the data-driven methods mentioned above, our approach introduces a novel prior that can improve the generalization ability of the network while taking advantage of the powerful mapping capabilities of CNNs.

2.2 Fourier Transform in Deep Learning

Some studies recently exploited Fourier transform on domain adaptation and domain generalization, demonstrating the importance of phase information. In [Yang et al., 2020], the authors propose a phase consistency in source-target image translations, more competitive than the cycle consistency used for domain adaptation segmentation tasks. Similarly, in [Yang and Soatto, 2020], by replacing a small area in the centralized amplitude spectrum, images in the source domain become target-like. The work of [Xu et al., 2021] explores the Fourier transform as a new type of data augmentation and achieves a remarkable performance. And their method is exchanging the Fourier amplitude and phase spectrum of different images from source and target domains. Different from the above work, we swap the amplitude and phase spectrum between paired clean/rainy images. By extensive observation, we summarize three characteristics and use them to conduct our network design.

3 Method

In this section, we will first introduce the definition of Fourier prior in Sec. 3.1. Then the network architecture and loss functions will be presented in Sec. 3.2 and Sec. 3.3.

3.1 Fourier Prior

Before introducing the Fourier prior, briefly reviewing the Fourier transformation is helpful for understanding our work. For an image x ∈ RH×W×1 , its Fourier transformation F(x) is formulated as:

Accordingly, F -1 (x) is the inverse Fourier transformation. Both transformations can be implemented efficiently using the FFT algorithm in [Frigo and Johnson, 1998]. A(x) and P(x) are amplitude and phase spectrums expressed as:

where R(x) and I(x) represent the real and imaginary part of F(x), respectively. For all images and feature maps in our method, the Fourier transformation is computed independently on each channel.

It is well-known that the phase component P preserves high-level semantics of the original signal, while the amplitude component A contains low-level statistics. Intuitively, in a rainy image, P and A would correspond to the image structures and rain streaks respectively. In order to evaluate this, we exchange the phase component P and amplitude component A of the rainy and clean images. Then, a series of common phenomena can be observed across synthetic and realworld datasets, as shown in Fig. 1. Let Xrain and Xclean represent the rainy image and clean image respectively. The rain streaks decrease in F -1 (A(Xclean), P(Xrain)). Meanwhile F -1 (A(Xclean),P(Xrain)) has the clean content similar to Xclean. On the other hand, F -1 (A(Xrainy),P(Xclean)) across synthetic and real-world domains have similar strip degradation which is evenly distributed in the whole image. From this, three important conclusions can be deduced: (1) the amplitude of rainy images contains most rain streaks information; (2) the phase of rainy images keeps fine-grained structures similar to ground truth; (3) the amplitude of rainy images shares the same physical characteristic across synthetic and real-world domains. So, the effective and generalizable deraining across synthetic and real-world domains can be performed on A, while P retains the structural details of clean images. Based on these prior knowledges from Fourier transformation, it's natural and necessary to learn A(Xrain) → A(Xclean) and P(Xrain) → P(Xclean) separately in the image rain removal model.

3.2 Network Architecture

Fourier Prior Network

To exploit the Fourier prior, we design a simple but effective two-stage architecture - the Fourier prior network (FPNet), as shown in Fig. 2. Our FPNet consists of the amplitude rain removal stage and phase structure refinement stage. The first stage restores the amplitude of rainy images to clean rain streaks, and the second stage restores the phase information to refine fine-grained background structures. F -1 (A(Xclean),P(Xrain)) are used as the supervision signal of the first stage. And Xclean are used as the supervision signal of the second stage. Specifically, we do not directly use the first stage output Y1 as the input of the second stage. Instead, we use F -1 (A(Y1),P(Xrain)) to avoid the loss of P(Xrain) since the phase information will be altered when passing through the network [Yang et al., 2020]. By now, the Fourier prior has been embedded into our network.

The first stage consists of 8 Fourier Amplitude Residual Blocks (FARBs) and the second stage consists of 8 Fourier Phase Residual Blocks (FPRBs). And there are three downsamplings and three upsamplings in each stage. Furthermore, the cross-stage feature fusion and supervised attention module are introduced from [Zamir et al., 2021] for the stable optimization procedure.

Fourier Amplitude and Phase Residual Block

To facilitate the transformation of the amplitude spectrum, we propose the Fourier amplitude residual block (FARB). Since the convolution is conducted in the spatial domain, it is non-trivial to directly operate in the amplitude spectrum. To avoid destructing convolutional operation in spatial features, the amplitude spectrum is refined in the residual branch. As shown in Fig. 3(a), before performing the fast Fourier transform (FFT), the input features Fin ∈ RH×W×Cin is fed into one 1 × 1 convolution layer to output Fres ∈ RH×W×Cout . A(Fres) ∈ RH×W×Cout and P(Fres) ∈ RH×W×Cout is calculated by F(Fres) in Eq. 2. Then, A(Fres) is fed into two 1 × 1 convolution layers to output ˆ A(Fres). Subsequently, Ffft = F -1 ( ˆ A(Fres), P(Fres)) can be calculated by using the inverse fast Fourier transform(iFFT) algorithm. Finally, the amplitude residual block can be expressed as:

where Fout ∈ RH×W×Cout is the output feature and Fmain ∈ RH×W×Cout is the main-branch feature.

The Fourier phase residual block (FPRB) is similar to FARB, just swapping the operations between A and P, as shown in Fig. 3(b).

3.3 Loss Function

The Mean Square Error (MSE) produces the over-smoothed image because of its squared penalty. Thus, Mean Absolute Error (MAE) is adopted to balance rain removal and detail preservation in our method.

Let Y1 and Y2 represent the output of the first and second stage respectively. For the first stage, we minimize the following loss functions:

where L1 and Lamp are performed in the space and frequency domain, respectively. For the second stage, we refine the phase spectrum and approximate clean images by using loss functions similiar to the first stage as follows:

The overall loss function used to train FPNet is formulated as follows:

where λamp and λfft are trade-off parameters, both set to 0.05 empirically.

4 Experiment

4.1 Datasets and Settings

Synthetic Data

As in [Jiang et al., 2020; Zamir et al., 2021], we train our model on 13, 712 clean-rain image pairs (for simplicity, denoted as Rain13k in the following) gathered from multiple synthetic datasets. With this single trained model, we perform evaluation on various test sets, including Rain100H [Yang et al., 2017], Rain100L [Yang et al., 2017], Test100 [Zhang et al., 2019], Test2800 [Fu et al., 2017b], and Test1200 [Zhang and Patel, 2018].

Real-world Data

To test the performance of our method in real scenarios, we conduct experiments on the testing samples of recent public real-world rainy dataset (for simplicity, denoted as RS in the following) [Quan et al., 2021], which contains nearly 150 rainy/clean image pairs for training and 100 pairs for testing. All the images collected use a DSLR camera to capture rain images. Rain streaks are generated by spraying water, which is widely used to mimic rainy scenes in film industry.

Implementation Details

The networks are trained with Adam optimizer. The learning rate is set to 2 × 10-4 by default, and decreased to 1 × 10-8 with cosine annealing strategy. our models are trained on 256 × 256 patches with a batch size of 64 for 8 × 105 iterations. Random flipping and rotation are applied as data augmentation. PSNR and SSIM are used for the quantitative evaluation. Note that as the human visual system is sensitive to the Y channel of a color image in YCbCr space, PSNR and SSIM are computed based on this luminance (Y) channel [Yang et al., 2017].

4.2 Evaluations on Synthetic Data

Table 1 reports the quantitative results with recent stateof-the-art methods. The proposed FPNet not only achives the best performance on each dataset, but also achieves 0.34dB improvement compared to the recent best method HINet [Chen et al., 2021] when averaged across all datasets. Fig. 4 illustrates that FPNet preserves more fine-grained structures, with powerful deraining performance. This phenomenon shows that keeping the phase information unchanged as the second stage input assists in detail recovery.

4.3 Generalization on Real-world Data

To test our method's generalizability, the RS dataset is evaluated by using the model trained on Rain13k without retraining. As shown in Fig. 5, more real rain streaks are removed by our method while retaining reliable content information and details. At the same time, the results reported in Table 2 prove the favorable generalizability of FPNet, achieving 0.27dB improvement compared with the second-best method (MPRNet). This illustrates the Fourier prior can help the network learn the intrinsic characteristics of rainy images across real-world and synthetic domains.

4.4 Ablation Study

Fourier Prior

To validate the effectiveness of the Fourier prior, we design three variants of networks: Model 1 represents performing P(Xrain) → P(Xclean) first and A(Xrain) → A(Xclean) second; Model 2 represents performing Xrain → Xclean at both stages; Model 3 represents performing A(Xrain) → A(Xclean) first and P(Xrain) → P(Xclean) second, but using Y1 as the input of the second stage.

As shown in Table 3, Model 1 presents the worst performance. It's hard for Model 1 to learn the transformation since F -1 (A(Xrain), P(Xclean)) is full of degradation across the whole image, which means the first sub-network is not fully utilized. Second, Model 3 surpasses Model 2 with 0.53dB on RS dataset. This verifies the effectiveness of the proposed architecture. Third, FPNet shows better performance than Model 3, meaning that keeping P(Xrain) invariant play an significant role in the Fourier prior. All of the above results reflects the superiority of FPNet and Fourier prior.

FARB & FPRB

To verify the effectiveness of FARB and FPRB, we ablate the usage of FARB and FPRB by utilizing the regular residual block. Table 4 shows our proposed FARB and FPRB significantly outperform the regular residual block. The results indicate that both modules are the key ingredients of FPNet.

Complexity

To validate the efficiency of our method, we test the MACs and inference time using 100 images with a size of 3 × 256 × 256 on one Nvidia 3090 GPU. Comparisons on average MACs and run-time are shown in Table 5. The results show that our FPNet is able to achieve efficient calculation consumption and inference time.

5 Conclusion

In this paper, we present the Fourier prior of rainy images that are effective for improving generalizability and performance and designed Fourier prior network for single image deraining. The experiments on synthetic and real-world rainy images illustrate our method can restore clean images from rainy images with contents and details well preserved. But if the image is degraded severely by heavy rain, the phase spectrum cannot preserve the original image structure information well. Despite this limitation, the effectiveness of our method suggests that physical priors are still worth exploring, and more attention can be paid to the frequency domain for single image rain removal.

Acknowledgements

This work was supported by the National Key R&D Program of China under Grant 2020AAA0105702, the National Natural Science Foundation of China (NSFC) under Grants U19B2038 and 61901433, the University Synergy Innovation Program of Anhui Province under Grants GXXT-2019-025, the Fundamental Research Funds for the Central Universities under Grant WK2100000024.