Revitalizing Real Image Deraining via a Generic Paradigm towards Multiple Rainy Patterns

1 Introduction

Due to the variety and intricacy of real rainfall scenarios and the ill-posed nature of rain removal problem, existing image rain removal methods tend to focus on specific rainy patterns only, and conduct research on a single or several synthetic rain datasets. However, as shown in Figure 1, real rainfall scenarios usually contain a variety of different rainy patterns, such as rain streak, rain drop, rain haze and rain block, and there is still a large research gap in the current work addressing multiple rainy patterns. Next, we comprehensively introduce some representative work in existing rain removal methods according to different types of supervision, as well as the main contributions made by this paper.

∗Corresponding author: Zhuo Su

1.1 RelatedWork

Synthetic Data-driven Method. Based on supervised learning of synthetic data pairs, image rain removal methods have achieved good deraining performance by designing elaborate network structures [Fu et al., 2017b; Li et al., 2018] and combining relevant priors [Chen and Hsu, 2013; Sun et al., 2014] of rain images.

Depending on whether deraining results are generated directly or not, researchers subdivide synthetic data-driven rain removal methods into end-to-end methods [Guo et al., 2022; Jiang et al., 2022] and rain image model-based methods [Hu et al., 2019; Gao et al., 2023]. End-to-end methods utilise deep neural networks to extract and separate key features of the rain and background layers separately, enabling the generation of clean derained images directly from the background features. Model-based methods obtain the final derained image by designing a model close to the composition of real rain images and generating components that make up the model one by one.

The deraining performance of these fully supervised methods depends largely on the synthesized dataset used. Although they have achieved excellent performance on synthesized data, their effectiveness is greatly compromised when there is a large data distribution difference between the used data and the real data.

Real Data-driven Method. In response to the inability of synthetic rain datasets to cover a wide range of complex rainy patterns, researchers have begun to introduce real data to enhance the generalization ability of image rain removal methods. According to whether synthetic data is used in the training process, real data-driven rain removal methods can be divided into semi-supervised methods [Cui et al., 2022; Huang et al., 2023] and unsupervised methods [Li et al., 2019a; Wei et al., 2021].

Semi-supervised methods often first learn an initial network based on a large amount of synthetic data, and then improve the generalization performance of the network by introducing real data and reducing the difference in feature distribution between different types of data. Unsupervised methods mostly employ the idea of generative adversarial networks [Radford et al., 2015; Jin et al., 2018]. The generator can not only obtain the derained image from the rain image, but also generate the corresponding rain image from the clean background to enrich the network's training data. Subsequently, the discriminator leverages the evaluation result to guide the generator's optimization direction.

Existing semi-supervised and unsupervised methods facilitate the task of image rain removal by effectively capturing rain information from real data. However, the insufficient amount of data mining has resulted in mediocre performance of these methods in rain removal.

1.2 Our Contributions

Faced with the problems of insufficient generalization ability under real scenarios and the inability to process diverse rainy patterns in existing image rain removal methods, we propose to explore them from both data and network levels. Our proposed method not only achieves effective response to multiple complex rainy patterns, but also achieves good generalization performance under multiple types of real scenarios. Our main contributions can be summarized as follows.

• We reconstruct a degradation model for real rain images, and propose a phenomenological degradation pipeline to simulate rain-world images in real scenes. The newly constructed synthetic dataset not only contains multiple rainy patterns, but also accounts for a variety of influences such as illumination and noise.
• A patterns-aware rain removal network is designed to fully leverage the global feature extraction capability of the self-attention mechanism and the local spatial exploration capability of the convolutional operation. Our network achieves effective alignment between the synthetic data domain and the real data domain at the feature level.

• A new generic paradigm for rain removal in real images is proposed. Extensive experiments on five challenging real-world datasets fully confirm the effectiveness and stability of our proposed approach in dealing with multiple rain patterns in real scenarios.

2 Data Preparation

Redesigning a more comprehensive data generation pipeline has proven to be an effective approach in multiple low-level vision tasks [Wang et al., 2021; Meng et al., 2023]. Therefore, we propose to construct a new real rain image degradation pipeline to solve the image rain removal problem at data level. The pipeline takes full account of multiple rainy patterns such as rain streak and rain drop produced by rainfall, as well as indirect effects of rainy weather on the global light and noise levels in the surrounding environment. Based on this pipeline, we synthesize a complex rain dataset MPRain which contains multiple rainy patterns.

As shown in the blue box in Figure 2, the data synthesis pipeline consists of a haze generation stage based on the atmospheric light scattering model and a rain layer superposition stage based on the rain image model. We separately describe each of these two stages below.

2.1 Haze Generation

In rainy weather, the increase of surface moisture and the decrease of temperature often form haze. Therefore, we first take into account the effect of haze when synthesizing rain images. Formula (1) shows the traditional atmospheric light scattering model for haze synthesis.

where I0, H, t, and A denote the input image, the generated haze image, the transmission map and the global atmospheric light value, respectively. On this basis, as shown in formula (2), we synthesize the haze image on the classical RESIDE dataset [Li et al., 2017] by introducing tiny variables and considering environmental factors.

here Iγ, N, and γ(·) indicate the generated haze image, ambient noise and Gamma correction operation respectively. Transmission map t = e-βd reflects the proportion of light that reaches the imaging device after particle attenuation, in which β is the haze density control parameter, d is the depth map obtained by the depth estimation method proposed in [Zhang et al., 2023]. The specific details of haze image synthesis are shown below.

Transmission Map. During the construction of haze images, to control the general concentration of the haze, we initialize β to a constant 2 and add a disturbance factor β1 with a variation range of [-0.5, 0.5) to change the intensity of the haze between different images. The adjusted expression for t is shown below:

Multi-interference Environment. Taking into account that rainy weather brings about cloud accumulation in the atmosphere, humidity rise, and water splashes, causing dimmer brightness and higher noise levels, we incorporate random noise N and Gamma correction into our data construction process to mimic these factors' effects on the imaging of hazy images. Since most of the natural noise is Gaussian noise, we first introduce Gaussian noise N with mean 0 and standard deviation 0.2 into the haze image, and then adjust the global color level of the image by Gamma correction.

2.2 Rain Layer Superimposition

In addition to inducing haze, rainfall also distorts images by introducing diverse rainy patterns with varying shapes, including rain streak, raindrop, rain block and so on. In the rain layer superposition stage, we adopt two rainy patterns generation approaches, convolution and rendering. The rain image generation process satisfies the rain image model [Hu et al., 2019] in formula (4).

in which I2, Iγ, and R indicate the output rain image, the input haze image, and the rain layer with values in the range [0, 255], respectively. Both q and α are luminance control factors for the rain layer. Rain layer R = Rp · tr, here Rp denotes rain pattern layer containing multiple rainy patterns, tr represents rain layer transmission map. The value of tr is affected by the depth of the scene, as shown in formula (5), when the depth of the scene is greater than d0, rainy patterns in the image are gradually blurred with the increase of depth.

here c and d0 are both constants that control the transparency of the R and the minimum depth at which blurring occurs, severally. The details of the synthesis of Rp are shown below.

Convolution Generation. Convolution operation combined with Gaussian filtering can synthesize sharp rain streaks with largely controllable factors such as position, length, width and direction. Combined with the observation that rain streaks are also denser in regions with greater depth of scene, we generate the rain streak layer Rs by stacking the rain streak layer by layer based on the depth map.

where Streak(x) is the sub-rain-streak layer randomly generated based on the convolutional approach, K represents the number of superimposed sub-rain-streak layers and the value is a random integer of [1, 3], and mask(k, x) is the mask of the kth sub-rain-streak layer. As shown in formula (7), mask(·, x) achieves a tight correlation between the density of rain streaks and the depth of the scene by gradually increasing the minimum depth at which rain streaks appear in the sub-rain-streak layer.

in which dk = max{d(x)}/K ∗ k, k ∈ [1,K] denotes the minimum depth at which rain streaks appear in the kth subrain-streak layer.

Render Generation. To ensure the similarity between synthesized rainy patterns and real rainy patterns, we also synthesize rainy patterns using the realism rendering method proposed in [Yang et al., 2016]. The final rain pattern layer Rp is obtained by superimposing many different rendered rainy patterns at multiple scales and combining them with the convolutional synthetic rain streak layer Rs.

2.3 Dataset Comparison

Table 1 provides a detailed comparison of differences in rainy patterns, environmental factors, and data volumes between MPRain and other four synthetic datasets: Rain200L, Rain200H [Yang et al., 2016], Rain14000 [Fu et al., 2017a], and Rain-render [Tremblay et al., 2020]. The Luminance Ratio (LR) represents the brightness comparison between the rain image and the Ground Truth (GT) in the test set, while the Data Volume (DV) signifies the ratio between the number of training images and test images in the dataset.

As can be seen in Table 1, MPRain not only contains many rainy patterns, but also integrates a variety of factors such as noise and light in the environment. Compared with the GT, the synthetic rain image in the MPRain has a lower brightness, which is not only more in line with people's visual feeling on rainy days, but also prompts methods using MPRain to generate clearer and brighter deraining results.

3 Patterns-aware Rain Removal Network

To cope with distributional differences between synthetic and real data, we innovatively design a Patterns-aware Rain Removal Network, which employs a semi-supervised learning strategy and exhibits strong deraining performance on a wide range of types of real data. The overall framework of PRRN is shown in Figure 2, which primarily consists of three components: the self-attentive feature mapping module, the multirepresentation inter-domain alignment module and the highprecision image reconstruction module, which are used for the extraction of image features, the feature alignment between different data domains and the reconstruction of derained images, respectively. Below, we introduce these three modules and the training process of PRRN in detail.

3.1 Self-attention Feature MappingModule

The self-attentive feature mapping module consists of two parts: the shallow feature mapping part only uses a convolutional operation with kernel size of 3 × 3, and the deep feature extraction part contains four basic blocks connected in series. Considering the excellent performance of the selfattention mechanism [Feng et al., 2023; Chen et al., 2023b] in feature extraction, the multi-receptive field feature extraction module based on self-attention proposed in [Liu et al., 2021; Liang et al., 2022] is selected as the basic block. In addition, skip connections are used between basic blocks to ensure the correct transfer of shallow features in the network.

3.2 Multi-representation Inter-domain Alignment Module

The specific structure MIAM is shown as Figure 3, which comprises four parallel submodules. Four submodules are composed of 1, 1, 2 and 3 convolutional layers with kernel sizes of 1×1, 3×3, 5×5 and 7×7, respectively. The outputs of parallel submodules are simultaneously subjected to (1) average pooling and concatenation operations to obtain the hidden-layer code for feature alignment, and (2) concatenation and convolution operations to obtain the multirepresentation output feature for image reconstruction. By employing a meticulous design of multiple submodules, MIAM effectively extracts multi-scale information from input features across various receptive fields. This not only enables the network to capture local information of variable sizes in diverse rainy patterns, but also ensures precise feature alignment in multiple feature subspaces.

3.3 High-precision Image ReconstructionModule

The high-precision image reconstruction module first processes the multi-representation output features using a 3×3 convolution layer. It then adds the resulting outputs to shallow features, and finally employs another 3×3 convolution layer to restore the added features to 3 channels, thus obtaining the final derained image. The addition of shallow features further ensures that the initially extracted important information is not lost during the propagation of the network. What is more, throughout the processing flow of PRRN, feature maps remain the same size as the original image, which effectively avoids information loss due to up and down sampling.

3.4 Training Strategy and Loss Function

The teacher-student network [Su et al., 2021; Cui et al., 2022] is widely used in the field of image restoration due to its powerful ability to integrate synthetic and real data information. Based on the idea of the teacher-student network, PRRN is trained in two stages to obtain a teacher model and a student model, respectively, both of which share the same network architecture.

Stage 1. In the first stage, PRRNuses pairs of synthetic data to pretrain a teacher network. The network is trained on our proposed MPRain dataset for 520 epochs to achieve stable image restoration performance. The loss function Lteacher for the teacher network during training is defined as the structure similarity loss LSSIM between the derained image and the GT, as indicated in formula (8).

where x, and y denote the GT and the derained image, µx, µy and σx, σy denote means and standard deviations of the GT and the derained image, severally. Both of C1 and C2 are constants taken as 0.0001 and 0.0009.

Stage 2. In the second stage, PRRNfirst initializes a student network using the trained teacher network. In the following training steps, the student network not only utilizes the synthetic data in the MPRain dataset, but also introduces 516 unpaired real rain images to improve the network's adaptability to real rainfall scenarios.

For the learning of synthetic data, the student network employs the same structural similarity loss LSSIM as the teacher network. For the learning of real data, we design a new interdomain alignment loss LDA based on MIAM. As indicated in formula (9), the computation of LDA is performed in each of four submodules of MIAM. Firstly, the KL divergence [Hou et al., 2020] between the synthetic feature and the real feature in each submodule is independently calculated. Subsequently, LDA is obtained by combining these individual calculations in equal proportion. The introduction of LDA enables multi-scale alignment of synthetic and real data in multiple feature subspaces.

here HC(·, i), i ∈ [1, 4] denotes the hidden-layer code extracted from the ith submodule of MIAM for the synthetic or real data. As in formula (10), the overall loss Lstudent of the student network consists of structural similarity loss LSSIM of the synthesized data and inter-domain alignment loss LDA of the real data. The network is trained for 20 epochs under the dual guidance of these losses.

in which λ1 and λ2 are constants, taken as 20 and 0.001, severally. Eventually, the total loss Ltotal of PRRN is represented in formula (11).

4 Experimental Results

The experiment is structured into two parts. The first part compares the deraining performance of the proposed method with existing state-of-the-art rain removal methods across various real-world rainfall scenarios. It comprehensively verifies the effectiveness and generalization of the proposed method in authentic conditions. In the second part, ablation experiments are conducted on the individual modules and loss function of the proposed method. This analysis aims to validate the efficacy of each module as well as the design of the loss function.

4.1 Implementation Details

The network is trained on RTX 4090 with initial learning rates of 0.0005 for the teacher network and 0.0001 for the student network. The learning rates are adjusted using MultiStepLR and the Adam optimizer with default settings is used to optimize network parameters. During training, images are cropped to sizes of 128×128, and data augmentation is achieved through random rotation and flipping.

Benchmarks Descriptions and Metrics. Experiments are conducted on five publicly available real datasets, DDNSIRR [Wei et al., 2018a], IVIPC DQA [Wu et al., 2019], MPID [Li et al., 2019b], SPA [Wang et al., 2019] and NTU [Chen et al., 2018]. These datasets contain 147, 206, 185, 146, and 103 images, respectively. Among them, only the rain streak part of images are selected in the MPID dataset, and only 103 frames from the real rain video are randomly selected in the NTU dataset. Experimental performance is evaluated using two non-reference metrics Brisque [Mittal et al., 2012] and Entropy [Gonz´alez et al., 2006].

Comparison Methods. Comparison methods in the experiment are as follows. Fully supervised methods include universal rain removal methods PReNet [Ren et al., 2019], MRFAN [Liang et al., 2022], IDT [Xiao et al., 2022] and DRSformer [Chen et al., 2023b], and the combined removal methods AMRR [He et al., 2020], TRNR [Ran et al., 2021] and UDR [Chen et al., 2023a] for multiple disturbance factors. Semi-supervised methods SIRR [Wei et al., 2018b], Syn2real [Yasarla et al., 2020], MOSS [Huang et al., 2021], SIDD-KD [Cui et al., 2022] and MUSS [Huang et al., 2023].

4.2 Comparison with State-of-the-art Methods

Quantitative Comparisons. Table 2 compares the deraining performance of PRRN with multiple existing state-of-theart methods on five real rain datasets. The deraining effect is reflected by two non-reference metrics Brisque and Entropy, where a smaller value of Brisque indicates a clearer image and a higher value of Entropy indicates a more detailed image. As can be seen from Table 2, whether comparing fully supervised methods or semi-supervised methods, PRRN's student network can demonstrate optimal or suboptimal performance on most datasets, which strongly proves the effectiveness and robustness of our proposed method in a variety of different real scenarios. In addition, compared with the teacher network, the student network trained on real data shows a significant improvement in deraining performance, indicating that our method can effectively capture key information from real rain data.

QualitativeComparisons. Figure 4 provides a comprehensive visual comparison of deraining capabilities of PRRN with other advanced rain removal methods under various real rainy patterns. The figure highlights the proficiency of PRRN in accurately identifying and eliminating various rain patterns in real-life scenarios, yielding final images that are not only clear but also visually appealing.

Figure 5 compares PRRN's performance against other rain removal methods for handling complex rainy patterns involving multiple degradation factors. From details of displayed images, it can be seen that PRRN can not only deal with multiple composite rainy patterns, but also restore better texture details of the image compared to other multi-factor joint rain removal methods.

4.3 Ablation Study

A series of ablation experiments are conducted to verify the effectiveness of our proposed real rain image degradation pipeline and patterns-aware rain removal network.

Effectiveness of the Degradation Pipeline. We validate the image degradation pipeline's effectiveness by assessing the impact of MPRain on the deraining performance of several advanced rain removal methods. Figures 6 (a)-(c) and (d)-(f) show deraining effects of IDT and MRFAN tested on real rain images after training using Rain200H and MPRain datasets respectively. It can be seen that both methods of training on the MPRain achieve better deraining performance on real-world rainy image, which fully demonstrates the practicality of the MPRain and the effectiveness of the degradation pipeline of the real rain image.

Effectiveness of MIAM and LDA. Table 3 verifies the effectiveness of sequentially adding MIAM and LDA into PRRN through the change in values before and after the DDN-SIRR dataset. The table shows that MIAM and LDA both enhance PRRN's training performance to some degree.

5 Concluding Remarks

In this paper, we propose a novel image deraining paradigm that aims to tackle challenging real-world rain removal problems more effectively. Through our innovative "phenomenological degradation pipeline", we have synthesized a broad and diverse rain dataset MPRain, which covers a wide variety of rainy patterns and provides a strong guarantee for practical applications. To better learn from synthetic and real data, we propose a patterns-aware rain removal network. By designing a multi-representation learning module and an interdomain alignment loss, the network realizes the alignment of synthetic and real data in multiple feature subspaces. Numerous experimental results strongly demonstrate the paradigm's exceptional performance and its potential for addressing complex rain removal in real settings.

Limitations. Given the current method's propensity to overexpose and lose sky detail, we plan to incorporate two techniques. The first is to reasonably control the luminance difference between the rain image and the ground truth in the synthesized dataset, to avoid overly obvious luminance enhancement in the result. The second is to separate high and low frequency parts of the image to eradicate rain textures while safeguarding low-frequency details of the sky.

Acknowledgments

This work was supported by the Key-Area Research and Development Program of Guangdong Province (No. 2021B0101410003), the Guangdong Basic and Applied Basic Research Foundation (No. 2024A1515011563), and the Key R&D Project of Guangzhou (No. 202206010091).