Explore Internal and External Similarity for Single Image Deraining with Graph Neural Networks

1 Introduction

Images captured in a rainy environment are usually corrupted by rain streaks. These rainy images seriously degrade visibility, which accordingly interferes with the following highlevel tasks, e.g., semantic segmentation, object detection, etc.

∗Wei Wang is the corresponding author

Therefore, it is of great need to develop an effective deraining algorithm to improve the quality of the images so that they can facilitate related applications.

Mathematically, the degradation model for image deraining is usually formulated as a linear combination of a rain streaks component R with a clean background image B:

According to (1), image deraining is a highly ill-posed problem as the rain streaks R and the clear image B are both unknown. To make this problem well-posed, conventional methods [Luo et al., 2015; Chen and Hsu, 2013; Li et al., 2016] usually impose priors on rain streaks and clear images to constrain the solution spaces. However, these priors-based methods usually lead to complex optimization problems that are difficult to solve and have higher time consumption.

Inspired by the success of the deep convolutional neural networks (CNNs) in many vision tasks, recent efforts have been devoted to applying CNNs to solve the image restoration [Jin et al., 2022a; Jin et al., 2023; Jin et al., 2022b; Wang et al., 2021; Wang et al., 2022; Wang et al., 2024a; Wang et al., 2024c; Wang et al., 2024b; Wang et al., 2024d; Cui et al., 2022; Zhu et al., 2020; Xu et al., 2019; Xu and He, 2022; Xu et al., 2023] and also deraining problem [Yang et al., 2017; Li et al., 2018; Zhang and Patel, 2018; Wang et al., 2019a; Wang et al., 2019b; Wei et al., 2019; Ren et al., 2019; Wang et al., 2020a; Yasarla et al., 2020; Wang et al., 2020c; Zamir et al., 2021; Zhou et al., 2021]. These deraining methods generally model the image deraining problem as a pixel-wise image regression process that directly learns the mapping from rain images to clear images by end-to-end trainable networks. Although these CNN-based approaches have achieved decent results, few of them consider patch recurrence into neural networks and hence are less effective in modeling the non-local self-similarity property of natural images. Although using large receptive fields can remedy this problem to some extent, it usually leads to larger or deeper networks that are difficult to train.

To better solve the image deraining task, we propose an effective method that explores the non-local similarity property of rainy images by a graph neural network. Figure 1 illustrates the main idea of our method that contains two branches: 1) internal data-based supervised branch is used to model the internal relations of similar patches from the rainy image itself and its multi-scale images and 2) external dataparticipated unsupervised branch is used to model the external relations of the similar patches in the rainy image and exemplar.

The proposed method is motivated by the patch recurrence property of natural images. We find that similar patches tend to recur many times in one image and its multi-scaled versions and external images, especially in rainy images as shown in Figure 2, where most image patches in rainy images have similar rain streaks. This property stimulates us to build the non-local graph modules that can aggregate these neighbors that have similar rain streaks into one query patch to achieve better deraining. To model the non-local similarity property of both the rainy image itself and external exemplars, we first model the internal relations of similar patches from the rainy image itself by a multi-scale graph network. Then, we explore the property of the exemplars and model the external relations of the similar patches in the rainy images and exemplars. Specifically, we construct a graph model by searching k-nearest neighboring patches in the multi-scale images and exemplar for each query patch of the input rainy image. We then obtain the corresponding k neighboring patches in the multi-scale images and exemplar and aggregate them with a graph attention mechanism. Finally, we formulate the constructed graph into an end-to-end trainable deep neural network to solve image deraining. The main contributions of this paper are summarized as follows:

• We propose a multi-scale graph module to explore internal non-local similarity by aggregating similar patches in multi-scale rainy images to the query patch of the input rainy image.
• We propose to use an exemplar image to explore external non-local similarity for enriching the representation of the query patch to furthermore improve the deraining quality.
• We analyze the proposed algorithm and show that it is able to remove rain streaks and preserve image details. Quantitative and qualitative experiments demonstrate that the proposed algorithm outperforms state-of-the-art methods on both synthetic and real-world datasets.

To the best of our knowledge, this is the first algorithm that explores the non-local property from both the rainy image itself and external exemplars by a graph neural network for single image deraining.

2 RelatedWork

2.1 Single Image Deraining

Priors-based deraining methods are typically built on some assumption about rain streaks and rainy images [Luo et al., 2015; Chen and Hsu, 2013; Li et al., 2016].

CNNs-based deraining approaches have dominated recent research and achieved great success. [Fu et al., 2017] maps the high-frequency structure of a rainy image to negative residual rain streaks and obtains final rain-free images by (1). After that, a series of deraining methods are proposed [Li et al., 2018; Wang et al., 2019b; Ren et al., 2019; Wang et al., 2020b; Wang et al., 2020a; Wang et al., 2022]

2.2 Non-local Self-similarity for Image Restoration

The non-local self-similarity prior assumes that similar patches frequently recur in a natural image, which has been exploited by many classical image restoration methods [Dabov et al., 2007; Buades et al., 2005; Chang et al., 2004]. Here, we briefly review recent approaches that exploit this prior with graph neural networks. Pl ¨otz . [Pl ¨otz and Roth, 2018] are the first to introduce a graph-based model for image denoising and image super-resolution. They propose a neural nearest neighbor network that learns to aggregate the neighbors of the query patch. Zhou et al. [Zhou et al., 2020] propose a cross-scale graph network for image super-resolution.

In this paper, we explore the non-local similarity property of rainy images by taking into consideration of multi-scale input images and external exemplars.

3 ProposedMethod

3.1 Overall Framework

The overall framework is shown in Figure 3. Our method intends to learn the rain streaks from an input image via a graphboosted CNN, and then subtract the learned rain streaks from the input image to obtain the deraining result. We express this process as follows:

where MSGNN denotes our proposed backbone CNN. ˜B refers to the estimated rain-free image. O12 and O14 refers to the 12 and 1 4 scaled images, respectively, while E represents the external exemplar. R(·, ·) denotes a graph model that is divided into two steps: the nearest neighbor search and attentional aggregation.

MSGNN takes a rainy image O as input and reasons internal multi-scale patch correlation with its multi-scale images and explores the external patch correlation with an exemplar in the network to search the most similar patch information and aggregate them for better rain streaks removal.

The MSGNN contains N sub-networks with an identical structure, and each sub-network consists of M Channel Transformation [Yang et al., 2020] Residual Blocks (CTResBolcks). Two adjacent sub-networks are connected via "fusion connection" (yellow blocks), which concatenates all the outputs of the convolution layers, then applies a 1×1 convolution on them, and sends the results to the next sub-network. The output of the graph model is injected into every direct connection between sub-networks via feature fusion, which concatenates all features together and then applies two convolution layers with kernel size 5 × 5 and stride size 2 × 2. The graph model is described later in the rest of this section, and the details of CTResBolck are provided in supplementary material.

To train the network, we minimize the negative structural similarity index measure (SSIM) [Wang et al., 2004] between the estimated rain-free image ˜B and the ground truth B:

3.2 GraphModel

Our graph model exploits both internal and external non-local similarity. Internal non-local similarity is utilized to identify similar patches to the query patch within the input image of different scales. The external non-local similarity is to find similar patches from another rainy image, the external exemplar.

As shown in Figure 4, the proposed graph model consists of the following steps: 1) extract features from images by a CNN; 2) for each query patch, search for patches with similar feature maps; 3) aggregate the similar patches with attention mechanism and then send the combined features to the backbone CNN to facilitate the deraining of the query patch.

Internal and External Similarity

As mentioned above, non-local self-similarity has been exploited in previous graph-based methods for various restoration tasks [Pl ¨otz and Roth, 2018; Zhou et al., 2020]. However, these methods overlook information in multi-scale images. In this paper, we explore the non-local self-similarity property in multi-scale inputs as in [Jiang et al., 2020]. Specifically, patch recurrence not only exists in the original image but also can be found across its different scaled versions due to the perspective nature of images - recurrent patterns may be located at different distances and thus have different visual sizes and are similar to each other only after properly scaled.

In addition to internal self-similarity, external images of the same domain, topic, or theme can also be similar to the input image and thus can provide more information for deraining. These external images are referred to as exemplars. Hence, in our model, we also explore the relation between the input image and an exemplar image for better deraining. Figure 2 provides a better illustration of the internal patch similarity in multi-scale images and external patch similarity in exemplars, which supports our motivation.

Nearest Neighbor Search

For a query patch of the input image, we want to find similar patches from each scaled input image and the exemplar image. Here, we take the exemplar image as an example to illustrate the graph construction process, which can be similarly applied to other images. The graph construction process includes the following steps: we first extract features from the original rainy image and the exemplar image via a three-layer CNN, and denote them by FO and FE respectively. Then, we divide FO and FE into patches of size l × l. For each query patch in FO, which is indexed by q and denoted by P qO , we find k nearest patches P nr E among patches in FE, based on the Euclidean distance between their CNN-extracted features. Nearest patches P nr E are indexed by nr and r = {1, · · · , k}.

As such, a k-nearest neighbor graph Gk(V, E) is constructed. V is the patch set (vertices in the graph) and equal to VO ∪ VE, the union of input rainy image patch set VO and an exemplar patch set VE. Set E is the edge set with size |E| = |VO| × k, which indicates k nearest neighbors for each query patch in VO. For each edge, one of its two terminal vertices is the input rainy image patch and the other one is the exemplar patch.

Attentional Aggregation

With the k-nearest neighbor graph, we can perform attentional aggregation to combine the nearest neighbors of the query patch. Notice that the query patch itself is its closest neighbor. Specifically, the aggregation is done by weighted averaging:

where αnr→q is the weight, Sq is the set of the k nearest neighbors of q, δq = ∑nr∈Sq αnr→q is the normalizing factor, and ̂ P qO is the aggregated patch representation.

The weight αnr→q is computed by an attention mechanism, which is devised to measure the similarity between the query patch and its neighbors in the sense that similar neighbors should have a large weight. It is implemented as follows. First, we calculate the difference between the query and the neighbor:

Then, we obtain the attention weight by a CNN with trainable parameters θ:

where the exponential function exp(·) ensures all weights are positive. Finally, for all query patches, we assemble the aggregated patch representations ̂ P qO into ̂ FO via patch2img [Pl ¨otz and Roth, 2018] and send it to downstream networks.

4 Experimental Results

To demonstrate the effectiveness of our proposed method for single image deraining, we evaluate it on both synthetic datasets and real-world datasets. For a fair comparison, we ensure that all the baselines are retrained using the codes provided by the authors.

4.1 Datasets

Synthetic datasets. We conduct deraining experiments on five widely used synthetic datasets: Rain200H [Yang et al., 2017], Rain200L [Yang et al., 2017], Rain1200 [Zhang and Patel, 2018], Rain1400 [Fu et al., 2017], Rain12 [Li et al., 2016]. Note that we use the model trained on Rain200H to test on the Rain12 dataset since Rain12 does not have training images. We use the Rain100H as the analysis dataset.

Real-world dataset. Note that [Yang et al., 2017; Li et al., 2019; Wang et al., 2020a] provide a mass of real-world rainy images. We use them as real-world datasets.

4.2 Implementing Details

We randomly select a rainy image as an exemplar from the dataset as input to the network. Although the exemplar is randomly chosen, it is able to improve the deraining performance, as shown in Figure 9. The number of channels is set as 32 and the nonlinear activation we used is LeakyReLU with α = 0.2 for all convolution layers except for the last one. For the last layer, the channel is 3 without any activation function. The patch size of input is 64 × 64, and the batch size is 8. We use ADAM [Kingma and Ba, 2015] to optimize our network parameters. The initial learning rate is 0.0005, and the rate will be divided by 10 at 300 and 400 epochs, and the model training terminates after 500 epochs. The number of sub-network N is 4 and the number of CTResBlocks M in each sub-network is 8. The patch size l is 3 and the stride length s is 3. The number of nearest neighbors k is set as 5.

4.3 Results on Synthetic Datasets

We compare our proposed network with eight state-of-the-art methods on five synthetic datasets, and the results are summarized in Table 1. It can be seen that our proposed method achieves the best results on all the datasets in terms of PSNR and SSIM. Moreover, we note that the number of parameters of our model is 62% less compared with DCSFN [Wang et al., 2020b], while the SSIM is increased by 1.62% on the Rain200H dataset. In Figure 5, we also visualize several deraining results on the synthetic datasets. It can be seen that our method is able to restore better details and textures and obtain cleaner and clearer background images, while other approaches hand down some rain streaks or lose some details.

4.4 Results on the Real-world Dataset

We further demonstrate the effectiveness of the proposed method by the comparison with state-of-the-art methods on the real-world dataset in Figure 6. It can be seen that the proposed algorithm is able to generate cleaner and clearer deraining results, while other methods hand down some rain streaks. This shows the effectiveness of the proposed method in deraining real-world images.

4.5 Ablation Study

Analysis on the Basic Components. In our model, we design the graph-based deep network using the fusion connection, channel attention (Channel Transformation), and graph model. We analyze them and present the results in Table ??. First, it can be seen that the results betweenM1 andM2 show that the fusion connection (FC) can improve the deraining performance. Moreover, as a comparison, we also analyze other channel attention modules, ECA [Wang et al., 2020d] and SE [Hu et al., 2018]. We can see that ECA is less effective for our model and SE can improve the deraining results in terms of PSNR and SSIM. Furthermore, CT is the best channel attention module among the three. At last, we can observe that our graph model can boost the deraining results. Hence, the fusion connection, CT, and graph modules applied in our framework are all meaningful for deraining.

Effect of the Stride Length. In Table 3, we present the results of different strides. Since the number of graph patches with stride 1 is increased sharply when the training patch size is 64 × 64, which is out of our computer memory, we adjust the training patch size to 48 × 48. We can observe that the model achieves the best results when the stride is 3 under the two conditions. The stride being 3 means that the obtained graph patches are not overlapped because the graph patch size is also 3. Moreover, we find an interesting phenomenon from Table 3 that the deraining results are influenced by the training patch size, and the larger the training patch size, the better the results.

Effect of the Patch Size. We analyze the effect of the size of the graph patch, i.e., l, and the results are reported in Table 4. We can see the deraining performance reaches its best when the patch size is 3 in both PSNR and SSIM compared to the cases of patch size being 5 and 7. Further, we can observe that the PSNR decreases as l increases. This demonstrates that a bigger patch size cannot help improve the deraining performance. As such, we set the default graph patch size as 3.

Effect of the Number of Nearest Neighbors. We analyze the effect of the number of nearest neighbors, i.e., k, and the results are presented in Table 5. It can be observed that the results are the best when k = 5 in terms of PSNR and SSIM.

N

Figure 7: Results on the number of sub-networks.

And, both k = 3 and k = 7 perform worse than k = 5. Hence, we select k = 5 as the default setting of the network. Analysis of the Number of Sub-networks. We analyze the effect of the number of sub-networks N, and the results are presented in Figure 7. It can be seen that N = 4 achieves good results and the deraining performance will not be significantly improved as N increases in terms of PSNR. Therefore, we set N = 4 as our default setting due to the trade-off between the deraining performance and the number of parameters. Note that the model also achieves state-of-the-art performance compared with other methods on the most challenging Rain200H [Yang et al., 2017] when N = 2, while the number of parameters only has 1.55M.

Analysis on the Use of Multi-Scale Images. Since our graph neural network is based on the multi-scale aggregation module, we provide an analysis on its effectiveness. We show the results on the real-world dataset in Figure 8. We can observe that the default model with all scales obtains clearer and cleaner results, while other models always hand down some rain streaks. Hence, the use of multi-scale input images in our framework is helpful for the deraining task.

Effect of the Exemplar. Here, we demonstrate the effectiveness of the exemplar on deraining, and the results are shown in Figure 9. We can observe that the model with an exemplar is able to improve the deraining performance, while the model without an exemplar hands down some rain streaks. It illustrates that our proposed model with exemplars can effectively utilize more information on rainy images, which makes the model more robust under different rainy conditions.

5 Conclusion

In this paper, we have proposed a deep multi-scale graph neural network with exemplars, called MSGNN, for image deraining. By exploiting internal non-local similarity in multiscale input images and external non-local similarity in an exemplar image with a graph model and attention mechanism, our model achieves favorable results on both synthetic and real-world datasets compared with state-of-the-art methods, demonstrating the effectiveness of our deraining model. In future work, we plan to explore different strategies for selecting the exemplar image for better deraining.

Acknowledgements

This work was supported by the National Natural Science Foundation of China No.62306343, Shenzhen Science and Technology Program (No.KQTD20221101093559018).