Rainy WCity: A Real Rainfall Dataset with Diverse Conditions for Semantic Driving Scene Understanding

1 Introduction

Out-door application, e.g., semantic segmentation under the driving situation, has achieved impressive progress with the existing segmentation models under clear weather conditions. However, the performance of these methods drops sharply when facing inclement weather, since adverse weather can significantly degrade the image quality and readability. One typical scenario is the rain condition, as shown in Fig. 1, with multiple degradation scenes and rain patterns, making the segmentation task more challenging.

∗These authors contributed equally to this work.

†Corresponding authors.

To promote this task, there are at least two urgent questions required to be addressed: how to construct a benchmark for comprehensively evaluating and training the segmentation model under rainy scenes? And how to make the segmentation model robust to diverse rainy scenes?

For the first question, some researchers recently put their efforts towards dataset generation and collection, including synthetic and real rainy datasets. Some studies aggregate the depth information [Hu et al., 2021] to generate the rainy samples or spray water on the glass to simulate driving on rainy days [Porav et al., 2020]. Although these schemes have boosted the development of segmentation tasks under driving scenes, there still exists significant distribution discrepancy in the real rainy scenes, and the synthetic data lose the randomness that may exist. In particular, the data discrepancy arises a problem that the model trained by the synthetic data losses the ability sharply in actual scenarios. For the existing real rainy datasets [Jin et al., 2021; Sakaridis et al., 2021], samples are closer to post-rain scenes since the characteristics of rainy days have not been included, e.g., raindrops, road reflection, and droplet occlusion.

As shown in Fig. 1, the representative segmentation algorithm DeepLabV3+ [Chen et al., 2018] (trained on the Cityscapes dataset) shows better performance on the clear Cityscapes or synthetic rainy Cityscapes scenarios. In contrast, the segmentation performance declines precipitously in the real-world rainy scenes, particularly in our dataset (from 76.2% to 23.2% of the mIoU), since there are more complex degradation factors.

To break these limitations aforementioned, we construct a novel rainy dataset with accurate semantic annotations, named Rainy WCity, to investigate the segmentation task for driving scenes comprehensively. Rainy WCity covers 5 common driving scenes on rainy days with a total of 24,335 real rainy samples, where each scene has 100 images with the corresponding well-annotation segmentation label.

Our dataset distinguishes others by featuring the following elements: 1) Diverse Rain Patterns: including light, moderate, and heavy rain scenarios; 2) Diverse degradation factors: besides the typical rain occlusion, our dataset considers numerous adverse effects of degradation, commonly occurred on rainy days but overlooked in other real rainy datasets, e.g., the droplet interference, road reflection, and windshield-blurring. In particular, we provide the elaborated classification of these scenes with the corresponding well-annotated segmentation labels.

For the second question, researchers have developed efforts to design the two-stage optimization model by simply cascading the image deraining and semantic segmentation tasks for the rainy segmentation [Jiang et al., 2020]. As a result, the segmentation results are barely satisfactory in real challenging scenarios, while the marginal improvement comes at the cost of extra computation and memory usage. The reason lies in that these two tasks have a significant discrepancy with regard to the optimization objective. The former tends to learn the pixel-wise fidelity, however, the latter focuses on the semantic-wise fidelity. Unlike these methods, we propose to achieve the joint training of the image deraining and semantic segmentation and construct an unsupervised rainy scene segmentation method. In particular, we introduce scene segmentation that meets real rain (S2R2). In addition, to eliminate the dependence on the real paired training samples, we train the deraining network via unsupervised contrastive learning. Meanwhile, the segmentation counterpart is optimized via cross-entropy loss. Experimental results on Rainy WCity demonstrate that our proposed S2R2 method achieves appealing improvements over the state-of-the-art methods in terms of deraining performance (image clarity) and segmentation accuracy.

Overall, contributions can be summarized threefold:

• We re-investigate the segmentation task under real rainy scenes and construct a real and comprehensive dataset Rainy WCity with well annotations. It provides a fair and unified benchmark for further research on this challenging task.
• We design an unsupervised joint optimization framework (deraining and semantic segmentation) to significantly promote segmentation performance in rainy scenarios, serving as a baseline method for further research.
• We conduct comprehensive experimental comparison under diverse rainy scenarios and provide thoughtful insights and analysis on this task, which is momentous for opening research and future direction.

2 RainyWCity Dataset

This section details the difference between our proposed dataset and the existing driving datasets, including Cityscapes [Cordts et al., 2016], Raincouver [Tung et al., 2017], KITTI [Alhaija et al., 2018], RID [Li et al., 2019], Apolloscape [Huang et al., 2020], nuImages [Caesar et al., 2020], BDD [Yu et al., 2020], ACDC [Sakaridis et al., 2021], RainCityscapes [Hu et al., 2021], and RaidaR [Jin et al., 2021]. Table 1 provides the comparison results in terms of the image resolution, sample number, scenarios, and rain patterns. Our dataset provides a further differentiation of some degradation factors under real rainy scenes.

2.1 Diverse Degradation Scenarios

Shading Effect of Droplet Droplet, the most common phenomenon on rainy days, will prevent light from passing straight through the object, and when it falls on the windshield, the content distortion caused by the reflectance and refraction becomes more pronounced. In Fig. 2(a), areas covered with rain droplet has significant misidentification, e.g., the green "vegetation" is identified as the grey "building", while the blue "sky" is misidentified as the grey "building".

Road Reflection The rain droplets accumulated on the road commonly form a mirror reflection effect, confusing the segmentation model with the fake and indistinguishable objective boundary. As shown in Fig. 2(b), the shadow of cars is incorrectly identified.

Blurring byWindshieldWiper The wiper eliminates the visual occlusion caused by the rain accumulation. However, for the real rainy days, the wiper operation could evidently promote the visual clarity of human observed views, but the unexpected change of scene contents is unfriendly to the segmentation model for the real-time application. As is shown in the yellow box in Fig. 2(c), the result is not consistent with the human's view.

2.2 Collection

Data collection starts at the end of March 2021 and has lasted over five months in total. We mount four cameras on four cars respectively on their roof behind the windshield. All four employed cameras are the same model: Xiaomi Recorder 2, which enables recording 30 frames per second with the resolution of 1,920×1,080. We have collected 5,619 videos in total and coarsely removed the rain-less and night parts, resulting in 1,685 videos of rainy scenes left. Most of the videos last around 3 minutes, and the rest last 90 seconds.

2.3 Selection by Two-stage Filtering

We adopt a two-stage filtering way to select target images that reflect the character of the rainy day scene as much as possible.

Stage i After finding out these raining videos, we first view each video and manually note down the approximate time intervals according to our understanding of rainy scenes. Then we extract full frames from the time intervals of clips we have recorded in the first step, containing 24,335 images in total.

Stage ii We apply BiSeNetV2 [Yu et al., 2021] segmentation models to quickly filter all of these frames from stage i to acquire corresponding segmentation results. The purpose of this step is to allow the model to provide its perspective on these data to know what kind of situation the current models can not handle. During this stage, besides the droplets impact, we have discovered extra degradation factors, which help us split the dataset into three parts in the next step. The details of these factors will be outlined later. Lastly, we make the final selection, picking 500 images from stage i by referring to the segmentation results from stage ii. Among those 500 images, 300 mainly focus on diverse rainfall driving scenes, 100 of which comprise the unique windshield wiper blurring effect, and the rest 100 contain the windshield reflection situation.

2.4 Annotation Procedure

We annotate our images based on the label tool of Cityscapes, so the class split also follows its manner. Annotating an image on a heavily rainy day is difficult because of too many complex factors (e.g., blurring and reflection). To make annotation more accurate and reliable, we propose using temporal information to enhance the quality: During the annotation, we will also refer to adjacent frames to decide the class of unclear pixels in current frame.

Following the construction of Cityscapes [Cordts et al., 2016] and ACDC [Sakaridis et al., 2021] datasets, we invite 20 annotators with professional image processing experience. In particular, annotated samples by one person are passed to another for a second confirmation to eliminate the bias and misinterpretation.

The time interval we choose in the selection part is usually at least 2 seconds long with 60 frames, which means that at least 60 adjacent continuous frames have accompanied every image we annotate and at least one video of reference. By combining spatial and temporal information, our annotation has richer information and reliable accuracy.

2.5 Rain Pattern Classification

To better study the scenes under specific rain intensity, we classified images. Specifically, we randomly selected 5,000 images and invited 20 relevant staff to rate each image with respect to rain intensity, according to the definition of Wikipedia, and finally, we assigned each image a specific intensity class based on the average score. This resulted in a total of three classifications: light rain, moderate rain, and heavy rain. We trained a classifier to classify the remaining images.

2.6 Comparison to Related Dataset

Finally, our dataset mainly contains 500 fine-grained pixellevel annotations, accompanied by corresponding 24,335 images of driving on rainy days, comprising diverse scenes and various degradation factors. The comparison of data diversity to other relevant driving datasets is shown in Table 1. The mixture of multiple elements has brought the most close-to-reality rainy scenes to Rainy WCity.

3 ProposedMethod

The architecture of our framework is brief and straightforward, as shown in Fig. 3. Simply rely on structure information may cause ambiguous mapping problems [Liao et al., 2021]. So we need to extract features in a more specific way. Motivated by [Ma et al., 2022], we design a feature disentanglement network as the basic unit of our method. Given images x1 and x2 from Cityscapes and our Rainy WCity domain, with the "shared content space" assumption [Huang et al., 2018], it can disentangle domain-invariant content features c1 and c2 of these images from the domain-specific counterparts z1 and z2. As has been validated by [Chang et al., 2019], the content features contribute most to the semantic segmentation task. Therefore, through feature disentanglement, we can transfer segmentation knowledge from x1 domain to x2 domain.

Specifically, we first need a shared content encoder Ec to extract c1 and c2 and two degradation encoders to extract degradation feature z1 and z2, respectively. Then, we use a shared image decoder Di to decode an image using the content features c1, c2, and degradation feature z1, z2. Depending on which c and z we use, we can perform cross-reconstruction to supervise the disentanglement learning. Besides, we use a segmentation decoder Ds to produce segmentation heatmaps ˆ y1 from the content feature c1, where label y1 is used as the supervision signal. After the cross-reconstruction, we achieve the basic function of deraining of our framework.

The traditional way of deraining is to train a model to decompose the rain image into the clean background and residual of the rain streak. Since pair clean label is difficult to obtain, we integrate the task of deraining into a segmentation framework, as described in the first step. This idea is based on the assumption that the segmentation and deraining share a similar goal: deraining is to restore clean image, which segmentation can benefit from it. Consequently, the segmentation result of the derained image should outperform the original raindegraded image. Specifically, we propose to utilize contrastive learning paradigm to distinguish the boundary between positive and negative pairs, where the comment latent content feature pair (c1, c2) as the positive pair and (c2, z2) as the negative pair. During training, we update the degradation encoders and segmentation encoders simultaneously. In this manner, the deraining module will study to restore rainy image progressively. Meanwhile, the segmentation network will also learn to discover the semantics in the restored images. The two modules complement each other in the process of training.

The goal of contrastive learning is to learn an encoder that encodes the private degradation pattern in our Rainy WCity dataset and helps disentangle the content, which contributes most to the semantics, from the rainy image. We assume that latent content feature c1 and c2 should be pulled as near as possible in the feature space, while content feature c2 and degradation feature z2 should be pushed as far as possible. Correspondingly, the definition of contrastive loss is as follows:

where · denotes the inner product, and τ > 0 is a temperature hyper-parameter. Note that all the features in the loss function are ℓ2-normalized.

This loss function can make the output of the deraining network closer to the ground truth and increase the distance between the restored image and the rain image in semantic space, which improves the classification result of scene pixels on the other hand.

In addition, we also introduce a pixel-level cross-entropy loss Lce to constrain the semantic segmentation task. This loss will check each pixel one by one and compare the prediction result ˆ y1 of each pixel category with our label y1. The definition of loss is as follows:

where y1 are ground truths annotations for images x1, ˆ y1 ∈ RH×W×C is the segmentation softmax output of x1, and H, W, and C represents height, width, and number of class categories, respectively.

The final overall loss function is:

where λ is the weighting parameter.

4 Experimental Results

To investigate the segmentation task under rainy days and evaluate our proposed S2R2 model, we conduct extensive comparison experiments with existing representative methods on our proposed Rainy WCity dataset. In detail, it consists of two deraining methods (DualGCN [Fu et al., 2021] and MPRNet [Zamir et al., 2021]) and five segmentation models (HRNet [Wang et al., 2021], BiSeNetV2 [Yu et al., 2021], DeepLabV3+ [Chen et al., 2018], SDCNet [Zhu et al., 2019], and PSPNet [Zhao et al., 2017]). For the segmentation part, all these comparison models are pre-trained on Cityscapes [Cordts et al., 2016] and run by their default settings.

4.1 EvaluationMetric

We use mean Intersection over Union (mIoU) to evaluate segmentation effects; The higher the value, the better. Since it is almost impossible to get clean pairs for real rainy images, we adopt two no-referenced image quality assessment methods for evaluation of the effectiveness of derain models: Spatial-Spectral Entropy-based Quality (SSEQ) and Natural Image Quality Evaluator (NIQE), the smaller the value, the better for these two assessments.

4.2 Comparison Results on RainyWCity

The rainy segmentation task involves two sub-tasks: image deraining and scene segmentation. To investigate the intrinsic contribution of these two tasks to the final segmentation performance, we construct several two-stage competitive models by directly cascading different deraining models (DualGCN, MPRNet, and our deraining model) and segmentation algorithms (HRNet, BiSeNetV2, DeepLabV3+, SDCNet, and PSPNet). In particular, we further design a one-stage method based on our deraining model and DeepLabV3+/BiSeNetV2 to promote the compatibility of these two sub-tasks via joint optimization. Quantitative evaluation on our proposed Rainy WCity dataset is tabulated in Table 2.

We use five segmentation models as baseline models, all of which fail to generate satisfied segmentation results. We guess the image deraining may destroy the intrinsic semantic information since these two sub-tasks have the obvious optimization discrepancy. We then test two deraining methods (DualGCN and MPRNet) and the deraining part (trained degradation encoder and image decoder) of our proposed method to see whether deraining the rainy images as a pre-processing is helpful for the segmentation. However, in contrast to the baseline models where the rainy images are directly input, the deraining processing has unstable effects, i.e., some have improved the performance while others lowered the performance, proving the pre-processing of deraining is not an effective way.

The segmentation part (trained content encoder and segmentation decoder) of our proposed unsupervised method brings about significant improvement, which shows the effectiveness of our method.

Visual comparison are provided in Fig. 4. As expected, our one-stage segmentation model shows significant superiority over other two-stage competitors, producing the predicted maps with clearer objective boundaries and cleaner contents. Especially for the wiper and reflection scenarios, only our proposed model can achieve the accurate segmentation of the road and cars, while other comparison methods gain the dirty segmentation results.

To further analyze the individual influence on each class, we tabulate the mIoU in Table 3, involving a total of 19 classes commonly. It is evident that our proposed one-stage solution gains the best scores in almost all classes. Especially for the autopilot related classes, e.g., road, building, car, and truck, our proposed method achieves significant improvements over other two-stage methods on the real rainy segmentation task. Taking the road, car, and bus as examples, the comparisons of the mIoU between ours and the second-best method are 73.8% (DualGCN) vs 91.8% (ours), 43.6% (DeepLabV3+) vs 81.8% (ours), and 0.5% (DualGCN) vs 17.0% (ours), respectively. These results reveal two key points: 1) simply cascading the deraining and segmentation models is far from producing the satisfying segmentation results on the real rainy days, our proposed Rainy WCity dataset in particular; 2) our proposed unsupervised one-stage scheme can significantly alleviate the compatibility, including the scenes (synthetic and real rainy conditions) and tasks (deraining and segmentation tasks), extensively promoting the final segmentation performance under real rainy days.

Table 4 provides the joint evaluation of image deraining and segmentation performance. Our proposed one-stage method achieves impressive performance on almost all metrics, gaining the best scores of the NIQE and mIoU in all scenarios. Although the two-stage method adopts DualGCN to promote the image quality and gains the smallest values of SSEQ, the final segmentation performance is undesired, even worse than that of the original input. We guess that the deraining operation may destroy the spatial semantic information, which is momentous for the segmentation task. By contrast, we adopt the joint optimization of the image deraining and segmentation, which greatly promotes the compatibility by achieving significant improvement in terms of the final mIoU (39.9% vs 23.2%). The metrics of deraining are negatively correlated with the metric of segmentation, which verifies our design.

5 Conclusion

This paper presents Rainy WCity, a semantic segmentation dataset for car-driving under the diverse real rainfall scenes. Apart from the common rainy situation, we have illustrated its complexity by proposing the fine-grained annotation of the driving circumstance where droplets, windshield reflection, and the windshield wiper exist, which are ignored in previous datasets but play a significant impact factor in this weather.

Besides an unsupervised joint optimization framework S2R2 is aimed to improve segmentation performance. Extensive experiments demonstrate the effectiveness of our method in real rainy scenarios.

Acknowledgments

This work was supported in part by the Department of Science and Technology, Hubei Provincial People's Government under Grant 2021CFB513 and 2021CFB281, and in part by the National Natural Science Foundation of China under Grant 62171325.