Align, Perturb and Decouple: Toward Better Leverage of Difference Information for RSI Change Detection

1 Introduction

Change detection is a vision task aiming at identifying pixelwise semantic changes between paired bitemporal images, this technique is helpful for analysis of remote sense imagery (RSI) with high resolution, which provides important information about changes in land surface and expansion of settlement during long time of observation [Hussain et al., 2013; Daudt et al., 2018b].

With the development of remote satellite sensoring, there exist more open source databases of remote sense imagery with fine-grained semantic annotations [Chen and Shi, 2020a; Ji et al., 2018; Zhang et al., 2020a], which makes it possible to exploit the data-hungry deep-learning approaches [Krizhevsky et al., 2017; He et al., 2016] to achieve more accurate change detection. Due to the nature of pair-wise input and dense prediction, encoder-decoder architectures with siamese backbones prevail in recent efforts, where features of single images are extracted separately and the difference information is exploited in a decoder to highlight areas of changing objects [Daudt et al., 2018a; Bandara and Patel, 2022; Chen et al., 2022b; Zheng et al., 2021; Chen et al., 2022a].

* Corresponding author

In a nutshell, recent approaches usually model the difference information between input pairs in simple and direct manners (e.g. taking difference or concatenation to obtain difference information) [Daudt et al., 2018a; Bandara and Patel, 2022; Zheng et al., 2021] or implicitly embed the difference into feature interaction [Chen et al., 2022b; Chen et al., 2022a], while still leaving some key issues open. Firstly, difference information is inherently vulnerable to pseudo-changes (e.g. the seasonal illumination changes during long-term observation), but few of previous works explicitly take such interference into account, hence these methods are not guaranteed to be robust enough to non-semantic changes. Secondly, most of previous literature take difference information from features for final decoding, ignoring the fact that difference naturally contains information of changeful objects, which reversely provides spatial guidance to representation learning [Hussain et al., 2013]. Finally, the relationship between image content and difference information is seldom discussed in prior works, the image content can be regarded as auxiliary prior information for change decoding, while also introducing some irrelevant change cues distracting prediction.

With the reviews above, we claim that the difference information in current research is still underutilized. Therefore, in this paper, we design series of operations aiming at mitigating the aforementioned issues and fully leveraging feature difference to boots change detection results. Concretely, we equip the hierarchical encoder-decoder network with three operations in sequential order: Alignment, Perturbation and Decoupling (APD), which is illustrated in Figure 1. Alignment: To alleviate noise from pseudo-changes, we first propose a graph-based alignment module, which exploits the contextual similarity between patches to aggregate information from areas of the same semantic as compensation, this results in more precise extraction of semantic difference in following stages. Perturbation: Following the alignment operation, we propose a perturbation-aided difference prediction module. Especially, this module is trained with discrete semantic perturbation as feature augmentation, thus can recognize more generalized change patterns as guidance for feature extraction in the following stages. Decoupling: We decouple the decoder into an asymmetric dual-stream structure for final prediction, one focuses on integrating bitemporal contents with difference while the other takes pure difference information for decoding, this helps utilize the complementary property between image content and difference information while avoiding irrelevant noise.

We conduct experiments on three challenging benchmarks for RSI change detection: LEVIR-CD [Chen and Shi, 2020a], WHU-CD [Ji et al., 2018] and DSIFN-CD [Zhang et al., 2020a], and demonstrate the superiority of our proposed approach over existing competitive methods. Plenty of ablation studies also verify the effectiveness of our proposed operations. In a nutshell, the contribution of this paper can be summarized as following

• We reconsider the problem of pseudo-changes and propose a graph-based alignment module to explicitly utilize contextual similarity for compensation.
• We propose a hierarchical difference extraction structure to guide the feature extraction process stage-by-stage, which is equipped with a specially designed discrete semantic perturbation scheme for feature augmentation.
• Different from most of prior works, we propose a dual decoder structure to decouple the utilization of image content from pure difference encoders.
• We integrate the proposed operation above and convert traditional encoder-decoder structure into a new change detector APD, which achieves competitive results on mainstream benchmarks.

2 RelatedWorks

2.1 Change Detection with Deep Learning

We roughly divide deep learning-based change detection methods into two types [Zhang and Shi, 2020]: two-stage and one-stage methods. In general, the two-stage method trains a CNN/FCN to classify the bitemporal images respectively, and compares their classification results to obtain change areas. To achieve this goal, both the bitemporal semantic labels and the change label should be provided [Nemoto et al., 2017; Liu et al., 2019a; Ji et al., 2019].

The one-stage method is a more prevailed framework in recent research, which takes change detection as a dense classification task and directly produces the change result from the bitemporal images. The FC series [Daudt et al., 2018a] was one of the earliest methods to adopt convolution neural networks for change detection, where three architectures were proposed: FC-EF, FC-Siam-Conc, and FC-Siam-Diff. FCEF adopted the early fusion strategy, while FC-Siam-Conc and FC-Siam-Diff adopted the medium fusion strategy with different difference policies. Besides, DTCDSCN [Liu et al., 2019b] takes inspiration from the two-stage methods, and converts change detection to multi-task pipeline with semantic map as auxiliary supervision. ChangeSTAR [Zheng et al., 2021] further relaxes the paired requirement by taking unpaired images as input.

2.2 AttentionMechanism in Change Detection

Although deep neural network achieves significant improvement over handcrafted features, traditional network structures can not explicitly capture the contextual reliance within RSI, thus can not accurately detect changes in object-level. Therefore, there appears works resorting to attention mechanism [Jaderberg et al., 2015; Hu et al., 2017; Vaswani et al., 2017] to help gather more contextual information. Zhang et al.[Zhang et al., 2020b] proposed a deeply supervised change detection network (IFN) , which applies channel attention and spatial attention for feature enhancement.

Besides the contextual modeling, recent works also make attempt to integrate the attention mechanism into bitemporal interaction as an implicit replacement of feature difference operation. Both BIT [Chen et al., 2022b] and TransUNetCD [Li et al., 2022] proposed a bitemporal image transformer by mixing CNN and transformer, where the cross image interaction is conducted in latent encoded space. ChangeFormer [Bandara and Patel, 2022] further developed a siamese variant of SegFormer [Xie et al., 2021] for feature extraction.

2.3 Feature Augmentation

Feature augmentation aims at perturbing data in the encoded feature space to help the neural network learn invariant representation. Different from data augmentation with low-level transformation, feature-level augmentation is regarded as increasing the semantic variance to ensure the generalization ability of models. ISDA [Wang et al., 2019] perturbed the deep features in terms of their statistical characteristic, thus improving the accuracy of recognition. The work of SFA [Li et al., 2021] utilized a similar adaptive noise for domain generalization. Further, Manifold Mixup [Verma et al., 2019] extended the widely-used mixup trick into feature space to smooth the semantic boundaries. MGD [Yang et al., 2022] introduced random mask for knowledge distillation among deep features to obtain better results of student models.

In change detection, the difference information is usually derived from the discrepancy between two feature maps, which should be invariant to the difference order and channelwise masking. Therefore, we take inspiration from feature augmentation methods and design a specialized discrete semantic perturbation mechanism to help difference prediction module recognize more general patterns.

3 Approach

Figure 2 depicts the pipeline of our APD change detector. Similar to previous work, our method is built based on an encoder-decoder architecture. We denote the bitemporal image pair as X0 and X1 ∈ R3×H×W with change label map Y ∈ {0, 1}H×W , then feed images to a classic hierarchical deep neural network to extract feature representations of multi-level {F l0 ,F l1 }l, where l ∈ [2,N] represents the stage index (at most N stages). However, to fully leverage the difference information, we propose the concept of "Align First and Then Difference", and insert a Graph Interactive and Difference Enhancement module (GIDE) between stages in the encoder part, which transforms F l i into more changesensitive manifold ˜ F l i

where Encl (·, ·) denotes the l-th stage of the encoder. Specifically, GIDE is composed of an alignment module A(·, ·), which aggregates pixel-level semantic information, and a perturbation-aided difference module P (·, ·) which augments feature difference and highlights local areas. Additional to the alignment and perturbation, P (·, ·) also generates feature difference Ol as pure difference information for the following decoding. In the part of the decoder, we decouple the classical single decoder into a complementary structure of contentaware decoder Daw and content-agnostic decoder Dag to excavate the content and difference information respectively for final prediction.

3.1 Context-aware Alignment

The alignment operation aims at alleviating the disturbance of pseudo-changes and amplifying the area of semantic change, thus we resort to an undirected bipartite graph as an aggregation structure to align features (as shown in Figure 2 (b)).

Formally, suppose both F l0 , F l1 are of resolution Hl ×Wl, then we can take each pixel in feature map as a graph node and build an adjacent matrix G, which is a bipartite graph, i.e. an edge can only link pixels from different feature maps, thus the adjacent matrix can be represented as four-dimensional, G ∈ {0, 1}Hl×Wl×Hl×Wl , and the node feature is the feature of the corresponding pixel. Considering the feature of a single pixel can be sensitive to non-semantic changes, while its contextual information can be more robust since it contains more structural information, therefore it is more suitable to encode the edge between nodes with the reference of contextual similarity. To do this, we first obtain the context feature F l0,c , F l1,c via a non-parameterized dilated convolution of dilate factor d and kernel size k, for m = 0, 1, we have

where [u, v] indicates obtaining the pixel-wise feature at location (u, v). To build graph G, for each pixel (u, v) on F l0,c , we compute top n nearest coordinates on F l1,c

with the grouped coordinate set of nearest neighbor Hu,v, the high-dimensional adjacent matrix can be expressed as

With the relation graph G based on context, we can take the naive graph convolution to aggregate information from similar node pairs to both feature maps as follows

where ˆ F l0 , ˆ F l1 are aligned features after aggregation of graph convolution and will be utilized in following difference prediction module.

3.2 Perturbation-aided Difference Prediction

After the feature alignment, we make attempts to leverage the aligned feature to produce coarse spatial guidance, thus helping feature extraction in the following stage. To this end, we introduce a feature augmentation mechanism to train a coarse difference prediction module (as described in Figure 2 (c)).

We start with a simple difference operation, however, we explicitly modulate the channel of feature difference with a random vector v filled with discrete values

where C l is the channel dimension of ˆ F l0 , ˆ F l1 , and ⊙ represents channel-wise modulation, i.e. the i-th element of vector v is broadcasted and multiplied to all pixels on the i-th channel. The value of vector v is constrained within 0 and ±1, thus each channel of feature difference is either reversed (-1), masked (0) or retained (+1). The insight behind such perturbation is that even though part of semantic information is masked out or reversed, the preserved semantic should be informative enough to coarsely highlight the change. Besides, to prevent too many channels from being masked, we define a mask ratio τ ∈ [0, 1] to control the ratio of 0 values in v. It should be noted that channel-wise modulation is only adopted during training and removed from inference.

The modulated difference ˆOl is fed into an ASPP [Chen et al., 2018]-like structure to obtain a one-dimensional coarse maskMl ∈ RHl×Wl

where GAP(·) denotes global average pooling and Cat(·, ·) indicates concatenation between two features along channel dimension, note that the pooled feature is broadcasted to all pixels to align the resolution, MLP(·) represents multilayer perceptron mapping the input to single channel mask, and a sigmoid function σ(·) is adopted to obtain coarselevel change area in l-th stage. In the end, GIDE feeds back such difference-dominant information to the original siamese-aligned feature

where ⊗ indicates spatial-wise modulation, i.e. Ml is broadcast to different channels.

Besides, we inject additional supervised objectives in the perturbation module as deep supervision to ensure the accuracy of estimated guidance mask. Concretely, we downsample the original change map Y to adapt the size of maskMl as Y l and apply the binary cross-entropy as loss

The deep supervision together with our designed random perturbation helps train more generalized modules to differentiate various change patterns.

3.3 Decouple Decoders

In the decoder part, to exploit the complementary property of image content and pure difference, we devise an asymmetric dual-decoder structure, which takes F N 0 ,F N 1 and Ol as input and predicts the change areas.

The content-aware decoder takes the encoded image features as input, and hierarchically appends difference information to generate the intermediate feature, finally a segmentation head is applied on the final output to obtain the prediction

where Decl,aw(·, ·) is the l-th decoding block of decoder, which consists of concatenation and two cascaded Conv-BNReLU blocks with kernel size 3 × 3, S↑(·) represents the upsampling operation, and Taw(·) is the segmentation head. As for the content-agnostic decoder, we only feed the pure difference information as input, the other structure is similar to the content-aware one

where the structure of Decl,ag(·, ·) is similar to that of content-aware decoder but the concatenation is replaced by summation of two input features. Finally, both output are summed up with an activation function to obtain final results

3.4 Loss Function

During training, we simply take the cross-entropy Lce to supervise the final output ˆ Y and the total loss function can be expressed as Equation (13) with balance factor λ1 and λ2

In Equation (13), we introduce an additional comparative loss Lcomp for feature regularization, which is expressed as

where [·]+ represents clipped by 0 if the value inside is negative and γ is a hyperparameter. This comparative term helps backbone to distinguish feature of changed objects.

4 Experiment

4.1 Experiment Setup

Dataset. We evaluate our proposed APD change detector on three publicly change detection datasets: LEVIRCD [Chen and Shi, 2020a], WHU-CD [Ji et al., 2018] and DSIFN-CD [Zhang et al., 2020a]. LEVIR-CD is a public large building change detection dataset that contains 637 bitemporal RS image pairs of resolution 1024×1024. We utilize the default train/val/test split. WHU-CD is another public building change detection dataset, which consists of one pair of ultra-resolution (0.075m) aerial images of size 32507×15354. We follow the default cropping policy of size 512×512 and dataset split(train/test) provided by the authors. DSIFN-CD contains the changes in six major cities' landcover objects in China. We divide the 512×512 images into 256×256 pixel patches without overlapping, and we follow the default standard train/val/test split. Consequently, there are 14400/1360/192 samples for training/val/test.

Implementation Details. We implemented our model under the Pytorch framework, using a single NVIDIA GeForce GTX 1080 Ti GPU for training and the batch size is set to 8. During training, we apply data augmentation through random flip, crop and photometric distortion. We use AdamW with weight decay equal to 0.05 for optimization. The initial learning rate is 0.001 and we train models for 60k, 40k and 100k iterations for LEVIR-CD, WHU-CD and DSIFN-CD datasets. For context aggregation, we set k = 1 and d = 16. The hyperparameters in loss term are λ1 = 1.0, λ2 = 1.0, γ = 1.0.

EvaluationMetrics. For horizontal comparison with other methods, we follow the common setting and use the F1 score and Intersection over Union (IoU) with regard to the change objects as the primary evaluation indices. Meanwhile, we also report precision(P) and recall(R) of the change objects.

4.2 Main Result

Methods for Comparison. To verify the effectiveness of our method, we make comparison with several advanced change detection approaches, including three purely convolutional-based methods (FC-EF [Daudt et al., 2018a], FC-Siam-Conc [Daudt et al., 2018a], FC-Siam-Diff [Daudt et al., 2018a]), three attention-aided methods (DTCDSCN [Liu et al., 2019b], STANet [Chen and Shi, 2020b], SNUNet [Fang et al., 2021]) and three methods with transformer-like structure (BIT [Chen et al., 2022b], ChangeFormer [Bandara and Patel, 2022], TransUNetCD [Li et al., 2022]).

Quantitative Results. Table 1 reports the overall quantitative comparison results on LEVIR-CD, DSIFN-CD and WHU-CD. In the datasets of DSIFN-CD and WHU-CD, our proposed APD change detector outperforms other methods, reaching the best level in terms of all four metrics. In the LEVIR-CD dataset, although our method does not achieve the best precision, APD can still detect more changing pixels, which makes significant advantages in the other three metrics. For example, the F1 score of our method exceeds the latest ChangeFormer by 1.31%, 1.2%, and 9.57% on the three datasets, respectively. In Table 1, we also indicate the information of the utilized backbone of different CD methods. It can be observed that APD only applies a simple and lightweight ResNet-18 network as a feature extractor and does not use complex structures such as UNet[Daudt et al., 2018a; Fang et al., 2021] or transformer-based network[Bandara and Patel, 2022], which are widely used in segmentation tasks. On the other hand, although our approach only takes ResNet18 as the backbone, it can still outperform competitors with larger model capacity (ResNet50) or more advanced structure (MiT-B1), which indirectly manifests the superiority of our proposed GIDE module and decoupled decoders.

Qualitative Results. In addition, Figure 3 also shows the visualization comparison of the different change detection methods on the three datasets. As highlighted in red, blue and yellow respectively, our proposed method captures more detailed change compared with other change detection schemes. In the visualization results of LEVIR-CD dataset, our APD detector can not only detect the building change more accurately, but also avoid some noise (e.g. changes in the appearance of land cover, seasonal illumination changes, etc.) that affects the contour of the changed area. From the visualization results of the WHU-CD dataset, it can be seen that most compared methods cannot eliminate the pseudochange caused by shadows when detecting changes. In contrast, our method can eliminate such pseudo-change, which proves that our method can learn effective context, eliminate the irrelevant change and express the real semantic variation. At the same time, our method can also effectively detect subtle changes, which are observed from the visualization results of the DSIFN-CD dataset. The compared methods can hardly detect the detailed changed area of the long and narrow road due to the lack of precise semantic difference information. However, our APD detector can effectively capture the subtle variation and generate a more precise change map, which verifies the superiority of APD detector.

4.3 Ablation Study

Verification Experiment on Proposed Operations. We design ablation experiments for each proposed operation, i.e. Alignment, Perturbation and Decouple. For each operation, we provide a specific baseline counterpart to demonstrate the effectiveness. The overall results are shown in Table 2. The experiment shows that the Alignment module, Perturbationaided difference module and Decoupled Decoder are helpful for change detection performance.

To evaluate the alignment operation, we remove the context-aided alignment in GIDE on purpose and directly fed the siamese features into difference prediction. In terms of Table 2, the context-aided alignment significantly improves the recall and IoU score. This is because the Alignment module effectively utilizes the contextual similarity to gather information from areas of the same semantic to enhance the semantic difference information. This improvement is also reflected in Figure 4, if the alignment part in the GIDE module is removed, the noise caused by the lack of the contextual information makes the model unable to distinguish between real changes and pseudo-changes, thus affecting the precise recognition of the changed areas.

To verify the effectiveness of perturbation, we also devise a counterpart by eliminating the deep supervision and semantic perturbation, instead we directly take the feature from both images and their difference as output. Table 2 demonstrates that our proposed perturbation-aided deep supervision enssentially improve the recall and F1-score, since combination between deep supervision and random disturbance can help difference predictor focus on more general patterns of changeful objects, consequently the estimated change area provides in-depth guidance for following feature extraction. In additional, we also visualize the heatmap of intermediate feature to show the impact from perturbation. Figure 5 shows the feature visualization before and after introducing the perturbation in the second stage of the backbone. It can be observed that with the perturbation-aided module, our model significantly reduces the impact of the pseudo-change (pond, road, etc.) and strengthens focus on the changeful objects.

Finally, we evaluate the impact from decoupled decoder, specifically, to build the baseline, we introduce a single decoder structure which takes the concatenation of image features and difference information as input. From the Table 2, we find decoupled structure brings substantial performance gain on detection precision and IoU score. This demonstrates that decoupled decoders make full use of the complementary between image content and difference information for better change detection. Figure 6 shows the feature visualization of two shunts in our decoupled decoders. We observe that the output of the Content-agnostic Decoder can retains the profile of real semantic change. Further, the dual-decoder structure can help detect some small and detailed changes that could not be detected by pure difference information.

Verification Experiment on Hyperparameters. Next we also conduct parameter verification experiments to test sensitivity to hyperparameters in the disturbance module on the LEVIR-CD dataset. The results are shown in Table 3.

First we evaluate the effect of the ratio of the masked channels, conceretly, we set the ratio τ to 50%, 25%, 12.5% and 6.25% of the input feature dimension. The experimental results show that there exist negative impact on detection performance when the perturbation ratio is too high or too low, this is because large perturbation ratio can result in too many masked channels and loss in difference information, thus the difference prediction underfit to underlying change patterns and affect feature extraction in following stages. On the other hand, too few perturbed channels will decrease the variance in difference information, thus the difference prediction overfit to specific change patterns. Therefore, we choose 25% as the most appropriate ratio.

Secondly, we evaluate the effect of different forms of deep supervision Lld . From Table 3, when replacing cross-entropy with dice loss, the results are degraded in different ratios of perturbation, because small objects account for a certain proportion in the change detection dataset, to which the dice loss is very sensitive. Therefore, once some pixels of small objects are incorrectly predicted, it will lead to large fluctuation in the backward gradient. Instead, the cross-entropy takes each pixel equally, which alleviates this problem.

5 Conclusion

In this paper, we review the issue of underutilization of difference information in previous methods of RSI change detection, and propose a new change detector termed as APD. The APD change detector features with three elaborately designed operations, i.e. Alignment, Perturbation and Decoupling, to fully leverage the difference information and bootstrap change detection results. With a lightweight backbone, our APD detector can effectively improve the performance on challenging benchmarks of change detection and achieve state-of-the-art results in most metrics, and extensive ablation studies also verify the effectiveness of each operation.

Acknowledgements

This work was supported in part by Natural Science Foundation of China under contract 62171139, and in part by Zhongshan science and technology development project under contract 2020AG016.