Hyperspectral Image Denoising Using Uncertainty-Aware Adjustor

1 Introduction

Hyperspectral image (HSI) captures plentiful spectral information of the same scene with contiguous wavebands and provides an internal perspective to analyze the observed targets, unlike gray-scale or RGB images. Thanks to the advantages of hyperspectral imaging [Cai et al., 2022b], a series of tasks based on HSI have been proposed, such as object recognition [Cai et al., 2021a], classification [Hong et al., 2021; Zhong et al., 2021], detection [Liu et al., 2021] and so on. However, due to photon effects, atmosphere, and other inevitable degradation factors, the practical HSIs are often suffered from various complex noises such as stripe, deadline, impulse, and more. These degradations largely hinder the analysis and interpretation of HSI. Therefore, HSI denoising is a critical pre-processing step before the downstream HSI applications.

∗Corresponding author

As the booming of the deep learning era, more researches [Wei et al., 2020; Rui et al., 2021; Cao et al., 2021] focus on extending the Deep Convolutional Neural Network (DCNN) to the HSI denoising task. Most of these methods attempt to extract and utilize rich spectral information to restore noisy HSI. Among them, the framework of most mainstream networks [Yuan et al., 2018; Yuan et al., 2021; Kan et al., 2022] is to recover one band at a time with the spectral complementary information from a fixed number K of adjacent bands and performs the HSI denoising task band by band. These spectral auxiliary networks consider spatialspectral correlation simultaneously and achieve impressing performance compared with directly applying the grayscale or RGB image restoration methods in the HSI denoising task.

However, achieving such improvements often requires complex network architectures and training strategies. Therefore, we aim to enhance denoising performance without modifying their original network structure. In existing methods, adjacent bands are typically treated equally, assuming that spectral information is uniformly distributed. However, it is widely known that the noise intensity and distribution vary across bands, as shown in Fig. 1, making it challenging to optimize denoising performance. Additionally, pixels in different spatial areas exhibit varying levels of degradation, indicating the varying importance of visual information. Therefore, for the current denoising band, the uncertainties of neighboring bands and pixels are not consistent. Based on these observations, we propose to enhance and adjust the adjacent spectral bands via uncertainty perception, leveraging their valuable information for network denoising.

In this paper, we take a novel perspective on HSI denoising and propose a general uncertainty-aware adjustor (UAAdjustor), which can be flexibly crafted on top of existing spectral auxiliary networks effortlessly to improve their performance significantly. An overview of the HSI denoising process with the proposed UA-Adjustor is illustrated in Fig. 2. It can be seen that UA-Adjustor generates adjusted adjacent bands preceding noise removal, which is decoupled from the spectral auxiliary network. The flowchart of UA-Adjustor is detailed in Fig. 3, including three designed modules, i.e., Adjacent Derivative Function (ADF), Spectral Significance Module (SSM) and Uncertainty Estimator (UE). ADF is designed to make the interaction between two bands by performing a novel bidirectional derivative, SSM is proposed to generate a list of significant weights of the adjacent bands, and UE is devised to estimate the uncertainty of spatial pixels adaptively. The end-to-end learning steps of UA-Adjustor can be described as follows. In the first stage, we evaluate the importance of each neighboring band. The more informative spectral bands tend to impose higher weights to benefit subsequent feature learning. In the second stage, we enhance each band by aggregating the spatial content of a short local range neighboring bands based on uncertainty awareness. In the last stage, we aim to adjust the weight of each spatial pixel in the enhanced adjacent bands by its uncertainty and produce the final spectral auxiliary bands. Unlike channel-wise attention modules like SE-Net [Hu et al., 2018] or S-MSA [Cai et al., 2022a], UA-Adjustor is a more comprehensive adjustor that enhances denoising performance by considering both the band-to-pixel and enhancement-to-adjustment aspects.

The main contributions are summarized as follows:

1. A general UA-Adjustor is proposed, which can be flexibly crafted on existing spectral auxiliary networks and improve denoising performance without any network structural modification.
2. We propose three progressive stages in UA-Adjustor, i.e., evaluating the significance of neighboring bands,
3. enhancing neighboring bands based on uncertainty perception, and adjusting the weight of spatial pixels in adjacent bands using estimated uncertainty.
4. Experimental results on several simulated and realworld benchmarks demonstrate that the proposed UAAdjustor can improve the denoising performance of recent SOTA networks with a huge boost which achieves a great trade-off between efficiency and performance.

2 RelatedWork

Different from grayscale or RGB image denoising methods [Cai et al., 2021b], hyperspectral image (HSI) denoising techniques face the challenge of inconsistent and unknown noise across abundant spectral bands. Consequently, most HSI denoising methods leverage the rich spectral information to restore noisy HSIs.

A prevalent category of HSI denoising approaches involves restoring each band in the noisy HSI using adjacent bands as auxiliary information, commonly referred to as spectral auxiliary networks. Among these methods, [Yuan et al., 2018] proposes a joint spatial-spectral learning strategy that captures multiscale spatial-spectral features to recover noisy HSIs. [Maffei et al., 2019] devises a downsampling kernel to achieve fast denoising without losing performance and uses the noise-level map to make the network more robust. [Yuan et al., 2021] attempts to estimate the noise intensity of each band and extracts multiscale spatial-spectral features to retain the texture details in HSI. [Kan et al., 2022] aims to decompose high-frequency features from the original features to better suppress noise. While these methods have shown promising results, they treat all adjacent bands equally, limiting the network's ability to attain optimal performance. To overcome this limitation, we propose a UA-Adjustor and elaborate on its workflow in the subsequent sections.

3 ProposedMethod

In this section, we first introduce the preliminary and overall pipeline of UA-Adjustor. Afterwards, we elaborate on each function and module in UA-Adjustor. Finally, we describe the workflow of the three progressive stages in detail.

3.1 Preliminary

The HSI can be treated as a 3-D image data of size H × W ×B, where H, W, and B denote the spatial height, spatial width, and number of spectral bands of the HSI. The goal of HSI denoising is to give clean results {xi}Bi=1 for all bands of noisy HSI {yi}Bi=1 . Suppose the current band to be denoised is yi and {yj}i+k/2 j=i-k/2 indicates the fixed number K of adjacent bands. The workflow of the previous spectral auxiliary networks can be represented as:

In UA-Adjustor, we aim to adjust the neighboring bands {yj }i+k/2 j=i-k/2 based on uncertainty perception. The workflow of UA-Adjustor can be denoted as:

where the y3 j is the j-th adjusted neighboring band after the third stage. In the following, the functions and modules of each stage will be explained.

3.2 Adjacent Derivative Function

As noted in previous studies [Zhang et al., 2019; Guan et al., 2022], it has been discovered that the spectral derivative can effectively attenuate irrelevant scene details while emphasizing noise components. In these works, the spectral derivative is defined as the forward difference along the spectral dimension between two adjacent bands, as given below:

However, we contend that the constraints on the direction of spectral difference and the specific pairing of spectral bands, as defined in Eq. 3, are not applicable for the extraction of long-range spectral information. Here, yi represents the current band, while yj represents an adjacent band that can be located either in front of or behind yi. In order to capture diverse spectral correspondences and ensure consistency, we introduce the concept of bidirectional derivatives, which extend the spectral derivative to incorporate both forward and backward differences between any pair of bands. This bidirectional derivative, illustrated in Fig. 4, effectively suppresses irrelevant and redundant spectral information, while highlighting the spectral disparities. To better leverage the spectral correlation, we propose the integration of the bidirectional derivative into an adjacent derivative function (ADF). The ADF explicitly binds the current denoising band with each of its neighbors as a spectrum pair and reasons the relations between every pair to exploit only valuable information for stronger denoising. Mathematically, the ADF is expressed as follows:

Ri→j = ADF(yi, yj) = concat(yi, yj, di→j, dj→i), (4) where i ∈ [0,B], j ∈ [i - k/2, i + k/2], and Ri→j is the interaction relation between yi and yj .

3.3 Spectral Significance Module

To efficiently evaluate the significance of different spectral bands, we introduce a lightweight spectral significance module (SSM) comprising a 3-D convolution operation, a 3-D adaptive average pooling operation, and a hyperbolic tangent activation function (Tanh). The SSM takes the interaction relation group {Ri→j}i+k/2 j=i-k/2 , generated by the ADF, as input and produces a significance list {Si→j}i+k/2 j=i-k/2 for the adjacent bands:

3.4 Uncertainty Estimator

To capture the uncertainty of each spatial pixel in the adjacent bands, we introduce a lightweight uncertainty estimator (UE) comprising two 3-D convolutions and a hyperbolic tangent activation function (Tanh). The UE takes the relation group {Ri→j}i+k/2 j=i-k/2 as input and dynamically learns the uncertainties {Ui→j}i+k/2 j=i-k/2 of different pixels in the adjacent bands during the training process. The workflow of the UE can be described as follows:

where γ represents the restriction factor, which is a learnable parameter. As illustrated in Fig. 5a, the current denoising band yi is heavily affected by high-intensity noise. To enhance the noise removal process, we leverage the relatively clean adjacent band yj (see Fig. 5b) as a spectral auxiliary band. Notably, we observe that the outline within the red rectangles in Fig. 5c is clearly visible in Fig. 5a. Consequently, these areas are assigned lower uncertainty values, while higher uncertainties are assigned to other pixels. In contrast, Fig. 5d shows the current denoising band yi contaminated by low-intensity noise. Here, the relatively clean adjacent band yj (see Fig. 5e) exhibits lower uncertainties in most areas (see Fig. 5f). As a result, pixels with higher certainty are given greater attention to extract valuable feature.

3.5 Spectral Significance Evaluation

Considering that the spectral information from different neighboring bands may contribute differently to the denoising process of the current band, we propose to evaluate the importance of each neighboring band. Consequently, we introduce the concept of modified adjacent bands {y1 j }i+k/2 j=i-k/2 , which can be calculated as follows:

where α represents the restriction factor, which is a learnable parameter. Si→j is the significance weight learned by SSM and y1 j denotes the modified yj of stage 1.

3.6 Spectral Enhancement

The second stage aims to enhance the adjacent spectral bands by leveraging the correlation and constraint between bands. However, enhancing the entire spectral range is computationally impractical due to the large number of bands in HSI. To address this, a novel spectral enhancement strategy based on uncertainty perception is proposed. This strategy involves the introduction of UE to estimate the uncertainty of each spatial pixel in a short local band range. Based on the estimated uncertainty, the adjacent bands are modified accordingly, generating uncertainty masks that represent spectral gain information. To normalize the uncertainty masks within a short local range, the softmax function is applied to these bands. This ensures that the enhancement is balanced and takes into account the relative importance of different pixels within the local range. Finally, the weighted uncertainty masks are aggregated to obtain the final enhancement information, which is then added to the corresponding band. This stage follows the workflow described bellow, allowing for efficient and effective enhancement of the adjacent spectral bands in the HSI:

where n is the number of short local range, yjn is the neighboring bands of yj, Uj→jn is the uncertainty of yjn learned by UE, Mj→jn is the uncertainty mask of yjn , ⊗ is elementwise multiplication and Aj is the aggregated enhancement information. Thus the enhanced spectral bands of this stage can be calculated as follows:

where β denotes the restriction factor and is a learnable parameter, the y2 j is the enhanced yj of stage 2.

3.7 Spectral Adjustment

Up till now, the enhanced adjacent spectral bands with more robustness are generated through the preceding two stages. In this stage, we aim to estimate the uncertainty of each spatial pixel in enhanced neighboring bands for current denoising and produce final spectral auxiliary bands. Thus the UE is also used in this stage. The workflow of this stage can be formulated as follows:

where Ui→j is the uncertainty of yj learned by UE and the Mi→j is the uncertainty mask of yj. Ultimately, the final adjacent spectral bands can be adjusted as follows:

where the y3 j is the adjusted yj of stage 3.

4 Experiments and Discussions

To illustrate the effectiveness of the proposed UA-Adjustor, in this section, we conduct comprehensive experiments on both simulated and real-world benchmarks. More results and future works are explained in the supplementary materials.

4.1 Experiment Setups

Dataset and Evaluation Setup

We integrate UA-Adjustor into several state-of-the-art systems, namely HSID-CNN [Yuan et al., 2018], P-DNet [Yuan et al., 2021], and AODN [Kan et al., 2022], to evaluate its effectiveness. The training process is performed on the CAVE dataset [Park et al., 2007], and we evaluate these models on the ICVL dataset [Arad and Ben-Shahar, 2016] and the remotely sensed dataset WDCM 1 for simulation experiments. For real-data experiments, we use the real-world datasets Urban [Mnih and Hinton, 2010] and Indian Pines [Landgrebe, 2003]. The details of dataset processing can be found in the supplementary materials. To assess the performance, we follow the commonly used protocols in HSI denoising methods and employ metrics such as peak signal-to-noise ratio (PSNR), structure similarity (SSIM) [Wang et al., 2004], and spectral angle mapper (SAM) [Yuhas et al., 1993]. For realworld datasets, we also use overall accuracy (OA) to evaluate the classification accuracy, which indirectly reflects the denoising performance. Additionally, we consider the parameters (Params) and runtime to evaluate the efficiency of UAAdjustor in the ablation study.

Simulated Noise Setup

To simulate the various noisy situations in corrupted realworld HSIs, we generate four types of complex noise following the settings in [Wei et al., 2023] (i.e., Case 1: Non-i.i.d. Gaussian + Stripe noise; Case 2: Non-i.i.d. Gaussian + Deadline noise; Case 3: Non-i.i.d. Gaussian + Impulse noise and

1 https://engineering.purdue.edu/∼biehl/MultiSpec/ hyperspectral.html

Case 4: Mixture noise) to contaminate clean HSIs.

Implementation Details

Following common protocols in existing spectral auxiliary networks, the number K of adjacent bands is set to 24. The number N of the short local range is set to 4. The working mechanism of UA-Adjustor is to be crafted on spectral auxiliary networks and these two parts are trained as a whole architecture. In practice, the complex noises randomly selected from Case 1 to Case 3 are added to the training set. For head-to-head comparisons, we fairly train baseline networks and their counterparts with UA-Adjustor integrated in the same experimental setup. All experiments are conducted on the NVIDIA GeForce RTX 3090 GPU.

4.2 Experimental Results and Analysis

In this subsection, we fairly compare the recent SOTA spectral auxiliary networks and their original networks with the UA-Adjustor integrated.

Simulation Experiments

Table 1 presents the performance comparison on the ICVL dataset across four types of complex noise cases. The integration of UA-Adjustor significantly improves the performance of each compared method. Notably, HSID-CNN and P-DNet achieve an absolute improvement of 1.95 dB in Case 4, which is the most challenging scenario and not learned in training process. AODN leverages the UA-Adjustor to enhance denoising performance by 0.31 to 1.12 dB. Furthermore, Fig. 6 provides visual comparisons under complex noise Case 4. It is evident that methods integrated with UA-Adjustor can better preserve the texture of the original images. To further assess the effectiveness of UA-Adjustor, we also evaluate the entire architecture on the WDCM remotely sensed dataset. The PSNR results and visual comparisons under complex noise Case 4 are shown in Fig. 7. Our method demonstrates satisfactory performance, with a significant improvement of 2.28 dB observed in P-DNet. HSID-CNN and AODN also achieve improvements of 1.84 dB and 1.24 dB, respectively.

Real-Data Experiments

We further evaluate the impact of UA-Adjustor on real-world noisy HSI datasets, namely Urban and Indian Pines. The assessments on the Urban dataset can be found in the supplementary materials. Given that image quality directly influences the performance of downstream HSI applications, we employ classification algorithms [Pedregosa et al., 2011] to objectively evaluate the denoising results of real-world HSIs.

The quantitative and qualitative evaluation results on the Indian Pines dataset are depicted in Fig. 8, the methods integrated with UA-Adjustor effectively remove noise and recover lost details without introducing noticeable oversmoothing artifacts. Consequently, they generate superior classification results compared to the original denoising network.

Ablation Study and Discussions

We conduct a series of ablation studies to examine the impact of different settings of the UA-Adjustor and reveal their contribution to our final performance. All ablations are validated on WDCM benchmark.

Analysis of progressive stages. We conducted experiments to thoroughly validate the functionality of each stage in UAAdjustor. The denoising results on the WDCM dataset are presented in Table 2. The original P-DNet and AODN serve as the backbones. From the table, it can be observed that the introduction of stage 1, which involves spectral significance evaluation, leads to a performance improvement of 1.60 dB for P-DNet and 0.68 dB for AODN. Subsequently, the addition of the modified adjacent bands from the first stage into the second enhancement stage results in further improvements of 1.74 dB and 0.55 dB for P-DNet and AODN, respectively, compared to their original versions. Finally, after the adjustment stage, P-DNet achieves an absolute gain of 2.28 dB, while AODN achieves a gain of 1.24 dB, compared to their respective backbones. Importantly, each stage introduces only a few parameters, making the lightweight UAAdjustor require a total of only 0.0089 M of additional parameters. As the stages progress, P-DNet and AODN integrated with UA-Adjustor demonstrate a significant improvement in denoising performance, while the increase in time costs remains within an acceptable range, especially for larger models.

Experiments on ADF. To evaluate the effectiveness of the proposed ADF, including the bidirectional derivative, we conducted ablation studies on different combinations of spectral bands and their derivatives in Eq. 4. The denoising results and efficiency evaluation on the WDCM dataset are presented in Table 3, using the HSID-CNN integrated with UA-Adjustor. As observed, one basic idea is to employ either di→j or dj→i as input, resulting in an improvement of 1.30 dB and 1.48 dB on HSID-CNN. However, this approach has limitations as it only uses spectral derivatives and loses image content. Conversely, combining (yi, yj) without spectral derivative information leads to a limited improvement of 1.47 dB. Therefore, we attempt to combine (yi, yj ) with either di→j or dj→i, resulting in improvements of 1.72 dB and 1.65 dB, respectively. Finally, the most comprehensive combination of yi, yj , di→j , and dj→i is adopted as our adjacent derivative function, achieving improvement 1.85 dB and optimal performance.

Analysis of short local range N. We conduct an ablation study to see how long the short local range N is optimal in the spectral enhancement stage. As shown in Table 4a, N = 2 costs the lowest runtime but achieves unsatisfactory denoising results. As the value of N increases, the denoising performance shows an initial improvement, but then starts to decline due to a larger range may introduce interference with high uncertainty. Ultimately, we observe that using a short local range of N = 4 offers the best trade-off between efficiency and performance.

Table 5: Ablations of different attention modules on P-DNet.

Analysis of adjacent bands K. We also conduct an ablation study to determine the optimal length of the adjacent bands, denoted as K. Table 4b presents the results of ablation. Similar to the analysis of short local range, when K is set to 12, we observe the lowest runtime, but the denoising results are unsatisfactory. Ultimately, the K is set to 24 provides the best trade-off between efficiency and performance.

Attention scheme comparison. We compare UA-Adjustor with other attention modules and present the results in Table 5. The baseline model achieves a denoising performance of 35.40 dB with 1.2791 M Params and 25.88 ms time cost. We apply the SE-Net, S-MSA, and our UA-Adjustor, respectively. By using these methods, the model's denoising performance improves by 0.52, 1.13, and 2.28 dB, while adding 0.0003, 0.0027, and 0.0089 M Params and 3.85, 2.13, and 7.11 ms time cost, respectively. UA-Adjustor achieves the most significant improvement in denoising performance with only three learnable 3D-conv layers, which maintains a great balance between efficiency and performance. However, it is anticipated that the denoising performance improvement may be limited when HSI is affected by low levels of simple noise, which can disrupt this balance.

5 Conclusion

In this work, we discuss the problems of existing HSI denoising methods and present UA-Adjustor, an approach that derives an effective strategy of adjacent spectral bands adjustment based on uncertainty perception so as to generate more suitable and powerful adjacent spectral bands on a perband basis. The UA-Adjustor is general and can be plugged into existing spectral auxiliary networks effortlessly, which contains three progressive stages to achieve improved performance, i.e., spectral significance evaluation, spectral enhancement, and spectral adjustment. Extensive results indicate that the UA-Adjustor can improve over recent state-ofthe-art methods on both simulated and real-world datasets.

Acknowledgments

This research is supported by National Key Research and Development Project of China grant number 2020AAA0105600, National Natural Science Foundation of China grant number 62006183, and Fundamental Research Funds for the Central Universities under grant Numbers xhj032021017-04. We also thank Yantao Ji and Jiehua Zhang for discussions.