DPMamba: Distillation Prompt Mamba for Multimodal Remote Sensing Image Classification with Missing Modalities

1 Introduction

Remote sensing image classification (RSIC) plays a vital role in earth observation, including applications such as urban planning [Quan et al., 2018], environmental monitoring [Nguyen and Liou, 2019] and more. With the advancement of image sensor technology, remote sensing images tend to be diversified. Numerous methods for multimodal RSIC [Yang et al., 2024; Roy et al., 2024] have emerged within the community, all of which have demonstrated promising results.

∗Corresponding Author

Note that these methods are based on an assumption that the input multimodal data is consistent in both the training and inference. However, in practical applications, due to limitations such as harsh shooting environment or low sampling frequency of certain sensors [Huang et al., 2024], the issue of missing modalities may arise during the inference. Existing multimodal classification methods are not designed and optimized for scenarios of missing modalities. The performance of these methods deteriorates significantly when certain modalities of the input data are missing during the inference.

Existing methods for addressing the problem of missing modalities can be categorized into three main approaches. The first approach utilizes generative models to synthesize missing modalities from the available modalities [Woo et al., 2023; Liu et al., 2023]. This approach requires initially training a generative model, followed by fine-tuning for scenarios with missing modalities. The classification performance largely depends on the quality of the synthesized modality. The second approach aims to learn a shared space that captures invariant information across the accessible modalities [Zhou et al., 2021; Zhang et al., 2022], which have demonstrated strong performance. Nevertheless, their effectiveness drastically decreases when only single modality data is available [Wang et al., 2023b]. To address this issue, the community has developed various knowledge distillation-based methods [Li et al., 2022; Wei et al., 2023a], which transfer the full-modality information from a teacher network (TN) (pre-trained with full-modality data) to a student network (SN) (input incomplete or even singlemodality data), as shown in Fig. 1. Due to the significant differences in inputs between the TN and the SN, existing knowledge distillation-based methods may encounter inefficiency and even overfitting issues during knowledge transfer. Furthermore, these methods often require training different models through knowledge distillation for various missing cases, which greatly increases computational costs. Recently, prompt learning has demonstrated potential in addressing the challenge of modality absence by designing taskaware prompts, which can significantly enhance the adaptability of multimodal tasks[Lee et al., 2023]. However, the optimization of prompts is typically confined to task-specific loss functions, failing to specifically optimize for missing modality information and thereby hindering the full potential of prompts in complex scenarios with missing modalities.

Based on the above observations, we propose a Distillation Prompt Mamba (DPMamba) framework for multimodal RSIC with missing modalities, as shown in Fig. 2. The core idea of DPMamba is to utilize a knowledge distillation technique based on shared text semantic space to guide the optimization of missing-modality aware prompts (MMAPs), effectively leveraging full-modality knowledge to supplement and enrich the missing modality information in MMAPs, thereby transforming them from "placeholder" to "adaptation" states. Specifically, to effectively enable the transition of MMAPs, we focus on two critical components. First, we introduce a novel modality-aware Mamba backbone for both the TN and the SN, which is designed to capture dynamic cross-modal interactions at both global and local levels, providing essential feature representations for optimizing MMAPs via backpropagation. Second, we propose a new teacher-student paradigm called text bridging distillation (TBD) to enable efficient transfer of full-modality knowledge. By projecting the features of TN and SN into a shared semantic space, TBD facilitates knowledge transfer through feature alignment and inter-class relational constraints, offering robust guidance for optimization of MMAPs. In summary, our main contributions are threefold:

• We propose a unified DPMamba framework for multimodal RSIC with missing modalities, leveraging TBD for optimizing MMAPs from "placeholder" to "adaptation" states by enriching missing modality information within MMAPs with full-modality knowledge.
• We design a modality-aware Mamba as the backbone for extracting dynamic and multi-level feature, providing critical representations that facilitate the backpropagation-based optimization of MMAPs.
• We design a text bridging distillation (TBD) to facilitate the efficient transfer of full-modality knowledge, providing robust guidance for the optimization of MMAPs.

2 RelatedWork

2.1 Multimodal RSICwithMissingModalities

Multimodal RSIC enables high-precision classification of complex scenes by integrating complementary information from multiple modalities [Qu et al., 2024; Wang et al., 2024]. However, most existing methods are not optimized for missing modality scenarios. Recent approaches address this by using autoencoders to impute missing modalities [Hafner and Ban, 2023; Chen et al., 2024b] or transferring knowledge from teacher models trained on complete modalities to student models with partial data [Wei et al., 2023c; Liu et al., 2024a]. However, these methods encounter notable limitations when addressing complex scenarios, such as ensuring high-quality generation by generative models on largescale datasets and the dependency of knowledge distillationbased approaches on training separate models for each missing modality case. This reliance significantly complicates the overall model design and training process.

2.2 Knowledge Distillation

Knowledge distillation (KD) is a deep learning technique for compressing large models into smaller, more efficient ones. In incomplete multimodal learning, KD enables the transfer of full-modality knowledge from a TN to a SN. KD methods fall into three categories: response-based, feature-based, and relationship-based. Response-based distillation [Wang et al., 2023c] uses the soft target output of TN as the training target for SN but lacks internal knowledge transfer. Feature-based distillation [Li et al., 2022] minimizes discrepancies between TN and SN hidden features, though mismatched features in incomplete multimodal data can cause overfitting [Garcia et al., 2019]. Relationship-based distillation [Xin et al., 2024] addresses this by leveraging inter-layer or inter-sample relationships for more effective knowledge transfer. Inspired by CLIP [Radford et al., 2021] and DIST [Huang et al., 2022], this paper introduces a new TBD that maps the representations of TN and SN into a shared text semantic space and utilizes feature constraints and inter-class relationship constraints to collaboratively guide the efficient transfer of fullmodality information.

2.3 State Space Models

State Space Models (SSMs) [Gu et al., 2021] are effective in capturing global context in deep learning due to their scalability. Gu et al.[Gu and Dao, 2023] introduced Mamba, an SSM-based model that even outperforms the Transformer [Vaswani, 2017] across multiple modalities. Recently, VMamba [Liu et al., 2024b] extends the Mamba architecture to 2D images, achieving strong performance in computer vision. The Mamba architecture is now being applied to remote sensing tasks like change detection [Chen et al., 2024a] and image classification [Yao et al., 2024]. However, most of these works only utilize single-modality remote sensing image data, and are not optimized for scenarios of missing modalities. The potential of the Mamba architecture in the multimodal RSIC with missing modalities remains to be explored.

3 Method

3.1 ProblemDefinition

In this paper, we focus on the task of multimodal RSIC with missing modalities that generates a classification map categorizing each pixel into one of the C classes. For simplicity and without loss of generality, we consider a multimodal dataset D = {Xm1 i ,Xm2 i , yi}Nc i=1 , where Xm1 i and Xm2 i denote the i-th pair of multimodal patches from m1 and m2, respectively, yi represents the label, Nc is the number of patches. There is a nonlinear mapping M : {Xm1 i ,Xm2 i } → yi from image patches to labels. Since certain modalities may be absent during inference, the mapping could be rephrased as M1 : { ˜Xm1 i ,Xm2 i } → yi or M2 : {Xm1 i , ˜Xm2 i } → yi, where ˜Xm1 i and ˜Xm2 i denote dummy inputs that keep the input structure consistent. Our goal is to optimize a unified and robust M that performs well regardless of whether the input data is complete or has missing certain modalities by fully exploiting the privileged modality information available only during training. The subscript i will be omitted in the following for brevity.

3.2 Overall Design

To achieve the above goal, we propose a unified framework DPMamba (Fig. 2), which leverages a knowledge distillation algorithm based on a shared semantic space to guide the optimization of missing-modality aware prompts (MMAPs) from "placeholder" to "adaptation" states, thereby enabling robust multimodal RSIC with missing modalities. To effectively facilitate the transition of MMAPs, we focus on two key aspects: first, we propose a novel modality-aware Mamba as the backbone of TN and SN to capture cross-modal dynamic interaction information at the global and local levels, which is important for backpropagation optimization of MMAPs. Second, we propose a novel teacher-student paradigm, termed text bridging distillation (TBD), for the efficient transfer of full-modality knowledge. TBD projects the features of the TN and the SN into a shared text semantic space and achieves knowledge transfer through feature alignment and inter-class relational constraints.

3.3 Modality-Aware Mamba

Multimodal remote sensing images typically contain rich information from different perspectives, and effectively extracting their complementary features remains a core research challenge. Inspired by the Mamba structure, which achieves global modeling with linear complexity, we propose a modality-aware Mamba that enables dynamic and multi-level visual feature extraction, based on the DWConv, a novel attention fusion (AF) and modality information scanning mechanism (MISM), as shown in Fig. 3. Different modality data are processed in separate branches, with injected fusion information enhancing cross-modal collaboration. The AF enables dynamic fusion, and the MISM collaborates with DWConv to capture multi-level features. Subsequently, we establish semantic associations through textvisual features alignment to achieve classification.

Attention Fusion

Due to the dynamic changes in the redundancy and complementarity of information between modalities in the case of missing data, a fixed fusion strategy is insufficient to adapt to these variations. To address this, we introduce a simple attention fusion (AF) mechanism that enables dynamic integration of information, thereby enhancing the network's robustness to modality missing scenarios. Formally, let {x1, x2} represent the collection of input feature tensors. The attention mechanism computes attention scores for each input feature tensor. Specifically, the attention score scorei, i ∈ {1, 2} for each input tensor xi is derived through a linear transformation, followed by a scalar output:

where W and b are the learnable parameters of the attention layer. Then, to transform the scores into attention weights, the attention scores are normalized using the softmax function, thereby ensuring that the weights are non-negative and sum to one:

where αi is the attention weight for xi, and subtracting the maximum score ensures numerical stability during the softmax operation. The final fused output ha is obtained by computing the weighted sum of the input tensors, with the weights determined by the attention scores:

Modality Information Scanning Mechanism

SSMs process input sequences in a causal manner, but simply flattening the non-causal image patch into sequential input fails to fully capture modal information. To this end, we introduce a modality information scanning mechanism (MISM), which models the context by interacting each pixel with its neighboring pixels through SSMs. The combination of MISM and DWConv enables efficient multi-level feature extraction while maintaining relatively low computational complexity. Specifically, as shown in Fig. 3, we follow four zigzag paths to scan the patch pixel by pixel, i.e., top left to bottom right (width-priority scan and height-priority scan) , and their respective reversals. Given the output ha of AF, we flatten the p × p feature patches into a sequence according to the scan path, and then calculate each element of the sequence Hseq = {[hi,1, hi,2, · · · , hi,p×p] |i ∈ {1, 2, 3, 4}} by iteratively using the equation:

where ¯A, ¯B, and C are trainable parameters of MISM. After rearranging to a consistent order, the outputs of each path are fused for comprehensive contextual information.

Text-Visual Features Alignment

For remote sensing image classification tasks, annotation information typically includes class labels and corresponding class names. Class text information has also been demonstrated as an effective form of supervisory signal for training, e.g., CLIP[Radford et al., 2021]. To be exact, class text descriptions provide additional semantic constraints, enabling the model to better capture inter-class relationships and improve discriminative capability in complex scenarios. Technically speaking, we utilize a large language model to generate text based on the class intrinsic attributes and inter-class relationships. The text is subsequently tokenized and encoded into text features using a frozen text encoder, with its parameters initialized fromCLIP's text encoder. Then, the projected visual features are aligned with the text features in the semantic space, thereby establishing deep semantic associations and enhancing classification robustness.

3.4 Text Bridging Distillation

To take full advantage of the privileged modal information available only during training, we design a text bridging distillation (TBD). Specifically, we employ contrast constraints to align the visual features of both the TN and SN with shared text features, mapping the visual features to a shared text semantic space. Subsequently, feature constraints and inter-class relationship constraints are utilized to collectively optimize the SN and MMAPs. The shared semantic space provides a unified context, ensuring semantic consistency between the TN and SN, thereby reducing information loss during the distillation process. Unlike directly imitating the prediction output of TN, TBD allows the SN to acquire a more generalized full-modality representations from TN. TBD guides the optimization of prompts, while the prompts alleviate distillation overfitting caused by significant input disparities due to modality absence, with both elements supporting and enhancing each other. Given the visual output features of the TN and SN as Vt ∈ RC×D1 and Vs ∈ RC×D1 , and the text features as T ∈ RC×D1 , where C and D1 denotes the number of class and channel, respectively. In more detail, we employ the feature constraint loss LFC to distill fine-grained full-modality information as follows:

Furthermore, to capture the inherent inter-class relationship from the TN and enhance the classification performance of the SN, we initially derive the similarity matrices St = cos(Vt, T) and Ss = cos(Vs, T), where cos (·, ·) represents the cosine similarity between two features. Following [Huang et al., 2022], we then utilize the following inter-class relationship constraint loss LICRC to facilitate the transfer of interclass information:

where dP(·, ·) is a function for calculating Pearson's distance between column vectors or row vectors of St and Ss .

In conclusion, the text bridging distillation loss LTBD can be expressed as follows:

3.5 Prompt Injection

To handle classification tasks involving missing modalities, we integrate trainable MMAPs into modality-aware Mamba seamlessly, enabling the learning of modality-specific information and mitigating the issue of distillation overfitting caused by input inconsistencies by dynamically optimizing. Specifically, we define two modality-specific prompts, pm1 and pm2 , which are collaboratively optimized by taskspecified criterion and distillation constraints. Given the complete input modalities {x1, x2}, if modality m1 is missing during inference, the prompt pm1 is introduced to replace the missing modality, resulting in the input {pm1 , x2}. This modified input is then fed into the backbone network, where pm1 serves as a surrogate to guide feature extraction for the missing modality. By dynamically updating modality-specific prompts, the model can effectively leverage prior knowledge to mitigate the performance degradation caused by missing modalities.

3.6 Training Objective

The total loss L during the training stage is defined as follows:

where LTBD has been described above, and LCA is the contrastive alignment loss. In both the pre-training stage of the TN and the training of the SN, we utilize the following LCA as a task-specific criterion to establish the association between visual and text features for classification and map the visual features into a shared semantic space for distillation:

which is composed of a text-to-vison contrastive loss defined as:

and a vison-to-text contrastive loss defined as:

4 Experiments

4.1 Datasets

We conducted experiments on three public multimodal remote sensing datasets: 1) Houston dataset [Debes et al., 2014], obtained in 2012, includes hyperspectral images (HSI) and LiDAR-derived digital surface model (DSM) data of the University of Houston campus and surrounding urban areas. It contains 15 object categories and 15,029 labeled samples. 2) Trento dataset [Rasti et al., 2017], captured in a rural area near Trento, Italy, includes HSI and LiDAR-derived DSM data with 6 object categories and 30,214 samples. 3) Augsburg dataset [Baumgartner et al., 2012], captured over Augsburg, Germany, contains HSI, LiDAR-derived DSM data and a dual-polarized SAR image with 4 feature channels. This dataset includes 7 object categories and 78,294 samples.

4.2 Implementation Details

All experiments are conducted on Ubuntu 18.04 with an NVIDIA GeForce RTX 3090 GPU, using the PyTorch framework for model building, training, and evaluation. The training process of DPMamba consists of pre-training and training stages, with the AdamW optimizer used in both stages. The batch sizes for the Houston, Trento, and Augsburg datasets are 150, 120, and 140, respectively, and the optimal patch sizes are 11, 15, and 9. Pre-trained text encoder parameters from CLIP are loaded and fixed. During pre-training, the initial learning rate is set to 3e-4 for 500 epochs. In the training stage, the model parameters from pre-training are used for both TN and SN, with the learning rate maintained at 3e-4 for training the SN over 500 epochs. Code is available at https://github.com/Jiahuiqu/DPMamba.

4.3 Performance Comparison

In this section, we conduct a comprehensive comparison with concurrent baselines. We set up three different experimental configurations: 1) Training and testing are conducted within a single modality in the proposed method. "HSI-Net" refers to the model trained and tested using only the HSI modality, with other modalities following the same naming convention. 2) Advanced methods for joint classification in scenarios with modality absence, including ShaSpec [Wang et al., 2023a], DGDNet [Wei et al., 2023b] and MSH-Net [Wei et al., 2023d]. 3) Advanced methods for joint classification with complete modalities, including HCT [Zhao et al., 2023], Sal2RN [Li et al., 2023] and Cross-HL [Roy et al., 2024]. All methods are trained using 40 samples per class, with all samples used for testing. As for evaluation metrics, we report overall accuracy (OA), average accuracy (AA), and the kappa coefficient (κ), with κ values scaled by a factor of 100.

Performance Analysis of HSI-LiDAR Classification

We conducted experiments on the Houston and Trento HSILiDAR datasets. As shown in Table 1, the proposed method achieves an OA of 98.27% on the Houston dataset with the complete modality, compared to 97.68% from baseline methods. In the W/o HSI scenario on the Trento dataset, the OA is 94.40%, and in the W/o LiDAR scenario, the OA is 98.35%, demonstrating robustness in scenarios with missing modalities. Due to its limited capacity, the unified DPMamba may perform slightly worse in certain scenarios compared to scenario-specific baseline methods. Additionally, visualizations in Fig. 4 and Fig. 5 show that the method accurately captures complex boundaries and heterogeneous regions, particularly in areas with high spectral variability, such as urban and vegetation zones. This highlights the effective use of the modality-aware Mamba to explore the consistency of spectral features, spatial structures and elevation information. The enriched text contextual semantics improve the recognition of subtle data distributions, thereby enhancing classification accuracy and reliability.

Performance Analysis of HSI-SAR Classification

Comparative experiments on the Augsburg HSI-SAR dataset are summarized in Table 1. The heterogeneity between HSI and SAR data, stemming from their distinct imaging mechanisms, challenges information fusion. The proposed method achieves an OA of 92.29% with complete modality, exceeding Sal2RN by 1.11%, and demonstrates notable improvements over MSH-Net in the W/o HSI (+13.06%) and W/o SAR (+0.31%) scenarios. This is attributed to the proposed backbone's effective fusion of heterogeneous information and highlights DPMamba's ability to efficiently transfer knowledge from the complete modality and accurately optimize SN and MMAPs. Visualization in Fig. 6 further confirms the robustness of DPMamba, even in complex urban distributions.

Performance Analysis of HSI-LiDAR-SAR Classification

This section presents experiments on the Augsburg HSILiDAR-SAR dataset to assess the scalability of DPMamba in scenarios with multiple missing modalities. The results are shown in Table 2, where the baseline models are trained on the complete modality and evaluated under missing modality conditions, without optimizing for scenarios of missing modalities. We observe that DPMamba consistently outperforms the baseline models, with the OA showing a maximum increase of 34.27% (HSI & SAR). These results demonstrate that when multiple modalities are missing, the optimized MMAPs effectively alleviate the performance degradation caused by modality absence.

4.4 Ablation Study

Effectiveness of TBD

To assess the effectiveness of TBD, we conduct an ablation experiment by replacing LTBD with KL Divergence in Variant-1. TBD enables full-modality information transfer through feature alignment and inter-class relation constraints based on a shared text semantic space. As shown in Table 3, in the W/o HSI scenario, DPMamba outperforms Variant-1 by 3.70% and 8.43% in OA on the Houston and Augsburg HSISAR datasets, respectively, demonstrating the advantages of TBD over KL Divergence replacement.

Effectiveness of Modality-Aware Mamba

To evaluate the modality-aware Mamba backbone and its components, we design three variants: Variant-2a (modalityaware Mamba replaced with ViT [Dosovitskiy, 2020]), Variant-2b (AF replaced with Add Fusion), and Variant-2c (MISM replaced with DWConv). Table 3 shows that DPMamba outperforms all variants, achieving average OA improvements of 6.60%, 4.73%, and 1.25% on the Houston dataset, and 4.96%, 2.33%, and 2.05% on the Augsburg HSISAR dataset across all scenarios of missing modalities. The performance drops in the Variants highlight the importance of dynamic, multi-level feature extraction, enabled by the AF and MISM, for robust multimodal fusion and handling missing modalities.

Effectiveness of Prompt Injection

The MMAPs are seamlessly integrated into the modalityaware Mamba, enabling the acquisition of missing modality information and mitigating performance degradation caused by modality absence. To further validate the effectiveness of prompt injection, we design Variant-3, which omits MMAPs and focuses solely on optimizing the SN for different scenarios of missing modalities. As shown in Table 3, DPMamba achieves average OA improvements of 4.89% and 3.67% over Variant-3 across all scenarios of missing modalities on the Houston and Augsburg HSI-SAR datasets, respectively.

5 Conclusion

We propose DPMamba, a unified framework for RSIC with missing modalities. By leveraging knowledge distillation in a shared text semantic space, DPMamba optimizes MMAPs with full-modality knowledge for enriching missing modality information, transforming them from "placeholder" to "adaptation" states. To enable the transition of MMAPs, we focus on two key elements. First, we introduce a novel modalityaware Mamba backbone for both TN and SN, designed to capture dynamic cross-modal interactions at global and local levels. Second, we propose a new TBD to efficiently transfer full-modality knowledge for optimizing MMAPs. Experimental results show that DPMamba outperforms baseline methods on the Houston, Trento, and Augsburg datasets, underscoring its effectiveness in multimodal fusion and handling missing modalities. Ablation studies further validate the contributions of TBD, modality-aware Mamba and prompt injection.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62201423 and 62471359, the Key Research and Development Program of Shaanxi under Grant 2025SF-YBXM-513, the Young Talent Fund of Association for Science and Technology in Shaanxi under Grant 20230117 and 20250133, the Young Talent Fund of Xi'an Association for Science and Technology under Grant 959202313052, in part by the Fundamental Research Funds for the Central Universities under Grant QTZX25084.