Enhancing Multimodal Model Robustness Under Missing Modalities via Memory-Driven Prompt Learning

1 Introduction

The field of multimodal learning has made remarkable progress, primarily driven by the capability of transformer to effectively integrate diverse modalities [Akbari et al., 2021; Nagrani et al., 2021]. These models exploit the complementarity and associations among modalities to achieve excellent performance in tasks such as image captioning, visual question answering, and cross-modal retrieval [Shao et al., 2023; Stefanini et al., 2022; Wang et al., 2016]. Despite these advancements, a critical challenge remains: the performance of multimodal models degrades substantially when one or more modalities are missing. Existing multimodal models typically assume that all modalities are available during both training and inference [Kim et al., 2021]. In real-world applications, incomplete data is common due to sensor failures or privacy concerns. This reliance on complete data limits their robustness and impedes their deployment in practical scenarios.

∗Corresponding author.

Previous researches on handling missing modalities primarily focus on multi-stream multimodal architectures, which often reconstruct missing modality features based on available modalities [Ma et al., 2021; Wang et al., 2023; Woo et al., 2023]. These paradigms require retraining the model on incomplete data to enable the model to cope with missing modalities scenarios. Recently, multimodal models based on unify transformers have demonstrated powerful performance. [Ma et al., 2022] first explored robustness against missing modalities in unified multimodal transformer, and introduce multi-task training approach to improve model performance. As the model scale continues to grow, some researcher introduced the parameter efficient finetuning to copy with missing modalities cases for avoiding expensive computation cost. [Lee et al., 2023] designed specific prompts tailored to different missing-modality cases, while [Shi et al., 2024] introduce modality-aware adapter to disease diagnosis for handling missing scenarios.

However, there are still several limitations. First, they fail to leveraging existing samples and available modalities to compensate for the missing modality information. Unlike multi-stream architectures, unified transformer model concatenate all modality features into a unified representation, which makes it challenging to disentangle individual modalities. This constrains the ability to reconstruct missing modality representations using existing ones. Second, current approaches focus on transferring pretrained models to missing modality scenarios, resulting in failing to handle unseen missing scenarios during training. These methods often rely on customized prompts tailored to specific missing scenarios during training. As a result, they rely heavily on pre-defined missing scenarios, making them inflexible when faced with unseen combinations of missing modalities during inference. For instance, the training data is only accompanied with missing text modality, the model performance deteriorates significantly when the visual modality is missing during inference. Therefore, in incomplete data scenarios, it is crucial to fully utilize the available information to compensate for missing information, thereby enhancing the multimodal model's robustness and its ability to handle diverse missing scenarios.

To address the above limitations, we propose MemoryDriven Prompt Learning (MemPrompt), an approach that adaptively compensates for missing modality information to enhance model generalization and robustness. Due to the difficulty of disentangling individual modality representations in multimodal transformer architectures, we design a promptbased strategy that leverages two complementary information sources: (1) semantically similar samples from the current modality, and (2) shared information from the available modalities. To achieve this, we introduce two types of prompts: generative prompts and shared prompts. Generative prompts are dynamically retrieved from a predefined prompt memory, which stores semantic information of all samples for each modality, enabling the retrieval of semantically similar samples to compensate for missing information. Additionally, shared prompts establish cross-modal associations, allowing existing modalities to provide relevant information for missing modalities. Extensive experiments demonstrate that our method achieves state-of-the-art performance, regardless of whether the missing modalities during training and inference are consistent or not. This is attributed to the effectiveness of our compensation mechanism, which dynamically leverages both intra-modal and cross-modal information.

The main contributions can be summarized as follows:

• We propose Memory-Driven Prompt Learning for multimodal models to handle diverse missing modality scenarios. This approach employs a prompt-based strategy to compensate for missing information.
• The missing information is compensated through two complementary prompts: generative prompts, which retrieve semantically similar features from a predefined prompt memory, and shared prompts, which utilize cross-modal associations to supplement the missing information.
• Extensive experiments on three benchmark datasets demonstrate the superior performance of the proposed method, validating its effectiveness and adaptability, even under inconsistent missing modality scenarios.

2 RelatedWork

2.1 Multimodal Learning withMissingModalities

Multimodal learning aims to fuse diverse modalities to leverage richer information, thereby enhancing model performance [Kim et al., 2021; Akbari et al., 2021; Radford et al., 2021; Bao et al., 2022]. Conventional approaches typically assume that all modalities are available during both training and inference. However, in real-world applications, certain modalities may be unavailable, resulting in model performance significantly deteriorates[Ma et al., 2021; Ma et al., 2022; Wang et al., 2023; Shi et al., 2024]. Consequently, many researchers have focused on improving the robustness of multimodal models in scenarios with missing modalities [Zhao et al., 2024b; Reza et al., 2024; Yang et al., 2024; Wu et al., 2024b]. Existing methods attempt to simulate the missing modality data during training so that the model can generalize to the missing modalities scenarios. They primarily focus on reconstructing missing modality features from available ones. For instance, [Woo et al., 2023] proposed ActionMAE, which reconstructs features of missing modalities using the remaining modalities. Similarly, [Wang et al., 2023] introduced Shared-Specific Feature Modeling, which improves robustness by learning both shared and modalityspecific features. However, the above approaches require to retrain the model, which undoubtedly increases the computational overhead. Some methods generalize the model to different missing scenarios through parameter-efficient finetuning. [Reza et al., 2024] proposed parameter-efficient adaptation techniques to compensate for missing modalities. [Lee et al., 2023] introduced missing-aware prompts to generalize pretrained models to handle missing modality scenarios, while MoRA [Shi et al., 2024] employs distinct modalityaware up-projections within LoRA [Hu et al., 2021] to enable modality-specific adaptations. However, these approach fail to employ existing information to compensate for missing modalities, thus resulting in performance drop under inconsistent missing cased during training and inference.

2.2 Prompt Learning

As the scale of transformer-based models continues to grow, the cost of fully fine-tuning these pretrained models is becoming increasingly prohibitive. Prompt learning has emerged as an efficient paradigm that leverages task-specific prompts to guide pretrained models, enabling them to adapt to downstream tasks with minimal fine-tuning while fully utilizing the knowledge embedded within the pretrained models [Li and Liang, 2021; Jia et al., 2022; Petrov et al., 2024]. This approach has been widely adopted in visual tasks [He et al., 2023a; Han et al., 2023] and multi-modal learning [Roy and Etemad, 2024; Park et al., 2024; Xin et al., 2024b; Khattak et al., 2023a; Zhao et al., 2024a]. Visual prompt learning [Jia et al., 2022; He et al., 2023b; Han et al., 2023; Wang et al., 2024] introduces a small number of trainable parameters at the input layer while keeping the pretrained model frozen. In the multi-task learning domain, works such as [Xin et al., 2024a; Xin et al., 2024b] have employed prompt learning to align text and visual modalities during the fine-tuning process. In multi-modal learning, studies like [Zhou et al., 2022; Khattak et al., 2023a] utilize prompt learning to enhance few-shot learning performance. Furthermore, researchers have addressed the issue of over-fitting in prompt learning [Khattak et al., 2023b; Roy and Etemad, 2024] and explored the integration of parameter-efficient fine-tuning (PEFT) with expert systems to enhance the model's representation capabilities [Li et al., 2024; Wu et al., 2024a]. In this work, we employ a prompt learning strategy to address missing modality scenarios by compensating for the absent information.

3 Method

3.1 ProblemDefinition

For simplicity and without loss of generality, we consider a multimodal dataset consisting of two modalities, denoted as M = 2, with modalities m1 and m2. The dataset is defined as D = {Dc , Dm1 , Dm2 }, where Dc = {xm1 i , xm2 i , yi} represents the subset with complete pairs, and Dm1 = {xm1 j , yj} and Dm2 = {xm2 k , yk} denote incomplete modalities (i.e., only m1 or m2 is available). To align with the input requirements of the multimodal model, we use dummy inputs to simulate missing modalities (e.g., empty characters/pixels for text/image). Specifically, the subsets Dm1 and Dm2 are transformed into ˜Dm1 = {xm1 j , ˜xm2 j , yj} and ˜Dm2 = {˜xm1 k , xm2 k , yk} , respectively, to ensure consistent input structure for the model. ˜xm1 and ˜xm2 denote dummy inputs for the missing modalities. The resulting training dataset, comprising both complete data and transformed incomplete data, is defined as ˜D = {Dc , ˜Dm1 , ˜Dm2 }.

3.2 The Overall Framework

Given the pretrained multimodal model F , which comprises a patch embedding layer and stacked transformer blocks, each block consists of a multi-head self-attention (MHSA) and an MLP layer. For multimodal input, each modality is encoded separately and combined with position embeddings, modality-type embeddings, and class embeddings to produce f 0m1 ∈ RLm1 ×D and f 0m2 ∈ RLm2 ×D , where Lm represents the embedding length and D denotes the embedding dimension. The encoded representations are then concatenated to form the model input: h0 = [f 0m1 , f 0m2 ]. Each transformer block is defined as Eq.(1):

where f i θ represents the parameters of the i-th block, and l denotes the total number of transformer blocks.

To address missing modality scenarios, we propose Memory-Driven Prompt Learning (MemPrompt), which compensates for missing modality information by querying semantically similar samples within the current modality and leveraging cross-modal associations. The whole framework is shown in Fig.1. Specifically, these two compensation strategies are implemented using generative prompts and shared prompts, respectively. The construction process of these prompts is detailed in Sec. 3.3. Here, we define the generative prompts as Pg,m and the shared prompts as Ps,m, where m = {m1,m2}. Unlike previous approaches that directly concatenate prompts with input representations, our method constructs separate prompts for each modality and prepends them to their respective modality features. After adding the prompts, the input to each transformer block is represented as:

where f im1 and f im2 denote the features of each modality in the i-th transformer block. The newly constructed hi is then processed by Eq.(1) to produce the final outputs.

During training, the pretrained backbone remains frozen, and only the added prompts and task-specific layers are finetuned. Let θ represent the additional prompts-related parameters and θt represent the task-specific parameters of the classification layer. The final loss function is defined as:

where xim1 and xim2 denote inputs that may contain missing modalities, and Ltask represents the task-specific training loss, typically the cross-entropy loss.

3.3 Prompts Design

In unified transformer models, disentangling individual modalities from the unified representation is challenging. To address this, we employ a prompt-based strategy to compensate for missing information. Specifically, the compensation is derived from two sources: (1) semantically similar samples within the current modality and (2) complementary information from the available modalities. To achieve these goals, we decompose the vanilla prompts into generative prompts and shared prompts. The generative prompts are adaptively constructed from a predefined prompt memory based on the current input. This prompt memory stores prior information about specific modalities, enabling it to provide approximate feature representations to compensate for missing modalities. In contrast, shared prompts primarily establish cross-modal relationships. When one modality is missing, the shared prompts allow the available modality to provide complementary information. Unlike previous approaches, our method does not rely on predefined missingmodality scenarios, enabling it to effectively handle a wider range of missing-modality conditions. Details of the generative prompts and shared prompts are discussed below.

Generative Prompts

Unlike custom-designed prompts tailored to specific missingmodality cases, generative prompts are adaptively constructed based on the current input, providing a flexible mechanism to enhance the robustness of multimodal models. Specifically, let the input features at the i-th transformer block be hi = [f im1 , f im2 ] , where f im1 and f im2 represent the features of each modality. For each modality, the generative prompts are constructed as:

where PromptMem is the prompt memory that contains a set of prompts and their associated weights, storing prior feature information for each modality to effectively handle missingmodality scenarios. During training, the feature information of each modality is stored in the memory. When a sample encounters a missing modality, similar features are retrieved from the memory, and generative prompts are adaptively built to compensate for the missing information.

The prompt memory is defined as PromptMem = {(pk, qk), k ∈ [1,K]}, where pk ∈ RL×D represents the prompt embeddings, qk ∈ RD denotes the associated coefficients, K is the memory size, L is the prompt length, and D is the feature dimension. To evaluate the similarity between the current input and the prompts in the memory, we first compute the cosine similarity between fm and qk, followed by softmax normalization:

where wk represents the similarity between the current input and each prompt in the memory. The N most similar prompts are then selected, and a weighted summation is performed to generate the final generative prompt:

Shared Prompts

In multimodal models, inter-modal semantic associations exist between different modalities and can be leveraged to compensate for missing modality information. Specifically, when one modality is absent, the available modalities can utilize these associations to partially compensate for the missing information. To achieve this, we establish cross-modal connections by shared prompts, thereby enhancing the model's robustness under missing-modality scenarios.

Shared prompts, denoted as pis , are first constructed and then mapped to the respective modalities. This design allows the shared prompts to capture common features across modalities while the mapping functions learn modality-specific information. The mapping process is defined as:

where Gm is a bottleneck network consisting of an MLP and activation functions, significantly reducing the trainable parameters.

The constructed generative prompts pg,m and shared prompts ps,m are integrated to Eq.(2). The updated hi are send to Eq.(1) to derive the output for each transformer block.

4 Experiments

4.1 Datasets

We evaluated our proposed method on three widely used datasets: MM-IMDb [Arevalo et al., 2017], UPMC Food101 [Wang et al., 2015], and Hateful Memes [Kiela et al., 2020].

MM-IMDb. The MM-IMDb dataset is a benchmark for multimodal movie genre classification, consisting of 25,959 movies. Each sample includes textual and visual information, such as movie posters and plot summaries, making it a challenging dataset for integrating diverse modalities.

UPMC Food101. The UPMC Food101 provides a multimodal benchmark for food image classification, featuring images and rich textual metadata across 101 food categories. The metadata includes ingredient lists and cooking methods, offering critical contextual information to enhance model performance in understanding complex visual and textual cues.

Hateful Memes. The Hateful Memes dataset, developed by Facebook AI, is designed to evaluate multimodal hate speech detection. This datasets is designed to promote multimodal models perform well while restrain the unimodal models performance by adding challenging samples ("benign confounders").

4.2 Implementation Details

For fair comparison, we adopted the same data processing methods as in prior work on the three benchmarks [Lee et al., 2023]. The maximum text input lengths were set according to the specific datasets: 1024 for MM-IMDb, 512 for UPMC Food-101, and 128 for Hateful Memes. We employed the multimodal transformer ViLT [Kim et al., 2021] as the backbone model. The pretrained ViLT model was frozen, and only the learnable prompts and classification layer were finetuned on the target datasets. The lengths of the generative and shared prompts were set to 16 for MM-IMDb and UPMC Food101, and 4 for Hateful Memes. This is due to the short text input length (128 tokens) of Hateful Memes necessitated shorter prompt lengths to facilitate model optimization. The prompt memory was configured with N as 5 and memory size as 16. Generative and shared prompts were added only to the first 6 transformer blocks. All experiments used the AdamW optimizer, with an initial learning rate of 5 × 10-3 for MM-IMDb and UPMC Food-101, and 1×10-3 for Hateful Memes. The weight decay rate was set to 2 × 10-2 . All experiments were conducted on NVIDIA 4090Ti GPUs.

Missing modalities setting. During training, we followed the missing scenarios configurations outlined in [Lee et al., 2023]. Three missing-modality scenarios were examined: missing text, missing image, and missing both. For the missing text and missing image cases, a total missing ratio of η% was applied, while for the missing both scenario, each modality was missing at a ratio of η2 % individually. The missing data were simulated by replacing the corresponding modality with dummy inputs (e.g., empty characters for text and blank pixels for images). To evaluate the generalization of the model, we trained it with specific missing modality configurations and tested it across all three scenarios.

4.3 Results and Analysis

Various missing modality scenarios are configured during both training and inference, including inconsistent missing cases between these stages. This setup provides a comprehensive evaluation of the model's robustness and generalization. Given a specific missing modality scenario during training, we assessed the model performance under different missing scenarios during inference. We compared our proposed method with two existing approaches, ViLT [Ma et al., 2022] and MAP [Lee et al., 2023], across three benchmarks: MM- IMDb, Food101, and Hateful Memes. The experimental results are summarized in Tab.1. Attention and input refer to different prompt placement strategies within MAP.

The compared results reveal two key points: First, when the missing modality scenario during training and inference are consistent, our model demonstrates noticeable improvements over the previous SOTAmethods. For instance, on the MM-IMDb dataset, our model outperforms previous method under different missing conditions, achieving improvements such as 36.03 to 38.47 in the text-missing case, 44.64 to 48.59 in the image-missing case, and 41.48 to 42.47 in the bothmissing case. Second, when the missing modality scenarios during training and inference are inconsistent, our model significantly outperforms the baseline methods. For instance, on the Food-101 dataset, when the model is trained under the text missing scenario but tested under the image missing scenario, our model achieves a remarkable performance improvement from 27.82 to 80.81. Conversely, when trained with missing images and tested with missing text, the performance increases from 39.98 to 60.63. These results demonstrate that our model effectively compensates for missing modality information, enabling it to handle unseen missing modality scenarios during inference.

4.4 Ablation Studies

Evaluation under different missing rates. This section evaluates the model's robustness under varying missing rates. The missing rate is fixed during training, while the model is assessed under various missing rates during inference. Experiments were conducted on the MM-IMDb dataset. During training, missing cases were configured as follows: 70% missing text, 70% missing images, and 70% missing both. For inference, the missing rates ranged from 0% to 100%. The results are presented in Fig. 2. The results indicate that our method consistently outperforms previous models across all scenarios. When the missing cases during training and inference are consistent, our method achieves slightly better performance. Notably, when the missing cases during training and inference are inconsistent, previous models exhibit a significant performance drop, while our model remains stable across varying missing settings.

Furthermore, as the missing ratio increases, the performance of our model degrades more gradually, demonstrating its ability to effectively compensate for missing information. These findings emphasize the adaptability of our method under varying missing ratios, highlighting its superior capability to maintain performance across unseen missing configurations compared to previous methods.

Effect of each module. Our method incorporates two key components: generative prompts and shared prompts. To evaluate the contribution of each module, we conduct ablation studies on the MM-IMDb dataset, assessing the model performance under three different missing cases. Using the ViLT pretrained model as the baseline, we separately add generative prompts and shared prompts to examine their individual impact. Results show that adding either generative prompts or shared prompts alone significantly improves the model's robustness in handling missing modality scenarios. Furthermore, the last column in Tab. 2, representing the full model with both prompts, demonstrates that combining generative and shared prompts provides a synergistic effect, achieving the highest performance across all settings. This confirms that the two modules complement each other, with generative prompts dynamically retrieving semantically similar features and shared prompts establishing cross-modal associations to compensate for missing modalities more comprehensively.

Comparison with complete training data. In real-world scenarios, complete data pairs are sometimes available for training. To further evaluate our model, we trained it on complete data and evaluated its performance under various missing rates during inference. For evaluation, we averaged the results across the missing-text, missing-image, and missingboth cases to derive the final metric. The results, shown in Fig. 3(a), indicate that all models perform well when the test data is also complete. However, as the missing rate increases, the performance of previous methods declines significantly, while our model demonstrates greater stability and robustness.

Results of different missing rates during training. We evaluated the model by training with different missing rates (10%, 30%, 70%, 90%) and testing under varying missing rates during inference, with results shown in Fig. 3(b). In this part, the missing-both case was used for training, and the results represent the average performance across the missingimage, missing-text, and missing-both cases. Our model demonstrates strong generalization ability, maintaining consistent performance across different missing rates. This suggests that the proposed method effectively compensates for missing information, allowing the model to adapt to diverse scenarios. Notably, under high missing rates during training, the model achieves better performance when tested under similarly high missing rates. This result aligns with previous findings [Lee et al., 2023] and highlights the robustness of our approach in handling incomplete data distributions.

Number of prompt layers. In the proposed method, we designed generative prompts and shared prompts and integrated them into the pretrained model. This section examines the impact of the number of prompt layers on model performance. Specifically, generative prompts are fixed in 6 layers, while the number of shared prompt layers is varied from 1 to 6. Conversely, shared prompts are fixed in 6 layers, and the number of generative prompt layers is varied. The results indicate that although performance fluctuates slightly at certain points, the overall trend shows improvement as the number of layers increases.

Analysis of memory size. The prompt memory is designed for each modality to store its feature information through a series of prompts. Generative prompts are adaptively constructed by retrieving relevant information from the prompt memory to compensate for missing modalities. In this section, we analyze the impact of memory size K on model performance using the MM-IMDb dataset. As shown in Tab. 3, model performance initially improves as memory size increases, reaching its peak at K = 16. This suggests that a larger memory provides more diverse prompts and effectively captures modality-specific features. However, further increasing the memory size slightly reduces performance, likely due to increased complexity. These results highlight the importance of selecting an appropriate memory size to balance adaptability and efficiency.

Effect of prompt location. In our proposed method, prompts for each modality are directly added to their associated modality features, as shown in Eq. (2). To analyze the impact of prompt location, we conducted an ablation study comparing two strategies: Pre-Appended Prompting and Distributed Prompting. The Distributed Prompting strategy, as defined in Eq. (2), places generative and shared prompts directly before their respective modality features. In contrast, the Pre-Appended Prompting strategy combines the generative and shared prompts of all modalities and appends them at the beginning of the input sequence. The results, shown in Tab. 4, demonstrate that Distributed Prompting consistently outperforms Pre-Appended Prompting. This suggests that aligning prompts closely with their associated features leads to better model performance.

5 Conclusion

In this paper, we proposed Memory-Driven Prompt Learning, a novel framework designed to tackle the challenges posed by missing modalities in multimodal learning. Our approach introduces a compensation mechanism that integrates two complementary strategies: retrieving semantically relevant information from the missing modality itself and leveraging cross-modal associations derived from the available modalities. These strategies are realized through two types of prompts-generative prompts, which dynamically retrieve information from a prompt memory to compensate for missing modality features, and shared prompts, which infer and supplement common semantic information from the available modalities. Extensive experiments on multiple benchmarks demonstrate that our method significantly enhances model robustness and achieves state-of-the-art performance under diverse missing modality scenarios.

Acknowledgments

This work was supported by the National Key R&D Program of China under grant 2022YFF0901800, and the National Natural Science Foundation of China (NSFC) under grants No. 62176205, 62302383, and 62372365.