Robult: Leveraging Redundancy and Modality-Specific Features for Robust Multimodal Learning

1 Introduction

Motivation: In the Big Data era, multimodal learning significantly improves data exploitation, outperforming singlemodality approaches [Huang et al., 2021]. However, most existing methods [Daunhawer et al., 2023; Peng et al., 2022] operate under idealized assumptions: fully labeled training datasets and consistently available modalities during evaluation. In practice, the challenges of missing modalities and semi-supervised learning often coexist, yet current research typically addresses these problems in isolation. For example, missing modalities may arise when autonomous vehicles lose sensor inputs due to environmental obstructions or when medical diagnostics in resource-limited settings lack access to all imaging modalities. Simultaneously, the scarcity of labeled data across domains remains a critical bottleneck, particularly in multimodal contexts where individual modalities often require specialized labeling.

Addressing these challenges independently limits the adaptability of multimodal systems and fails to capture the complexities of real-world deployments. This work uniquely addresses these dual challenges by integrating strategies to handle incomplete modalities and leverage unlabeled data simultaneously. By doing so, our approach enhances both flexibility and applicability, enabling robust performance in diverse and imperfect scenarios. Unlike current literature, which rarely addresses both issues concurrently, this work bridges a critical gap with an innovative and unified solution.

Existing literature: One of the primary challenges in multimodal learning is handling corrupted or missing modalities. Existing methods to address missing modalities typically fall into two categories:

1. Generative approaches, such as VAE-based models [Wu and Goodman, 2018], which reconstruct missing modalities. While recent advancements [Feichtenhofer et al., 2022; Woo et al., 2023] demonstrate promising performance, these methods often depend on specific architectures, limiting their flexibility.
2. Transfer learning methods, which align latent spaces for cross-modal knowledge transfer [Ma et al., 2022; Lee and Van der Schaar, 2021; Wang et al., 2020]. These methods focus on adaptable training strategies [Chen et al., 2023] but often lack a strong theoretical foundation and are primarily guided by empirical intuition.

Simultaneously, the need for semi-supervised learning arises from practical challenges in labeling raw data, especially in domains where annotations are scarce or labor-intensive to obtain. This challenge is amplified in multimodal learning, as each modality may require distinct expertise for labeling. For example, tasks such as object segmentation across video and lidar data in autonomous driving [Zhang et al., 2022b] or medical segmentation across imaging modalities [Acosta et al., 2022] demand diverse and often non-standardized labeling procedures. Recent advancements in semi-supervised learning, such as knowledge distillation [Su et al., 2021] and pseudo-labeling [Aberdam et al., 2022], have shown promise, but these techniques are often designed for specific applications and struggle to generalize across varied multimodal settings.

#! , #" ; %

A detailed discussion of existing literature in both areas is provided in Appendix A.

Proposed approach: The dual challenges of missing modalities and semi-supervised learning remain critical open problems in multimodal research. This work advances transfer learning methods for missing modalities (category 2) that pertain architectural flexibility and dataset compatibility, while introducing a novel design to reduce dependence on labeled data, ensuring robustness and adaptability in real-world scenarios.

Using Partial Information Decomposition [Williams and Beer, 2010], the mutual information provided by an input X with M modalities (X1 ↪ ▷ ▷ ▷ ↪XM ) for a given task Y can be decomposed into:

where R represents redundancy (shared task-relevant information among M modalities), U denotes unique information specific to the ith modality, and S quantifies synergy, the additional knowledge generated through interactions among modalities (Figure 1 illustrates PID with 2 modalities).

In an ideal setting with access to all modalities, a fusion technique would efficiently capture R and S to optimize predictions for Y . However, in real-world scenarios where some modalities are unavailable, replicating S becomes challenging. For example, with two modalities, Y = linear(X1 ↪X2 ) = non linear(X1 ) demonstrates that synergy (S) improves predictions when both X1 and X2 are available. Access to X1 alone makes the relationship more complex and harder to model with deep networks.

Approaches like knowledge distillation [Chen et al., 2023] and contrastive learning [Radford et al., 2021] aim to address missing modalities by mimicking the representations produced by fused modalities in their absence. From an information-theoretic perspective, these methods focus on replicating redundant information (R) through latent-space alignment. Building on this foundation, we explicitly introduce a mutual information maximization objective (Objective 2.1) to align unimodal and fused representations. This alignment ensures efficient knowledge transfer while minimizing reliance on labeled data using a novel soft-positive pseudo-labeling mechanism that accounts for pseudo-label uncertainty.

While alignment effectively captures R, it can unintentionally diminish unique information (U) from individual modalities. This loss is particularly detrimental in semi-supervised settings or when other modalities are unavailable. To address this, we propose preserving modality-specific information (U i ) via Objective 2.2. This is achieved using a simple yet effective reconstruction procedure in the latent space, universally applicable across modalities. Our results and ablations (Tables 1 and 3) demonstrate how retaining U i improves performance.

Together, Objectives 2.1 and 2.2 form the foundation of our semi-supervised multimodal learning method, Robust Multimodal Pipeline (Robult). Robult effectively balances redundancy alignment and unique information preservation, ensuring accuracy and robustness in scenarios with missing modalities. Empirical results (Section 3) and theoretical underpinnings highlight Robult's superior performance compared to existing methods, demonstrating its adaptability to diverse real-world settings.

Contributions. Our primary contributions are:

• Jointly addressing the dual challenges of missingmodalities and semi-supervised learning.
• Framing two objectives under an information-theoretic perspective and deriving novel loss functions to achieve these goals.
• Introducing a soft Positive-Unlabeled contrastive loss that efficiently utilizes limited labeled data through selective weighting of potential positives.

2 Methodology

Training setting: We consider a training scenario where each sample includes all modalities, but only some samples have corresponding labels (Figure 2). Formally, let the training dataset be Dtrain = ¶(x1↪ y1)↪ ▷ ▷ ▷ ↪ (xk↪ yk)↪ xk+1↪ ▷ ▷ ▷ xn♦, where each data point xj = (x1j ↪ ▷ ▷ ▷ ↪ xMj ) contains M modalities. The dataset consists of n samples, of which k are labeled (0 → k < n).

Evaluation setting: To mimic real-world deployment, the evaluation dataset Xtest = ¶x1↪ ▷ ▷ ▷ ↪ xm♦ consists of samples with potential missing modalities - e.g. xj = (xaj 's); ↑a ↓

Figure 3. (3a) For each available modality i → ¶1↪ ▷ ▷ ▷ ↪M♦, f i (▷) extracts the latent representation hij. The modules gi (▷) and g0 (▷) then extract the unique information uij and redundant representation zij, respectively. These are combined and processed by the shared module c(▷) to predict ˜ yij. Late fusion strategies aggregate these outputs into the final prediction ˜ yj. (3b) When all modalities x1:M j are available, f 0 (▷) extract a fused latent vector h0j . This vector is directly passed through c(▷) to generate the final output ˜ y1:M j .

AMj ↔ ¶1↪ ▷ ▷ ▷ ↪M♦. Here, AMj denote the indices of all available modalities for sample xj (Figure 2).

Notation: For clarity, in the theoretical framework, we denote the input random variables corresponding to the ith modality as Xi (Figure 2). We would refer back to the observation-level notation xij whenever needed.

Figure 3 illustrates our versatile multimodal pipeline named Robult. Robult consists of M modality-specific branches and a fusion branch indexed with zero. For each input variable Xi , it is first projected into a shared latent space:

In parallel, a fusion network f 0 (▷) processes all input modalities jointly to produce a fused latent variable (also reside on the shared latent space):

where index 0 denotes the joint latent variable using all modalities. This latent-space projection provides two key benefits:

• Generalization: Robult supports different combination of modalities and their preferred projection methods/architectures.
• Efficient Fusion: Various strategies can be adopted before Robult's main logic to efficiently attain fused representations.

Each modality-specific module gi (▷) extracts unique information of corresponding modality:

while the shared module g0 (▷) effectively captures the redundant information by processing individual or joint latent variables:

Objectives: As discussed in Section 1, we draw inspiration from the second category of literature on missing modalities, which primarily leverages knowledge distillation and contrastive loss techniques [Poklukar et al., 2022b]. These methods aim to align unimodal (student) representations, such as Z i , with the fused (teacher) representation S, enabling unimodal representations to efficiently replicate the redundant information encapsulated in the fused representation. Although this redundant information exists within each modality, extracting it directly from a single modality can be significantly more complex without access to all modalities. For example, encoding object shape from an image may be challenging, whereas a textual description can explicitly provide the same information. Mimicking the fused representation offers a shortcut, simplifying the extraction of redundant information within unimodal representations.

However, these approaches fail in the semi-supervised training setting, owing to 2 emerging challenges:

• (1) The over reliance on label signals (e.g. classification clusters) in the alignment process.
• (2) the diminishing of unique information contained in different modality after alignment process.

Regarding the former challenge, we directly address the label need of Contrastive Learning with Label-level sampling [Zhang et al., 2022a] by a novel soft Positive-Unlabelled (PU) contrastive loss, together with an adaptive weighting strategy, detailed about which is covered in Section 2.1. Theoretically, this PU contrastive loss corresponds to a mutual information maximization problem between the desired fused representation S and the learned unimodal representation projected into that same latent space Z i 's. Objective 2.1 of our method can be expressed as follow:

Objective 2.1. Aligning S and Z i by maximizing the mutual information I(S↪ Z i ) (i = 1↪M).

For the latter challenge, we observe that attempting to only align modalities during training diminishes the unique information provided by each modality, thus the model is losing information that could help inform it when only that specific modality is available. Therefore, a model should ideally benefit by maintaining the redundancy while also explicitly preserving each modality's unique information. This claim is later supported by experimental results and ablations (Section 3 - Table 1). To address the challenge of vanishing modality-specific information from multimodal alignment during training, we emphasize a disentanglement strategy that preserves unique information while still facilitating the redundancy learning process of Objective 2.1. Robult integrates a set of modules gi (Hi ), where i = 1↪ ▷ ▷ ▷ ↪M, to produce unique representations U i for each modality. We aim to preserve the unique information for each modality via the learning of U i with Objective 2.2 as follows:

Objective 2.2. Learning U i by minimizing the conditional entropy H(Hi ♣Z i ↪U i ) (i = 1↪M).

During training, all branches are executed and three loss functions update the learned modules. The soft positive unlabeled loss, LPU , maximizes knowledge extraction from the few labeled samples in a batch (Subsection 2.1). The reconstruction loss, Lrec forces each branch to extract unique modalityspecific information (Subsection 2.2). The task-specific supervised loss, Lsup is used on the labeled samples across all modules to learn label information (Subsection 2.3).

2.1 MaximizingMutual Information with Soft Positive-Unlabeled Contrastive Learning

To address the impact of missing modalities, we aim to learn redundant information by aligning the fused latent variable S with unimodal representations Z i , as formalized in Objective 2.1. This is achieved by maximizing their mutual information I(S↪ Z i ). However, direct computation of this quantity is infeasible without access to the joint distribution pS↪Zi or the marginal distributions pS and pZi . Instead, we derive and optimize a lower bound for this mutual information.

Lower Bound Derivation. Let F be a binary random variable indicating whether a pair (sj↪ zik ) is sampled from the joint distribution pS↪Zi (F = 1) or from the product of marginal distributions pS ↗ pZi (F = 0). Then, a lower bound for I(S↪ Z i ) is expressed as:

where v(S↪ Z i ) is a non-parametric approximation of p(F = 1♣S↪ Z i ). For a sampled pair (sj ↪ zik ) in a batch of B samples, where sj is the fused representation for sample j and zik is the ith modality-specific representation for sample k, v(sj ↪ zik ) is defined as:

The derivation of Result 2 is detailed in Appendix B.2.

Challenges with Sampling. The lower bound in equation 2 relies on expectations under EpS↪Zi , a key source of deviation in existing studies, which presents challenges in practice. Two common sampling strategies include:

1. Instance-level sampling, which considers only intrasample pairs (sj ↪ zij ) within q mini-batch [Radford et al., 2021].
2. Label-level sampling, which uses label information to sample inter-sample pairs (sj↪ zik ) with the same labels, e.g. yj = yk [Zhang et al., 2022a].

The first approach risks introducing false negatives, while the second requires fully labeled data, limiting its applicability in semi-supervised settings.

Soft Positive-Unlabeled (PU) Contrastive Loss. To overcome these limitations, we propose a novel soft PositiveUnlabeled (PU) contrastive loss with adaptive weighting. Let L indicate whether a sampled pair ⇐sj ↪ zik ⇒ is labeled (L = 1) or not (L = 0) (where L = 1 if the label information of both samples i and k is known, and L = 0 otherwise), The lower bound in equation 2 can then be decomposed as:

We formulate the two terms in equation 3 as separate loss components: Llb for labeled data and Lulb for the unlabeled data.

For labeled samples, let Bj1↪1 denote the index set of inputs in the batch that share the same class as sample j (F = 1) and are labeled (L = 1), i.e. k ↓ Bj1↪1 ⇑⇓ (sj↪ zik ) ⇔ p(S↪ Z i ♣F = 1↪ L = 1). The labeled loss is then defined as a NT-Xent-like contrastive loss [Chen et al., 2020]:

For unlabeled samples, directly sampling from p(S↪ Z i ♣F = 1↪ L = 0) is challenging due to the absence of label information. To address this, we propose leveraging the output of the Robult classifier to generate soft labels, regularized by adaptive weights calculated dynamically within each minibatch. This approach effectively balances the influence of soft-labeled pairs and mitigates the limitations of traditional pseudo-labeling methods [Aberdam et al., 2022].

A key challenge during initial training stages is the instability of the Robult classifier, which may produce unreliable outputs and hinder effective filtering of false positives, as illustrated in Figure 4. To counteract this, we adjust the contribution of soft-labeled pairs to the loss function, ensuring robust training. For each anchor sample sj within a mini-batch, there are labeled positive partners or, in unsupervised scenarios, unimodal representations zij . The average proximity of these labeled partners to sj serves as a reference for determining "positive" proximity thresholds. Unlabeled positive pairs identified by the Robult classifier are expected to exhibit proximity close to this reference mean. To refine the loss computation, we increase the weight of pairs that align closely with the reference mean and decrease the influence of outliers. This weighting mechanism, implemented using an RBF kernel, allows precise adjustment of each couplet's contribution based on its proximity. By dynamically adapting weights, our method effectively reduces the impact of potential false positives and enhances the overall performance of the final loss function, as demonstrated in Figure 4.

Let Bj1↪0 denote the index set of inputs in the batch that share the same class as anchor j (F = 1) but are not labeled (L = 0), i.e. k ↓ Bj1↪0 ⇑⇓ (sj↪ zik ) ⇔ p(S↪ Z i ♣F = 1↪ L = 0). The unlabeled loss is given by:

The complete soft Positive-Unlabeled (PU) loss is the sum of the labeled and unlabeled components:

2.2 Minimizing Conditional Entropy with Latent Reconstruction Error

This section details the procedure for achieving Objective 2.2, which focuses on preserving unique information U i . Let pU i ↪Zi denote the joint distribution of U i and Z i , where (uij ↪ zij ) ⇔ pU i ↪Zi are derived from the corresponding instance hij . Inspired by [Chen et al., 2016], the conditional entropy H(Hi ♣Z i ↪ U i ) is expressed as:

Upper Bound Derivation. Since directly computing p(Hi ♣ U i ↪ Z i ) is challenging, we approximate it using a distribution q(Hi ♣ U i ↪Z i ). Substituting q into Eq. 8, we derive:

The last inequality arises because the KL divergence KL(p♣♣q) is non-negative. Thus, minimizing H(Hi ♣Z i ↪U i ) reduces to minimizing its Evidence Lower Bound (ELBO)- like [Kingma and Welling, 2014] upper bound using the approximating distribution q (Hi ♣ U i ↪Z i ).

Modeling q (Hi ♣ U i ↪Z i ) with Latent Reconstruction. We model q (Hi ♣ U i ↪Z i ) through a latent reconstruction procedure in the shared latent space. Specifically, we define a reconstruction module ri (U i ↪Z i ) = ˜Hi where ˜Hi approximates Hi . For each pair (uij ↪ zij ) ⇔ pU i ↪Zi generated from hij , the module ri (▷) attempts to reconstruct ˜hij such that it closely resembles hij . The reconstruction loss is formulated as:

where ⇐▷; ▷⇒ denotes the L2-normalized dot product operation, B is the size of mini-batch, and M is the number of modalities.

This reconstruction in latent space is computationally efficient and alleviates the complexity of directly reconstructing raw modality data. Moreover, it generalizes well across various modalities, enhancing the flexibility of Robult. The reconstruction loss Lrec is back-propagated exclusively through the M unimodal branches. This design ensures that the unique information U(Xi ; Y ) is preserved through U i , without interfering with the shared branch's focus on learning redundant information, as discussed in Section 2.1.

2.3 Training strategy

The objectives outlined in Sections 2.1 and 2.2 are optimized through their respective loss functions. Due to the distinct purposes of these objectives, it is advantageous to learn them separately. Specifically, we apply Lrec to guide the learning of gi (▷) (i = 1↪ ▷ ▷ ▷ ↪M), while LPU drives the optimization of f i (▷)↪ f 0 (▷), and g0 (▷). To maximize the utility of labeled data, we incorporate an additional supervised loss Lsup. This loss directs the learning process for the entire Robult network and adapts based on the task type: L1 loss for regression tasks and cross-entropy Lce for classification tasks. A detailed training procedure, including loss formulations and implementation details, is provided in Appendix B.3.

3 Experimental Results

3.1 Datasets andMetrics

Dataset: We conduct experiments on the following datasets.

• CMU-MOSI [Zadeh et al., 2016] & CMU-MOSEI [Zadeh et al., 2018b]: Containing three modalities - text, audio, and visual, supporting sentiment analysis and emotion recognition tasks. Each video is labeled on a scale from -3 (negative) to 3 (positive) sentiment.
• MM-IMDb [Arevalo et al., 2017]: Containing image and text modalities, serving genre classification task - which involves multi-label classification as a movie has several genres.
• UPMC Food-101 [Wang et al., 2015]: Containing two modalities, text and images, this dataset is a classification dataset consisting of 101 food categories.
• Hateful Memes [Kiela et al., 2020]: Containing text and image modalities, this dataset aims at identifying hate

speech in memes. This dataset includes challenging examples that are similar to hateful ones but are actually harmless.

Metrics: For sentiment analysis related to CMU-MOSI and CMU-MOSEI datasets, we adopt mean absolute error (MAE), correlation (Cor), binary accuracy, and F1 score, following [Poklukar et al., 2022b; Tsai et al., 2018]. Here, binary categories determine positive sentiment scores (> 0) or negative ones (< 0). For the evaluation of the three remaining datasets, we adhere to the metrics specified in [Lee et al., 2023b]. With the MM-IMDb dataset, the multi-label classification performance is assessed using F1-Macro. The classification accuracy is employed for the UPMC Food-101 dataset. Lastly, for Hateful Memes, the evaluation is based on the AUROC metric.

3.2 Baselines and Experimental Settings

Baselines: We incorporate several state-of-the-art approaches representing popular strategies into our comparative evaluation. Specifically, GMC [Poklukar et al., 2022b] serves as a contrastive learning-based approach, ActionMAE [Woo et al., 2023] represents a generation-based method, and we include a Transformer-based approach proposed in [Lee et al., 2023b], referred to as Prompt-Trans for brevity. To ensure optimal reproducibility, we inherit the implementations of all baseline methods from their original code bases. Additionally, we implement unimodal frameworks (Unimodal) for each modality, trained in a supervised manner with available labels, to serve as our baseline comparison.

Implementation details: To ensure a fair comparison, we use similar encoder architectures for processing raw data modalities whenever possible. The unimodal baselines are designed with the same architectures as Robult, each with its own classifier. For Robult, positive samples for the soft P-U loss are determined after discretizing labels if needed. Specifically, in the cases of CMU-MOSI and CMU-MOSEI datasets, label information in the range of [≃3↪ 3] is quantified into 7 discrete categories (≃3↪≃2↪ ▷ ▷ ▷ ↪ 3). Additionally, for the multi-label dataset MM-IMDb, two samples are considered positive if they share all the same labels. Regarding Prompt-Trans, we only report its results for three datasets involving two modalities (MM-IMDb dataset, UPMC Food101 dataset, and Hateful Memes dataset), as the extension to multiple modalities cannot be directly inferred from the original work [Lee et al., 2023b].

Experimental details: The primary focus of our performance reporting is on two extreme scenarios: semisupervised settings with only 5% labeled data and scenarios where only a single modality is presented during evaluation. All reported results are averaged over 3 different random seeds. In the semi-supervised setup, the newly created labeled portion is ensured to maintain the correct label ratio as the original training sets. Additional experiments extending these two settings to more modalities and a higher percent of labeled data are detailed in Appendix D.1 and D.2 respectively. Specific details on implementation settings relating to each dataset are provided in Appendix - C.2.

3.3 Main Quantitative Results

All results are shown in tables with the best outcomes in red and the second-best in blue.

Sentiment Analysis: The results for CMU-MOSI and CMUMOSEI datasets are summarized in Table 1. For both datasets, Robult significantly outperforms all the compared methods, suggesting its effectiveness and consistency in semi-supervised and missing modality scenarios. Regarding CMU-MOSI, due to its smaller scale compared to CMUMOSEI, the labeled portions are also smaller. This condition poses a challenge for existing baselines that heavily rely on label information. In contrast, Robult effectively addresses this challenge, demonstrating the ability to extract meaningful representations even with limited labeled data. On CMUMOSEI, Robult consistently produces superior representations, achieving the best performances across all recorded metrics. Notably, Robult improves the correlation (Corr) between the predicted sentiment levels and ground truth by up to 19▷8%, outperforming the second-best method, which is the unimodal for textual data.

Classification tasks: In Table 2, empirical results for three classification tasks show that Robult consistently outperforms existing approaches and baselines in most cases, except for one scenario on the Hateful Memes dataset with the full modality available, where Robult achieves comparable performance with Prompt-Trans [Lee et al., 2023b]. Notably, the Hateful Memes dataset includes samples with "benign confounders", negatively impacting performance when models rely solely on single modalities [Kiela et al., 2020]. Leveraging the soft Positive-Unlabelled loss, Robult effectively addresses and mitigates performance gaps with either single modality inputs or the full ones. In addition, we calculate F1 macro scores for all methods on these three datasets in the unimodal and multimodal cases. We further visualize a Critical Difference Diagram [Demˇsar, 2006] in Figure 5. This diagram visually represents the performance among different machine learning algorithms across various datasets by displaying the mean performance ranks, with lower being better, and connecting statistically indistinguishable groups (within 95% confidence level) with a thin horizontal bar, as per the Friedman hypothesis test. From the diagram, Robult exhibits a clear improvement gap compared to other state-of-the-art methods in average ranks, while ActionMAE and PromptTrans show no statistically significant difference in their performance.

3.4 Main ablation studies

Owing to space constraint, we provide here a key analysis of ablation study for different components of Robult. For a more comprehensive analysis, please refer to the additional experiments in Appendix D, which offer further insights into how architectural choices, Soft-PU loss, and weighting schemes influence Robult's performance quantitatively and qualitatively.

We evaluate the impact of each loss component on Robult's performance using Hateful Memes dataset, which mirror the semi-supervised and missing modalities conditions of our main experiments. This analysis involves testing variations of Robult with different ablations. (1) Removal of Lsup - this setting utilizes available label information only in L(u)lb, so Robult can only produce latent representations. An additional Logistic Regressor is trained with these representations as its input, and this pipeline's final scores are reported. (2) Removal of Lrec - this setting discards Lrec, corresponding to our Objective 2.2. (3) Removal of Llb - this setting makes the learning of Objective 2.1 rely only on Lulb. (4) Removal of Lulb - this setting associates Objective 2.1 exclusively with Llb. Table 3 summarizes the results of this ablation experiment. Overall, any ablation negatively impacts the performance of Robult. In particular, the absence of Lsup significantly worsens the performance, as there is no loss guiding the learning of Robult's classifier, which is crucial for generating soft label information consumed by the soft PositiveUnlabeled loss Lulb. Consequently, this ablation adversely affects two loss components, explaining the poorest result among all variations. The removal of Lrec particularly harms the performance with unimodal inputs, aligning with the motivation for Objective 2.2, as the unique information U i is no longer preserved. In two remaining cases, both ablations diminish Robult's overall performance, indicating their equal contribution to achieving Objective 2.1.

4 Contributions &Limitations

Contributions: Our Robult pipeline leverages limited label data through a soft Positive-Unlabelled (PU) loss and latent reconstruction loss, enhancing modality interactions and preserving unimodal data integrity. It supports various modality types and quantities, scales linearly with modalities, and functions independently of specific architectures. This flexibility facilitates integration with existing DL frameworks, advancing multimodal learning in practical settings.

Limitations. Robult's design presumes that the proximity of positive couplets follows a Gaussian distribution, a method proven empirically but not theoretically. Future work should seek theoretical validation for this assumption. Moreover, with our setting, the potential of labeled data in scenarios with missing modalities in training remains untapped. Exploring these cases could further improve Robult's effectiveness in complex real-world applications.

Acknowledgements

The work of Duy A. Nguyen was supported in part by a PhD fellowship from the VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam.