Trust, but Verify: Uncertainty-Driven Evidential Multimodal Representation Learning

1 Introduction

Multimodal perception lies at the core of modern AI [Baltruˇsaitis et al., 2018; Zhang and Han, 2025; Wang et al., 2026], revolutionizing how machines understand the world in high-stakes domains, from autonomous driving [Caesar et al., 2020; Martin et al., 2019] and medical diagnostics [Kaczmarczyk et al., 2024] to human-computer interaction [Liu et al., 2026; Zhang et al., 2021a; Liang et al., 2023]. Unlike curated lab datasets, real-world data is inherently imperfect, characterized by sensor noise, environmental ambiguity, and conflicting signals between modalities. Learning robust representations in these "in-the-wild" settings demands more than simply aligning modalities; it requires a sophisticated capacity to handle uncertainty. However, this uncertainty is not a monolithic problem. It is a multi-level phenomenon that manifests from the quality of individual data streams to the complex interactions between them. Prevailing paradigms, as illustrated in Figure 1(A), often fail because they address these challenges in a disconnected manner. A reliabilityagnostic feature extractor may blindly trust a noisy input (e.g., a blurry X-ray), while a separate uncertainty analysis module correctly identifies the poor quality, but this critical insight is ignored. By failing to integrate these steps, such methods often produce brittle and dangerously overconfident models, limiting their real-world applicability. In contrast, our work proposes a systematic, hierarchical approach to resolve these uncertainties, leading to a correct and calibrated decision, as conceptualized in Figure 1(B).

∗Corresponding author.

The foundational level of this challenge is Intrinsic Uncertainty, which pertains to the reliability of each individual modality. A meme's image might be heavily compressed and artifact-ridden; a product review's text could be sarcastic or contain typos; a video's audio track may be corrupted by background noise. A dependable model must first act as a discerning critic, asking: "Before I fuse any information, how much should I trust this single piece of evidence?" Many current approaches [Liang et al., 2023; Mai et al., 2022; Yang et al., 2022] sidestep this crucial first step or employ black-box reliability scores that lack interpretability, failing to identify why a modality is deemed uncertain.

Beyond this individual assessment, a deeper Relational Uncertainty emerges from the interplay between modalities. This can manifest as direct conflict (e.g., an image of a peaceful landscape paired with hateful text) or as a nuanced interplay of complementarity and redundancy. For instance, a sharp product photo might merely be redundant to a glowing text review, but a blurry photo becomes critically complementary to it. This raises a more complex question: "How can the model intelligently arbitrate between modalities, dynamically deciding whether a modality-specific cue is a critical complementary signal, a simple redundancy, or misleading noise?" Factorization-based methods [Zhang et al., 2021b; Liang et al., 2023; Madaan et al., 2024; Wang et al., 2025] attempt to separate shared and unique features, but they do so under an implicit assumption of equal reliability, failing to adapt their factorization strategy when one modality is clearly less trustworthy than another.

Ultimately, the intrinsic and relational uncertainties identified in the preceding stages must be reconciled and combined into a single, trustworthy final decision. This brings us to the third level of our hierarchy: Aggregated Uncertainty. This concerns the challenge of coherently fusing multiple, potentially conflicting streams of evidence and their associated uncertainty signals. When faced with low-quality inputs or irreconcilable disagreements, a robust model should not be forced to make a confident guess. The critical question becomes: "How can the model aggregate all known ambiguities to produce a final, well-calibrated belief that transparently reflects the total uncertainty of the system?" Naive averaging or static fusion rules [Huang et al., 2022; Zhang et al., 2023] often fail at this aggregation step, producing overconfident errors in such scenarios, which is unacceptable in safety-critical applications.

To systematically address this uncertainty hierarchy, we introduce Adaptive Evidential Multimodal Representation Learning (AEMRL), a framework architecturally designed to tackle each level. To address Intrinsic Uncertainty, we propose Disentangled Evidential Uncertainty Encoding (D-EUE), a novel mechanism that provides an interpretable, class-aware reliability assessment. To manage Relational Uncertainty, we pioneer Uncertainty-Conditioned Dynamic Factorization (UDF), which uses a hypernetwork to leverage the intrinsic uncertainty from D-EUE and dynamically generate a tailored factorization strategy. Finally, to resolve Aggregated Uncertainty, our Adaptive Fusion & Conflict-Aware Calibration (AFCAC) module intelligently integrates the purified evidence streams and scales the final belief based on inter-modal conflict, ensuring a wellcalibrated and trustworthy output. Extensive experiments on five diverse benchmarks, spanning domains from medical imaging to human activity recognition, demonstrate that AEMRL consistently sets a new state-of-the-art in performance, calibration, and robustness over strong baselines.

2 RelatedWork

2.1 Multimodal Representation Learning

Early large-scale image-text models such as CLIP and ALIGN use dual encoders with a contrastive objective to embed paired inputs into a shared space [Radford et al., 2021; Jia et al., 2021]. Later "one-stream" architectures (e.g., FLAVA [Singh et al., 2022], VATT [Akbari et al., 2021]) introduce joint Transformers for feature fusion but assume that shared information is both necessary and sufficient for downstream tasks. To relax this assumption, recent work explores factorized representations. MIB maximizes I(Z; Y ) while limiting I(Z;X) the learning of minimal task-relevant codes [Mai et al., 2022]. FactorCL extends this by decomposing features into shared and modality-specific components using contrastive objectives [Liang et al., 2023]. However, these approaches neither quantify epistemic uncertainty nor treat cross-modal conflict as informative. AEMRL advances beyond this static paradigm by introducing a synergistic, uncertainty-aware pipeline. Instead of treating uncertainty as a post-hoc fusion tool, AEMRL leverages an evidential framework as a dynamic control signal to guide feature factorization, enabling per-sample adaptive disentanglement based on modality reliability.

2.2 Uncertainty-Aware Multimodal Fusion

Quantifying predictive reliability has been extensively studied in single-modal settings, from MC-Dropout [Gal and Ghahramani, 2016] and Deep Ensembles to Dirichlet-based Evidential Deep Learning (EDL) [Sensoy et al., 2018]. In the multimodal domain, recent methods introduce quality gates at the logit level-e.g., Quality-Aware Multimodal Fusion (QMF) [Zhang et al., 2023]-or attach evidential heads per modality, as in DDEF (Dual-level Deep Evidential Fusion) [Shao et al., 2024]. However, these approaches treat uncertainty primarily as a fusion-stage issue, focusing on combining modality-specific outputs while ignoring how uncertainty should shape the feature learning process itself. If the features are already corrupted, even optimal fusion yields subpar results. AEMRL fills this gap by shifting from uncertainty-aware fusion to uncertainty-driven representation learning. Its key innovation lies in the D-EUE and UDF pipeline: interpretable uncertainty signals are first generated, then used to dynamically guide feature factorization prior to fusion. This proactive purification ensures that inputs to the final conflict-aware calibration are already reliable, resulting in state-of-the-art performance and calibration.

3 Method

3.1 Overview

Our proposed framework, Adaptive Evidential Multimodal Representation Learning (AEMRL), is architecturally designed to systematically address the multi-level uncertainty inherent in real-world scenarios. As illustrated in Figure 2, it operates through three synergistic stages, each tailored to a specific level of our proposed uncertainty hierarchy. The following sections detail the technical implementation of each stage: first, assessing intrinsic uncertainty via Disentangled Evidential Uncertainty Encoding (D-EUE); second, managing relational uncertainty via Uncertainty-Conditioned Dynamic Factorization (UDF); and finally, calibrating global uncertainty via Adaptive Fusion & Conflict-Aware Calibration (AFCAC).

3.2 Disentangled Evidential Uncertainty Encoding

To address the challenge of varying reliability and noise across different data sources in multimodal learning, we introduce the Disentangled Evidential Uncertainty Encoding (D-EUE) module. Traditional evidential methods [Xu et al., 2024; Han et al., 2022; Sensoy et al., 2018] often employ a monolithic "black-box" evidence head, which obscures the connection between input features and the resulting classlevel evidence, thereby limiting interpretability. Our D-EUE module enhances this paradigm by introducing a structured, parallel architecture that not only quantifies modality-specific epistemic uncertainty but also provides a more interpretable, disentangled pathway for evidence generation for each class.

D-EUE transforms each modality's feature representation into a Dirichlet distribution using a set of class-specific evidence generators, instead of a single shared network. This design enables fine-grained analysis of uncertainty sources. For each modality m, the input x(m) is first encoded by a modality-specific encoder fm to produce features h(m) . These are fed into a structured evidence head comprising C parallel generators gm,c, one per class. Each gm,c (a twolayer MLP) outputs a non-negative evidence value em,c dedicated to class c. The resulting evidence vector e(m) = [em,1, . . . , em,C] aggregates class-wise evidence through distinct, specialized pathways.

The evidence vector e(m) is used to parameterize a Dirichlet distribution with concentration parameters α(m) :

where 1 is a vector of ones, ensuring α(m) c > 0. The total evidence for modality m is α(m) 0 = ∑c α(m) c , where c represents the classes. The predictive probability for modality m and the epistemic uncertainty Um for modality m are quantified as:

A higher Um indicates lower confidence in the information from modality m, stemming from less total evidence α(m) 0 .

This explicit uncertainty Um is then leveraged as a critical control signal in subsequent stages. Unlike prior approaches that use a single, fully coupled evidence head, D-EUE's disentangled architecture provides a more transparent and interpretable uncertainty assessment. This allows the model not just to know that it is uncertain but potentially why, by inspecting the evidence contributions from each specialized pathway. This fine-grained signal enables a more sophisticated, reliability-aware adaptation in downstreammodules.

Evidence generators are trained via a loss that promotes both prediction accuracy and functional specialization. For modality m, the standard evidential loss is defined as:

To ensure the parallel generators learn distinct functions, we add our novel disentanglement constraint. By promoting orthogonality among the generator weights (Gm,c), this loss encourages each generator to focus on a unique subspace of features. This specialization prevents evidence meant for one class from interfering with another, leading to a cleaner and more interpretable final uncertainty assessment. The loss is:

The complete objective function for the D-EUE module is the weighted sum of these two components:

3.3 Uncertainty-Conditioned Dynamic Factorization

To address relational uncertainty under varying modality reliability, we introduce the Uncertainty-Conditioned Dynamic Factorization (UDF) module. While prior methods [Liang et al., 2023; Yang et al., 2022; Robinet et al., 2024; Wang et al., 2025] perform static, reliability-agnostic factorization, they apply fixed projections regardless of input quality-risking the extraction of spurious "unique" features from noise. In contrast, UDF conditions the factorization process on intrinsic uncertainty Um, enabling dynamic adaptation of feature extraction.

At its core, UDF employs a hypernetwork [Ha et al., 2016], HyperNetθ , which maps a modality's uncertainty to its optimal factorization parameters, transforming factorization into a sample-aware operation. For each modality m, the two-layer MLP hypernetwork takes the scalar uncertainty Um as input and generates the weights for the unique projection matrix W (m) u . This allows the model to learn contextsensitive strategies: generating expressive projectors for reliable modalities and conservative ones for noisy inputs.

The unique feature projection matrix W (m) u is thus dynamically generated, while the shared feature projector W (m) s is a statically learned parameter to capture cross-modal commonalities. The feature h(m) is decomposed into shared features sm and unique features um as follows:

This dynamic factorization enables the model to disentangle complementary cues from conflicting noise based on realtime reliability estimates. Unlike prior methods with fixed projection matrices, UDF offers a principled, data-driven approach to managing relational uncertainty by adapting the feature space. The resulting purified representations, sm and um, form a cleaner foundation for the final fusion stage.

The UDF module is trained by minimizing a hybrid loss that combines an information-theoretic objective with a decorrelation term. It enhances the relevance of shared features while suppressing irrelevant or unique ones through the following criterion:

To estimate the mutual information lower bound, ILB, we employ the widely used Donsker-Varadhan (DV) estimator [Belghazi et al., 2018]. This estimator utilizes a critic network, Tθ(·, ·), to learn a lower bound on the KL-divergence between the joint and the product of marginal distributions. The DV lower bound is formulated as:

We apply this estimator to both the shared features (ILB(sm; Y )) and, in conditional form, to the unique features. For the upper bound IUB, we use the corresponding dual formulation. The critic Tθ is a two-layer MLP. This informationtheoretic objective encourages the model to encode all taskrelevant information into the shared representation sm.

To further push the shared and unique channels into complementary sub-spaces, we add a soft decorrelation term:

3.4 Adaptive Fusion &Conflict-Aware Calibration

Leveraging the purified features from UDF and the initial evidence from D-EUE, the final stage addresses the challenge of producing a well-calibrated global decision via the Adaptive Fusion & Conflict-Aware Calibration (AFCAC) module. Traditional methods often use naive averaging or static fusion rules [Zhang et al., 2023; Peng et al., 2022], which can lead to overconfident errors when faced with conflicting or lowquality evidence. AFCAC, instead, intelligently integrates the available information and calibrates the final prediction's confidence based on inter-modal agreement.

The core of AFCAC is a two-step process of dynamic weighting and conflict-aware scaling. First, it computes a fusion weight γm for each modality that reflects both its task relevance and its reliability. The relevance score, Rm, quantifies the total task-relevant information frommodalitym. We define it as the sum of the mutual information lower bounds for its shared and unique features, as estimated by the critic networks Tθ in the UDF stage:

The reliability is given by (1 - Um) from the D-EUE stage. These two components are combined and normalized across all M modalities to produce the final dynamic weight γm:

The modality-specific evidences e(m) are then fused via this adaptive weighted sum:efusion = ∑Mm=1 γme(m) .

Second, the module explicitly detects inter-modal conflict to prevent overconfident predictions. We measure the disagreement by calculating the symmetric Kullback-Leibler (KL) divergence between the predicted probability distributions, ˆp(v) and ˆp(w) , of each pair of modalities. The total conflict score, ∆, is the sum of all pairwise disagreements:

This conflict score is then used to scale the fused evidence:

where κ is a tunable hyperparameter that controls the sensitivity to conflict. The final Dirichlet parameters are then computed as αfinal = escaled + 1.

This explicit calibration ensures the model's final output reflects the cumulative uncertainty from the entire pipeline. Unlike prior approaches that might overlook conflicts or use static fusion rules, AFCAC provides a robust mechanism to manage disagreements and prevent overconfidence. It allows the model to "fail gracefully" by reporting high aggregated uncertainty when the collective evidence is contradictory.

The entire AEMRL framework is trained with a composite loss that combines module-specific objectives, anchored by the final classification loss. The overall objective is:

Here, LCE is the cross-entropy loss between the final prediction pfinal and the one-hot label y. The hyperparameters λevd and λudf weight the uncertainty and factorization objectives. This unified loss guides all components to jointly manage the full uncertainty hierarchy.

4 Experiment

4.1 Experimental Setup

Datasets. To demonstrate AEMRL's ability to generalize across diverse web scenarios-including safety-critical settings, sensor fusion under challenging conditions, and multimedia understanding-we evaluate on five complementary benchmarks: MIMIC-CXR [Johnson et al., 2019], Drive&Act [Martin et al., 2019], VGG-Sound [Chen et al., 2020], Hateful Memes [Kiela et al., 2020] and MM-IMDb [Arevalo et al., 2017]. Full dataset statistics and preprocessing details are in Supplementary Material.

Baseline. We compare AEMRL against a broad range of SOTA methods that reflect key directions in multimodal representation learning: (1) Conventional alignment and fusion: CLIP [Radford et al., 2021], ALIGN [Jia et al., 2021], FLAVA [Singh et al., 2022], and VATT [Akbari et al., 2021]; (2) Shared-unique factorization: MMIB [Mai et al., 2022], FactorCL [Liang et al., 2023], I2M2 [Madaan et al., 2024], and DISENTANGLEDSSL [Wang et al., 2025]; (3) Uncertainty-aware fusion: DDEF [Shao et al., 2024], Contextual Discounting (CD) [Huang et al., 2022], QMF [Zhang et al., 2023], OGM-GE [Peng et al., 2022], and EAU [Gao et al., 2024]; (4) Missing-modality adaptation: ModDrop [Neverova et al., 2015] and MLA [Zhang et al., 2024]. More details are provided in the Supplementary Material.

Implementation Details. All models are implemented in PyTorch and trained on 4 A100 GPUs. For each dataset, we use standard pretrained backbones-ResNet-50 [He et al., 2016] for images, 3D ResNet-50 [Hara et al., 2018] for video, BERT-base [Devlin et al., 2019] for text, and CNN14 [Wang et al., 2022] for audio/IMU-and fine-tune end-to-end. We adopt canonical task-specific metrics: AUROC (MIMIC-CXR), F1 (Drive&Act), Top-1 (VGGSound), Accuracy (Hateful Memes), and mAP (MM-IMDb), and uniformly report Expected Calibration Error (ECE). Optimization uses AdamW [Kingma and Ba, 2014] with a cosine learning-rate schedule and batch size of 64 per GPU. Results are averaged over five seeds (0-4). Key module configurations are as follows. The D-EUE module uses lightweight, class-specific generators, each a two-layer MLP with hidden size 128. The UDF hypernetwork is also a two-layer MLP with hidden size 64 and GELU activation. The trade-off parameter β in the mutual information objective (Eq. 8) is set to 0.5. Loss balancing hyperparameters, selected via grid search on a validation set, are λevd = 1.0, λudf = 0.5, and λorth = 0.1 across all datasets. The conflict scaling factor in AFCAC is κ = 2.5. Full experimental details, including dataset-specific preprocessing, splits, and hyperparameter listings, are provided in the Supplementary Material.

4.2 Main Results

As Table 1 illustrates, our framework consistently establishes a new state-of-the-art on the primary performance metrics across all tasks, but the nature and magnitude of these gains vary in ways that directly support our central thesis.

Firstly, AEMRL's largest gains appear on datasets with unreliable modalities, notably Drive&Act and VGGSound. On Drive&Act, which contains noisy IMU sensor data, AEMRL achieves an F1 of 0.732, exceeding DISENTANGLEDSSL by 2.9 points. On VGGSound, where visual cues are weakly aligned with audio, the Top-1 accuracy improves by 2.3 points (0.380 vs. 0.357). These results strongly validate our Uncertainty-Conditioned Dynamic Factorization (UDF): unlike static methods that uniformly handle modality-specific features, AEMRL adaptively suppresses noisy IMU signals or irrelevant video frames, preventing them from degrading the final representation. Secondly, On cleaner datasets such as MIMIC-CXR, AEMRL shows smaller accuracy gains but a clear advantage in calibration. This indicates that even with relatively clean modalities, the combined reasoning of D-EUE and AFCAC yields a more trustworthy model-one that not only improves accuracy but also recognizes subtle ambiguities, an essential property for high-stakes domains such as medical diagnosis. Finally, On tasks involving semantic ambiguity, such as Hateful Memes and MM-IMDb, AEMRL consistently outperforms all baselines. The accuracy gain on Hateful Memes highlights its ability to handle not only signal-level noise but also high-level semantic conflict. Through UDF's adaptive factorization and AFCAC's explicit conflict scaling, the model learns to correctly resolve cases where one modality alters the interpretation of another.

4.3 Ablation Studies

Table 2 details our ablation study across five benchmarks, validating the necessity and synergy of the three-level uncertainty hierarchy in AEMRL. First, removing all evidential components to create a deterministic baseline severely degrades performance, elevating Expected Calibration Error, ECE, by 40 to 50% on Drive&Act and MIMIC-CXR. Substituting the decoupled D-EUE with a standard monolithic EUE head similarly impairs both accuracy and calibration, confirming the advantage of disentangled uncertainty signals. Next, replacing the uncertainty-directed factorization, UDF, with static factorization triggers the largest overall performance drop, proving that processing must dynamically adapt to modality reliability. When comparing the UDF hypernetwork against scalar and vector gating on Drive&Act, the full hypernetwork strictly outperforms both alternatives. This demonstrates its superior capacity to capture fine-grained reliability variations, particularly under noise. Finally, ablating the conflict scaling mechanism within AFCAC reduces AUROC by a mere 0.006 on MIMIC-CXR but substantially worsens calibration, increasing ECE by 15 to 25% across all benchmarks. Additionally, substituting dynamic weighting with naive averaging degrades overall performance, proving that reliability-aware fusion remains indispensable even following UDF purification.

4.4 Robustness toModality Conflict

To evaluate AEMRL's ability to handle high-level semantic disagreement-a central challenge in Web content analysis-we conducted a conflict stress test on the Hateful Memes benchmark. The test set was partitioned into three bins of increasing conflict, based on the pre-fusion symmetric KLdivergence between image and text modalities. We then measured accuracy and reported uncertainty for AEMRL and key baselines within each bin. As shown in Figure 3, AEMRL demonstrates superior resilience to inter-modal conflict. In the left panel, all models show degraded accuracy as conflict increases, but AEMRL degrades more gracefully. In the highconflict bin, it maintains 54.1% accuracy-an 11.8-point gain over FactorCL, which drops by over 46 points-indicating AEMRL's stronger capacity to arbitrate between contradictory signals. The right panel reveals the source of this robustness. Although all models report higher uncertainty under conflict, AEMRL's increase is more pronounced and calibrated. In the high-conflict regime, its average uncertainty rises to 0.495, accurately reflecting input ambiguity. This behavior results from the AFCAC, which detects high KLdivergence and down-weights the fused evidence, preventing overconfident errors. These results confirm AEMRL's principled and robust handling of conflicting inputs.

4.5 Robustness to Signal-Level Degradation

The results in Figure 4 validate AEMRL's adaptive mechanisms. As shown in Panel (a), the static model's performance collapses under image degradation, with AUROC dropping by over 15 points. In contrast, AEMRL degrades gracefully and maintains high AUROC even under severe blur (σ = 4), effectively ignoring the corrupted image and relying on the clean textual report. Panels (b)-(d) reveal the internal mechanisms behind this robustness. Panel (b) shows that D-EUE acts as a reliable quality sensor: as blur increases, the intrinsic uncertainty Uimg rises predictably. Panel (c) provides direct evidence of UDF's adaptation. As blur increases, the spectral norm of the unique feature projector ∥W img u ∥2 decreases, indicating that the hypernetwork learns to suppress noisy modality-specific features. Panel (d) shows that this assessment propagates to fusion: the image modality's weight γimg is progressively down-weighted. This coordinated response from uncertainty assessment to adaptive factorization to fusion is the key to AEMRL's robustness. The experiment confirms that AEMRL is not only performant on average, but also equipped with a principled, verifiable mechanism for handling signal degradation.

4.6 Robustness toMissingModalities

We evaluate zero-shot robustness to missing modalities on Drive&Act. All models are trained only on complete data, while at inference the RGB stream is randomly dropped with probability η; F1 and ECE are then measured. As shown in Figure 5(a), all methods degrade as η increases, but AEMRL remains the most stable and achieves the highest F1. At η = 80%, AEMRL drops by only 6.1 points, while MLA drops by 5.7 points from a much lower baseline and DISENTANGLEDSSL collapses by 14 points. This robustness comes from AEMRL's uncertainty-aware design, where missing modalities are treated as extreme intrinsic uncertainty and their influence is suppressed by D-EUE, UDF, and AFCAC. Thus, unreliable or absent streams contribute little to the final decision. This design avoids overfitting to complete-modality assumptions. It also enables the model to adapt to incomplete inputs without retraining. Figure 5(b) further shows that AEMRL maintains low and stable ECE, whereas baselines become poorly calibrated, with DISENTANGLEDSSL increasing by over 50%. These results confirm that AEMRL preserves both accuracy and calibration under incomplete multimodal inputs.

5 Conclusion

We introduced AEMRL, a novel framework that systematically addresses the multi-level hierarchy of uncertainty inherent in real-world multimodal data. AEMRL's core innovation is an uncertainty-driven pipeline where reliability assessment (D-EUE) dynamically controls feature factorization (UDF), followed by a conflict-aware calibration (AFCAC). This synergistic design moves beyond the limitations of prior work, which treats uncertainty and feature learning as disconnected processes. Comprehensive experiments validated our approach. AEMRL established a new state-of-the-art in both performance and calibration across five diverse benchmarks. Through a series of targeted ablations and stress tests, we provided direct empirical evidence that our framework's superior robustness stems from its principled, multi-level handling of semantic conflict, signal degradation, and missing modalities.

Acknowledgements

This research was partially supported by the National Natural Science Foundation of China (Grants No.62406303), Anhui Province Science and Technology Innovation Project (202423k09020010), Preliminary Research Pilot Project of the Chinese Academy of Sciences (XDA0490203) and Meituan.