Probabilistic Multimodal Learning with von Mises-Fisher Distributions

1 Introduction

Multimodal learning integrates consistent and complementary information from diverse data sources to enable a comprehensive perception and understanding of the real world, which is crucial for promoting the intelligence of unmanned systems. Much like humans utilize various sensory organs to perceive the world, machines process and fuse multimodal data from sensors such as cameras, radars, and ultrasonic systems to achieve holistic perception and understanding. By leveraging the complementary strengths of distinct modalities, multimodal learning has significantly advanced applications such as multimodal classification [Xu et al., 2024a; Geng et al., 2021], audio-visual recognition [Afouras et al., 2018], and multi-view clustering [Yang et al., 2022; Wen et al., 2023b]. The core of multimodal learning lies in effectively extracting and integrating consistent and complementary information from different modalities to enhance performance for various tasks.

∗Corresponding author.

To integrate various modalities, numerous multimodal learning techniques have been developed, ranging from early fusion [Yu et al., 2021], which concatenates features at the input or feature level, to late fusion [Cao et al., 2024], which integrates predictions at the decision level. Although these approaches have achieved promising results, they often face significant challenges arising frommodality imbalance [Peng et al., 2022], noise [Cao et al., 2024], and incomplete or uncertain data [Xie et al., 2023; Xu et al., 2024a]. For instance, some modalities might dominate others during fusion, causing the model to overlook weak but informative signals. In addition, noise in one modality or misalignment between modalities (a.k.a. noisy correspondence) would remarkably degrade performance. Although current efforts have introduced robust fusion mechanisms and cross-modal attention to mitigate these issues [Cao et al., 2024], the aleatoric and epistemic uncertainty [Kendall and Gal, 2017] inherent in multimodal data and model remains underexplored. The uncertainty arises from both aleatoric factors (intrinsic noise in data) and epistemic factors (limited knowledge or model capacity), posing a fundamental limitation to the reliability of existing multimodal learning systems [Gao et al., 2024].

To address these challenges, reliable multimodal learning has emerged as a critical research direction, focusing on building models capable of managing uncertainty and ensuring robustness [Geng et al., 2021; Xie et al., 2023; Xu et al., 2024a]. By explicitly quantifying and incorporating uncertainty, these methods aim to enhance trustworthiness and interpretability, enabling more reliable decision-making in safety-critical applications, such as autonomous driving and medical diagnosis [Tang et al., 2022; Zou et al., 2024]. Techniques like Bayesian learning [Kendall and Gal, 2017], ensemble methods [Lakshminarayanan et al., 2017], and uncertainty-aware attention mechanisms [Heo et al., 2018] have shown promising results, but their practical applications often encounter significant computational burdens or strong assumptions, which limit their scalability and generalizability. As a remedy, Evidential Deep Learning (EDL) [Sensoy et al., 2018; Xu et al., 2024a] offers an alternative by treating predictions as subjective opinions and directly inferring uncertainty. However, existing methods struggle to distinguish between epistemic and aleatoric uncertainty, limiting their capability to handle open scenarios. For example, noisy data would produce aleatoric uncertainty, while noisy correspondence would induce epistemic uncertainty as shown in Figure 1.

In this work, we propose a novel Probabilistic Multimodal Learning framework (PML), as shown in Figure 2, which models each multimodal data point as a von Mises-Fisher (vMF) distribution for reliable fusion. Unlike Gaussian-based models, which rely on variance to capture uncertainty, the vMF distribution represents data as directional distributions with a concentration parameter that directly quantifies reliability, thus embracing better stability. Specifically, our PML first projects each data point into a distribution described by a mean direction and a concentration parameter, enabling intrinsic uncertainty estimation within the data. We then present a von Mises-Fisher Prototypical Contrastive Learning paradigm (vMF-PL) to pull within-class samples closer to their category prototype while pushing between-class prototypes apart on the hypersphere, inherently capturing the directional discrimination and reliability in the multimodal data. By leveraging the estimated reliability, a Reliable Multimodal Fusion mechanism (RMF) is proposed to dynamically weight mean directions of different modalities for reliable classification, enhancing robustness to data noise, cross-modal misalignment, and uncertainty. Moreover, cross-entropy is utilized to evaluate the epistemic reliability between the probabilistic predictions of each modality and fusion results. In the inference stage, the learned aleatoric and epistemic reliability values are used to fuse the predictions for reliable and robust classification. Extensive experiments on nine diverse benchmarks validate the effectiveness and reliability of our PML, demonstrating its superiority over existing methods.

The major contributions of this work can be summarized below:

• We propose a novel vMF-based Probabilistic Multimodal Learning framework (PML), which captures intrinsic aleatoric and epistemic reliability to dynamically mitigate unreliable modalities, facilitating stable and robust classification.

• To help capture both directional information and reliability of each modality, we present a von Mises-Fisher Prototypical Contrastive Learning paradigm (vMFPCL), which enhances within-class compactness and between-class scatterness.
• Extensive experiments on nine widely-used multimodal benchmarks demonstrate the effectiveness and robustness of the proposed PML, outperforming 14 state-ofthe-art baselines.

2 RelatedWork

In this section, we briefly review two key areas most relevant to this paper: multimodal learning and uncertainty-aware learning.

2.1 Multimodal Learning

Multimodal learning approaches can be broadly categorized into three types: early fusion, middle fusion, and late fusion, depending on the stage at which information aggregation occurs. Early fusion methods commonly combine multimodal data at the input level [Yu et al., 2021], allowing direct interactions between modalities. However, these methods often struggle with scalability and modality-specific noise. Middle fusion methods [Natarajan et al., 2012] integrate multimodal data at the feature level by fusing intermediate representations. Unlike these methods, late fusion approaches aggregate predictions at the decision level [Cao et al., 2024], offering greater flexibility while sacrificing the exploitation of inter-modal relationships. Recent advancements focus on more sophisticated fusion techniques, such as attention mechanisms [Nagrani et al., 2021], cross-modal interactions [Chen et al., 2019], and graph-based methods [Mai et al., 2020]. These approaches aim to address challenges such as modality imbalance [Peng et al., 2022] and noisy inputs [Wen et al., 2023a; Zhang et al., 2023; Cao et al., 2024] by dynamically adjusting contributions from each modality. Despite their potential, most methods do not explicitly account for uncertainty, limiting their robustness and reliability in realworld scenarios.

2.2 Uncertainty-aware Learning

Uncertainty modeling has gained significant attention as a critical aspect of reliable AI systems, particularly in safety-critical applications. Epistemic uncertainty, arising from model limitations, and aleatoric uncertainty, stemming from intrinsic data noise, are two primary types of uncertainty [Kendall and Gal, 2017]. Various deep learning techniques have been proposed to quantify and leverage uncertainty, including Bayesian neural networks [Kendall and Gal, 2017], Monte Carlo sampling [Zhang, 2021], and ensemble learning [Lakshminarayanan et al., 2017]. Recently, in the context of multimodal learning, robust methods incorporating uncertainty have emerged, such as uncertainty-aware attention [Heo et al., 2018], probabilistic embeddings (PE) [Gao et al., 2024; Shi and Jain, 2019] and evidential deep learning (EDL) [Sensoy et al., 2018; Xu et al., 2024a]. Most existing techniques focus on modeling only one type of uncertainty, often neglecting the potential benefits of jointly quantifying both epistemic and aleatoric uncertainties. While EDL provides a direct estimation of uncertainty through subjective opinions, it struggles to disentangle the two types of uncertainties. Our work fulfills this gap by leveraging the concentration parameter of the vMF distribution to directly quantify aleatoric reliability, and the cross-entropy across probabilistic predictions to estimate epistemic reliability, thus enabling more comprehensive uncertainty modeling in multimodal learning.

3 Methodology

In this section, we introduce the proposed Probabilistic Multimodal Learning framework (PML), which leverages the von Mises-Fisher (vMF) distribution for reliable multimodal fusion and classification. Our PML consists of three key components: (1) vMF-based feature representation, (2) vMFbased Prototypical Contrastive Learning (vMF-PCL), and (3) Reliable Multimodal Fusion (RMF). Each component is elaborated below.

3.1 vMF-based Feature Representation

For ease of representation, we first give definitions for multimodal classification. Let D = {x1i, x2i, · · · , xMi, yi}Ni=1 denote the multimodal training set, where xvi represents the v-th modality of the i-th instance, yi ∈ {1, 2, · · · ,K} denotes the corresponding ground-truth label, K is the number of classes, M is the number of modalities, and N is the number of instances. Multimodal learning aims to effectively utilize information from each modality to achieve comprehensive perception and understanding. However, there is inevitable noise in the multimodal inputs, such as corrupted data and noisy data, resulting in aleatoric uncertainty.

To capture both directional representation and intrinsic aleatoric uncertainty, we model each unimodal sample xvi as a vMF distribution z ∼ Vd(µvi, κvi), where µvi and κvi denote the mean direction and concentration parameter, respectively. The larger value of κvi refers to the higher concentration of the distribution around the mean direction µvi, as well as the reliability. Specifically, given an input sample xvi from the v-th modality, two modality-specific sub-networks are exploited to estimate the mean directional representation µvi and the concentration parameter κvi as follows:

where d is the dimensionality of the latent space, hv refers to the backbone for the v-th modality, and fv(·), gv(·) are two different sub-networks used to estimate the mean direction and the concentration parameter respectively. Then, each point xvi could be modeled by a specific vMF distribution defined on a d-dimensional unit sphere Sd-1 ⊂ Rd [Li et al., 2021; Conti et al., 2022], with the following probability density function:

where Cd(κvi) is the normalization constant dependent on the dimensionality d of z, and Iν(·) is the modified Bessel function of the first kind, defined as:

The concentration parameter κvi quantifies the reliability of the sample xvi, with larger values indicating higher certainty.

3.2 vMF-based Prototypical Contrastive Learning

In the ideal scenario, a well-trained model should group within-class samples together while separating between-class samples apart, thereby enhancing the representation discriminability. In other words, each class could be ideally represented as a spherical Dirac delta distribution δ(z - ¯µvk), which is a degenerate distribution to concentrate all mass at a single point ¯µvk, i.e., the unit vector of the k-th class from the v-th modality. Here, δ(z - ¯µvk) represents the features of the k-th class from the v-th modality, serving as the k-th class prototype. Intuitively, it is reasonable to assume that the samples of the same class are distributed around their corresponding category prototype δ(z - ¯µvk) on the latent hypersphere. Let Cv = {δ(z - ¯µvk)}Kk=1 represent the prototypes of K classes for the v-th modality. Formally, for the given prototype δ(z - ¯µvk) of the k-th class on the hypersphere, the difference between a point Vd(µvi, κvi) and a prototype δ(z - ¯µvk) could be computed as follows:

Then, the probability pvij of a point xvi belonging to the j-th class could be computed as:

where σ represents the softmax function. Ideally, if the point xvi belongs to the j-th class, pvij should be maximized; otherwise, it should be minimized. Following contrastive learning principles [Chen et al., 2020], we formulate the following loss function by using the negative log-likelihood:

By minimizing this loss function, we could maximize the agreement between the data and their corresponding class prototypes in the latent hypersphere, thus encapsulating the discrimination into directional representations. Then, we could obtain the overall loss for all modalities as:

3.3 Reliable Multimodal Fusion

Multimodal fusion aims to learn m modality-specific transformations fv(·), hv(·) to project each modality Xv = {xvi}Ni=1 into feature representations Zv = {zvi}Ni=1 , which are then integrated for comprehensive decision as follows:

where ϕ(·) is a fusion function that leverages both consistent and complementary information for holistic predictions. For the early fusion, ϕ(·) is a nonlinear function, while f(·) could be a linear or nonlinear function. On the other hand, for the late fusion, f(·) is a nonlinear transformation and ϕ(·) is a linear transformation. In this paper, we focus on late fusion. To simplify the presentation, the widely-used softmax function is employed to compute the probability of xvi belonging to the j-th class, namely:

where σ is the softmax function, wvj is the projection of the j-th category for the v-th modality, W = [w·1,w·2, · · · ,w·K] is the projection matrix acting as a classifier for the concatenated representations, and w·k = [w⊤1k ,w⊤2k , · · · ,w⊤vk ]⊤. Obviously, the prototypes could be these modality-specific classifiers {wvj|v = 1, 2, · · · ,M; j = 1, 2, · · · ,K}. Then, Equation (11) could be rewritten as:

However, the naive fusion approach treats all modalities equally, making it susceptible to intrinsic uncertainties in the data [Han et al., 2022a; Xu et al., 2024a]. To address this problem, we incorporate the estimated reliability to weight the mean directional representations for more reliable predictions:

where αvi = κvi ∑Mv=1 κvi , V = [v1, v2, · · · , vK] represents the reliability weights for each modality, and vk = [µ⊤1k ,µ⊤2k , · · · ,µ⊤vk ]⊤.

Let µ∗i = ϵ [α1iµ1i, α2iµ2i, · · · , αMiµMi] ∈ Rd×M , which could also be viewed as a directional vector on a hypersphere, where ϵ is a normalization factor ensuring ∥µ∗i∥ = 1. Consequently, the fused representations are modeled as a vMF distribution Vd×M(µ∗i, κ∗i), where κ∗i = 1M ∑Mv=1 κvi is the concentration parameter of the fused vMF distribution, capturing the overall aleatoric reliability of the fused multimodal inputs. Following vMF-PCL, we could utilize the negative log-likelihood to formulate the fusion loss function as follows:

In order to estimate epistemic reliability, the entropy is employed on the probability as follows:

where ruv i measures the prediction consistency between the u-th and v-th modalities for the i-th point. When v = 0, ru∗ i denotes the prediction agreement between the v-th modality and its final decision, and when v = u, ruu i indicates the reliability of the u-th modality, particularly for the fused prediction (u = v = 0). Intuitively, since epistemic reliability should align with aleatoric reliability within the same modality, we present a reliability consistency regularizer to enhance reliability estimation:

where βvi = rvv i ∑Mv=1 rvv i represents the normalized prediction reliability for the v-th modality. Then, we could obtain the final loss of RMF as:LRMF = Lf +Lr. (17)

3.4 Optimization Objective

The overall training objective combines the vMF-PCL losses across all modalities, the fused hypersphere, and a reliability consistency regularizer. The final objective is given by:

Our proposed PML could be iteratively trained by minimizing the objective function Equation (18) in a batch-bybatch manner using one stochastic gradient descent optimization algorithm, such as Adam [Kingma and Ba, 2014]. By minimizing L, our model could effectively capture both the discriminability and reliability from the multimodal data.

During inference, the prediction consistency rv∗ i and learned reliability κvi are utilized to weigh the mean directions of each modality:

Then the obtained weights are applied to fuse the learned directional representations like [α′1iµ⊤1i , α′2iµ⊤2i , · · · , α′Miµ⊤Mi , ] for final classification according to Equation (13).

4 Experiments

In this section, we conduct comprehensive experiments to demonstrate the effectiveness of our PML across nine widelyused benchmarks and compare it with 14 state-of-the-art baselines. Besides, we provide detailed ablation studies to analyze the contributions of individual components.

4.1 Datasets and Experimental Setup

Datasets. We evaluate our framework on nine publicly available benchmark datasets spanning diverse modalities, such as image-text, audio-visual, and feature fusion tasks, to highlight the generalizability of our approach. These datasets comprise Handwritten[Duin, 1998], MSRC-V1 1 , NUSWIDE-OBJECT 2 (NUSOBJ) [Chua et al., 2009], FashionMV [Xu et al., 2022], Scene15 3 , LandUse [Yang and Newsam, 2010], Leaves100 4 , MVSA [Niu et al., 2016] and UPMC-Food101 [Wang et al., 2015]. Specifically, the Handwritten consists of six feature sets to characterize 2,000 instances of handwritten digits from "0" to "9", with each category containing 200 instances. The NUSOBJ dataset contains 30,000 instances from 31 categories with five different views. The MSRC-v1 dataset contains 210 images categorized into seven classes, each with five views. The Fashion-MV dataset is a multi-view version of the Fashion dataset [Xiao et al., 2017], designed for multi-view learning analysis, which consists of 30,000 examples of fashion products, divided into ten categories (e.g., T-shirt, Dress, Coat). The Scene15 dataset has 4,485 images categorized into 15 indoor and outdoor scene classes, wherein each image is represented using GIST, PHOG, and LBP. The LandUse dataset contains 2,100 satellite images classified into 21 categories, where each image is represented using three different feature extraction techniques. The Leaves100 dataset contains 1,600 leaf samples from 100 plant species, with three feature views extracted for each leaf image: shape descriptors, fine-scale edges, and texture histograms. The MVSA dataset is a sentiment analysis dataset that consists of more than 2,000 image-text pairs collected from social media. The UPMC-Food101 dataset is a large multimodal food dataset and consists of more than 100,000 recipes for a total of 101 categories.

Experimental setup. In our experiments, in addition to the normal experimental setting, we also construct a noise setting following [Xu et al., 2024a] to further evaluate the robustness of our method by introducing data noise and noisy correspondences (misaligned views/modalities) on test sets. We report the accuracies on the test set to measure the performance. To be further convincing, we report the mean and standard deviation of experiments conducted with 10 random seeds following the format of "mean ± std".

4.2 Implementation Details

In our experiments, the sub-networks fv and gv, which estimate the mean direction and concentration parameter, respectively, are designed as multi-layer linear networks. We exploit the Adam optimizer with a batch size of 32 to train all models, using a learning rate of 1e-4 for all datasets.

4.3 Comparisons with State of the Art

To verify the effectiveness and robustness of our method, we compare our PML against 14 baselines, including DUA-Nets [Geng et al., 2021], TMC [Han et al., 2020], ETMC [Han et al., 2022b], TMDL-OA [Liu et al., 2022], DFTMC [Han et al., 2022a], DCP-CV [Lin et al., 2022], DCP-CG [Lin et al., 2022], QMF [Zhang et al., 2023], UIMC [Xie et al., 2023], PDF [Cao et al., 2024], EAU [Gao et al., 2024], ECML [Xu et al., 2024a], TMNR [Xu et al., 2024b], and CCML [Liu et al., 2024]. As shown in Tables 1 to 3, our method consistently outperforms all baseline methods across all datasets. Non-convergent results for certain methods on specific datasets (e.g., DFTMC on NUSOBJ and Fashion-MV) are marked with '-'. More specifically, key observations from the results include:

1 https://mldta.com/dataset/msrc-v1/home/

2 https://lms.comp.nus.edu.sg/wp-content/uploads/2019/

3 https://doi.org/10.6084/m9.figshare.7007177.v1

4 https://archive.ics.uci.edu/dataset/241/one+hundred+plant+ species+leaves+data+set

1. In the normal testing setting, our PML demonstrates a
2. consistent performance advantage on all datasets, highlighting its superiority and generalizability to various modalities. For example, our method outperforms the best competitor, DCP-CG, by 6.31% on the LandUse dataset. In addition, on most datasets, our method shows small performance fluctuation, i.e., small std, indicating its promising stability.
3. From the results of noisy setting, as shown in Table 2, although most competing methods suffer remarkable performance degradation due to noisy and misaligned modalities/views (e.g., QMF and PDF), our approach maintains superior accuracy across all datasets, e.g., NUSOBJ, Scene15, LandUse, and Leaves100. More specifically, on the Leaves100 dataset, our method outperforms the strongest baseline UIMC by 9.44% far beyond expectations, which is sufficient to demonstrate the effectiveness of vMF-based uncertainty modeling and the robustness of our PML.

4.4 Ablation Study

To further analyze the contribution of each component, we conduct ablation experiments on normal and noisy test sets for key designs of PML. The experimental results are shown in Table 4. From the table, one could see that all components contribute to the performance. On the one hand, with-

5 Conclusion

In this paper, we proposed a novel Probabilistic Multimodal Learning (PML) framework based on von MisesFisher (vMF) distributions to enhance the reliability of multimodal classification. By modeling each data point with a vMF distribution, characterized by a mean direction and a concentration parameter, our method effectively encapsulates discriminative information into the directional representations while capturing the intrinsic data uncertainty. Unlike Gaussian-based models, our PML leverages the concentration parameter to directly measure data uncertainty, providing more stable and robust reliability modeling. To achieve robust multimodal fusion, we propose a Reliable Multimodal Fusion mechanism (RMF) that dynamically balances contributions from different modalities based on the predicted consistency and learned reliability. Extensive experiments on nine benchmark datasets demonstrate the effectiveness and robustness of our PML over 14 state-of-the-art approaches, in both normal and noise settings, showcasing its robustness in handling noisy and misaligned modalities. Furthermore, ablation studies validate the necessity of the key components in our framework, including our vMF-PCL and RMF. While this work focuses on the classic multimodal classification task, in the future, we plan to extend the proposed PML to broader applications such as multimodal clustering, cross-modal retrieval, etc.

Acknowledgments

This work was supported in part by the National Key R&D Program of China under Grant 2024YFB4710604; in part by NSFC under Grant 62472295, 62176171, U21B2040,

out vMF-PCL, the modality-specific information cannot be fully exploited. On the other hand, without RMF, the features of different modalities cannot be effectively and robustly integrated, both leading to inferior performance. Notably, vMFPCL has a higher contribution to the performance than RMF, indicating that modality-specific information is the basis of fusion to ensure complementary features are extracted from each modality. In summary, the full version of PML achieves the best performance in both normal and noisy settings, which shows that each proposed component plays an important role in the proposed PML.

4.5 Reliability Study

In this section, we present quantitative experiments to visually demonstrate the model's capability to estimate reliability under varying conditions. After training the model on the standard training set, we introduced different types of noise into the test set to assess whether our approach could effectively quantify the associated reliability. The experimental results are illustrated in Figure 3, covering four distinct scenarios: (1) data with normal conditions (i.e., , clean data), (2) data with additive Gaussian noise, (3) data with noisy correspondence (NC), and (4) data subjected to both Gaussian noise and noisy correspondence (Noise & NC). From Equation (2), it can be observed that the introduction of noise leads to a noticeable decline in the model's estimated reliability. Specifically, the model demonstrates high confidence under normal conditions (yellow region), while the reliability estimation deteriorates as noise is added. In the case of Gaussian noise (blue region), the reliability scores exhibit moderate degradation, reflecting our model could correctly capture the uncertainty. Similarly, when noisy correspondence is introduced, the model identifies the inherent uncertainty stemming from incorrect cross-modal relationships, resulting in a more significant reduction in the estimated reliability. The most pronounced impact is observed when both noise sources are present, where the reliability is substantially diminished, indicating that our method successfully captures compounding uncertainties. These findings underscore the effectiveness of our proposed approach in accurately quantifying reliability in the presence of various noise types. The results not only validate the model's ability to differentiate between clean and noisy data but also demonstrate its potential to serve as a trustworthy mechanism for real-world applications where data uncertainty is prevalent.

624B2099 and 623B2075; in part by Sichuan Science and Technology Planning Project under Grant 2024NSFTD0047 and 2024NSFTD0038; in part by System of Systems and Artificial Intelligence Laboratory pioneer fund grant; and in part by the Fundamental Research Funds for the Central Universities under Grant CJ202403 and CJ202303.