DFMU: Distribution-based Framework for Modeling Aleatoric Uncertainty in Multimodal Sentiment Analysis

1 Introduction

MSA is a pivotal research task that seeks to comprehend individuals' sentiment states and analyze multimodal information present in online video data, to predict the intensity of expressed sentiment. This capability is central in diverse realworld applications, such as affective intelligence [Li et al., 2023a] and human-computer interaction [Jiang et al., 2020].

However, multimodal signals inevitably contain redundant and noisy information, which causes unavoidable Aleatoric Uncertainty(AU). According to previous studies [Kendall and Gal, 2017][Gawlikowski et al., 2023], AU in deep learning reflects the inherent uncertainty issues within the data, which are related to data noise. This uncertainty significantly affects model performance and makes it difficult for the model to achieve reliability. Most current work in MSA overlooks the inherent AU problem. A few works that consider AU primarily focus on modeling noise rather than implementing targeted modeling based on its origin. There is a lack of further exploration into the sources of AU in the MSA field, with understanding limited to the presence of noise in the data. As a result, previous models that have attempted to model uncertainty in MSA struggle to genuinely mitigate the impact of AU, leading to continued limitations in model performance. According to experiments, this work finds that in the MSA field, the generation of AU primarily stems from two problems: subjective biases in the annotation of MSA datasets and the complex set relationships of sentiment features.

† Corresponding author.

As shown in Figure 1, some samples in the dataset express sentiment with ambiguity, increasing the difficulty of annotation. Additionally, the subjectivity of the annotators leads to inconsistent label values for the same sample. The degree of inconsistency reflects the bias of the samples, referred to as subjective annotation bias. Samples with high subjective annotation bias contain greater AU. Moreover, in Figure 1, points represent correlations solely through distances in the representation space, without expressing their inclusion relationships. For instance, in distribution representations, the sentiment features of joy and pride are almost inclusive, while they do not intersect with sadness. Therefore, there exist relationships between sentiment features that cannot be expressed through deterministic point distances, termed complex set relationships of sentiment features. Our model needs to specifically address this complex relationship.

In response to the discovered source of AU generation in MSA, we designed a specialized model to achieve better dataset performance, and we obtained improved results in AU-related noise resistance experiments. To address Problem 1, we designed an uncertainty modeling block that utilizes Gaussian distributions to encode uncertain distribution representations. These representations can store AU information of samples with high subjective annotation bias in the variance dimension of the distribution. Subsequently, the proposed uncertainty regularization can adaptively adjust the optimization objectives based on the obtained uncertainty information. This regularization enables the model to learn variance during the training process, ultimately displaying AU through information from the variance dimension. To tackle Problem 2, we proposed a distribution-based contrastive learning with sentiment words replacement to model the complex set relationships of sentiment features. We employed a sentiment dictionary data augmentation method to enhance the fine-grained sentiment-capturing capability and then used distribution-based contrastive learning to endow DFMU with the ability to model complex set relationships.

The main contributions of this work can be summarized as follows:

• This paper presents DFMU, which reveals through empirical research that the source of AU generation in the MSA lies in the issues of subjective bias in MSA dataset annotations and the complex set relationships of sentiment features.
• This paper introduces an uncertainty modeling block and uncertainty regularization, modeling sentiment features as uncertain distribution representations to enable DFMU to quantify the subjective annotation bias.
• This paper proposes distribution-based contrastive learning with sentiment words replacement, utilizing distribution contrastive loss and sentiment dictionary data augmentation methods to model complex set relationships of sentiment features.
• DFMU achieves state-of-the-art performance across multiple widely adopted datasets. This work provides analysis of uncertainty issues and offers comprehensive

empirical results to demonstrate the effectiveness and necessity of DFMU.

2 RelatedWork

In this section, we review relevant research in the areas of MSA uncertainty modeling, and contrastive learning.

2.1 Multimodal Sentiment Analysis

Previous work primarily relied on deterministic approaches to model sentiment representations, often employing contrastive learning [Yu et al., 2023]or complex fusion methods [Zhang et al., 2023] to address noise in modality information. Some works [Liang et al., 2023] attributed the noise to redundancy and conflicting information across modalities, while [Hazarika et al., 2020] suggested projecting each modality into modality-invariant and modality-specific representations to model the noise. However, these approaches overlooked the issue of uncertainty in the noise that cannot be resolved through deterministic point representations.

2.2 UncertaintyModeling

Recently, the MSA field has gradually begun to consider the impact of AU on prediction. AU refers to the inherent noise in the provided training data. TMSON [Xie et al., 2024] proposed to use Trustworthy Multimodal Fusion to estimate the reliability of each multimodality. EAU [Gao et al., 2024] believes that each modality in a multimodal task has separate noise, and proposes to use a novel generic and robust multimodal fusion strategy to better model unimodal noise. TMSC [Xu et al., 2024] constructs the modal private task (unique) by using the Dirichlet distribution and evidence theory to solve the uncertainty of each modal noise. However, previous methods have stopped at linking AU with noise, without exploring the source of AU in the MSA field, which prevents them from truly modeling the complete AU.

2.3 Contrastive Learning

Contrastive learning aims to learn effective representations by bringing positive samples closer together and pushing negative samples farther apart in embedding space. In multimodal tasks, HyCon [Mai et al., 2023] explored relationships between samples and classes within and across modalities, while ConFEDE [Yang et al., 2023] and ConKI [Yu et al., 2023] process each modality into two different features to explore similarities and differences. Previous works treated sentiment features as deterministic points in the feature space, failing to capture their complex set relationships.

3 Method

The overall processing flow of DFMU for handling uncertainty in MSA tasks is illustrated in Figure 2. As shown in the figure, DFMU primarily consists of three components: Uncertainty Modeling Block, Distribution-based Contrastive Learning, and Uncertainty Regularization.

3.1 UncertaintyModeling Block

To encode uncertain distribution representations, DFMU utilizes Gaussian distributions N (µ, σ2 ) to represent the sentiment features of different modalities in the sentiment distribution space. We assume Gaussian distributions to model our uncertain representations and learn mappings for mean and covariance vectors. As shown in Figure 2, the Uncertainty Modeling Block (UMB) is proposed to better model the single-modality data relationships by using multi-head selfattention. Let E ∈ {V,A, T} be a matrix in RL×D , where V,A, T represent the initial visual, audio, and textual embeddings, respectively. Here L is the sequence length and D is the embedding size. E is sent to an Feed Forward layers(FFD) and two pathways, µ and σ2 . Then it is split into k heads. Within each head, the input hidden state of each path is E(i) ∈ RL×D/2k in i-th head. After E(i) processing in corresponding Self-Attention heads, the k outputs are concatenated and then passed through an additional selfattention(SA) layer and an FFD to model the relationships across the sequences. The σ2 path is similar to the µ path. Since the input point representations are related to the mean of the distributional representations, a residual network is employed to learn the mean vector. Formally, the operation in the µ path is:

3.2 Distribution-Based Contrastive Learning

The Distribution-Based Contrastive Learning module is proposed to model the complex set relationships of features.

To measure the distance between multivariate Gaussian distributions, the 2-Wasserstein distance [Mallasto and Feragen, 2017] is employed, different modality embeddings E ∈ {T, A, V }, as an example:

where Ei,Ej represent the samples of corresponding modality embeddings. µEi and σ2Ei are the mean and variance values obtained from Ei after UMB processing.

Sentiment Words Replacement

To further enhance the model's learning of textual sentiment distribution, we design a distribution-based text augmentation for contrastive learning: Sentiment Words Replacement (SWR) method, which is shown in Figure 3. Unlike random words processing, we first explore the Sentiment scores of each word in the sentence by using SentiWordNet [Baccianella et al., 2010], where higher scores represent stronger sentiment. The words with high sentiment scores are chosen for a series of augmentations. Specifically, two types of text augmentations are constructed: positive sentences and negative sentences. Each type of augmentation operates on kt% of words in the sentence. Positive sentences are constructed by replacing words with synonyms and negative sentences are constructed by replacing words with antonyms.

In each training iteration, DFMU feeds the batches of original sentences, positive sentences, and negative sentences into the Feature Extractor, obtaining corresponding sets of text embeddings, which denoted as {T 1 , ..., T b }, {T 1 p , ..., T b p }, and {T 1 n , ..., T bn }, respectively. Here, b is batchsize, T represents the text embedding of an original sentence, Tp and Tn represent the text embedding of a positive sentence and negative sentence respectively.

The SWR contrastive learning loss is defined as LSWR c , using InfoNCE [van den Oord et al., 2018] and r represents either positive sentences or negative sentences:

where r ∈{p,n}.

Intra-Modal and Cross-Modal Contrastive Learning

If the sentiment scores of two samples are close, we consider their intended sentiment intents to be similar, and uncertain distribution representations should be close. Specifically, positive pairs are defined as samples whose absolute difference in sentiment scores is less than a threshold value ke and negative pairs are on the contrary:

where, P and N represent positive pairs and negative pairs respectively. Ei,E′j represent the samples of corresponding modality embeddings, Z represents the sentiment score of corresponding sample, which can be acquired from datasets. The contrastive learning loss for both Intra-modal and Cross-moda Contrastive Learning is defined as L, where (Ep,E′q ), (Es,E′t ) represent the corresponding combination in P or N:

For Intra-modal Learning, we compare samples in the same modality of data to model intra-modal information. Based on Equation 5, 6, it requires E = E′ , that is, the two samples' modalities in combinations in P and N are the same. Then uses Equation 7 to compute Intra-modal loss Lintra c .

For Cross-modal Contrastive Learning, we compare samples in the different modalities of data to learn the relationships between different modalities and contribute to model cross-modal information. Based on Equation 5, 6, it requires E ̸= E′ , that is, the two samples' modalities in combinations in P and N are different. Then uses Equation 7 to compute Cross-modal loss Lcross c .

3.3 Uncertainty Regularization

To learn the distribution of the representation's mean and variance, the original Mean Square Error(MSE) regression loss is modified to prevent the variance from collapsing to zero or exploding to extremely large values. In our experiments, we discovered that replacing the MSE loss entirely with the regularization loss made the model difficult to converge. Therefore, a hyperparameter α is proposed to balance the MSE loss and the regularization loss. The Lmse is Mean squared error loss, and the Lreg is regularization loss:

To simulate the discrepancies among n annotators during data annotating, we designed the method to perform multi-point sampling within the distribution, as shown in Figure 1 (b) and Figure 2. Each sampled point is individually predicted and takes the average of the multiple predictions. Subsequent ablation experiments confirmed that this approach helps model the subjective biases that may arise during data annotation. It is worth noting that the sampling operation of the distribution poses challenges for backpropagation. By applying the reparameterization trick [Kingma and Welling, 2014], the model firstly samples random noise from the standard normal distribution instead of directly sampling from the distribution :

Following the equation, the output z follows the predictive distribution of the partial differential equation. Hence, the computation of mean and standard deviation from the sampling operation can be separated, and they are trainable.

The final loss for all the tasks is:

where αSWR , αcross, αintra are hyperparameters of losses, n is the number of samples, and Lireg is the regularization loss of the i-th sample.

4 Experiment

4.1 Experimental Settings

The experiments were conducted on three publicly available benchmark datasets in MSA: CMU-MOSI [Zadeh et al., 2016], CMU-MOSEI [Zadeh et al., 2018] and CHSIMS [Yu et al., 2020].

For fairness, we employ the same feature extractors adopted in ALMT [Zhang et al., 2023] for vision, audio, text and also retrain all baselines under the same conditions. All the experiment records are aggregated over three independent runs.

Experimental results are reported in two forms: regression and classification. For regression, MAE (mean absolute error) and Corr (Pearson correlation) are reported. For classification, Acc-2 (binary classification accuracy), Acc7 (seven-class classification accuracy) and F1 score are reported. Higher values indicate better performance for all metrics except for MAE.

4.2 Baselines

We compare DFMU with the following state-of-the-art baseline models in MSA: based on contrastive learning methods: ConKI [Yu et al., 2023], ConFEDE [Yang et al., 2023], Hycon [Mai et al., 2023]; uncertainty modeling models: TMSON [Xie et al., 2024], EAU [Gao et al., 2024], TMSC [Xu et al., 2024] ; deterministic modeling models: MMIM [Han et al., 2021], MISA [Hazarika et al., 2020], ALMT [Zhang et al., 2023], Self-MM [Yu et al., 2021].

4.3 Performance Comparison

Table 1 and Table 2 list the comparison results of our proposed and state-of-the-art methods on the MOSI, MOSEI, and CH-SIMS, respectively. It can be observed from these tables that DFMU yields better results to a range of baseline models on all datasets.

It was noteworthy that DFMU was the only model to exceed an F1 score of 87 on both the MOSI and MOSEI datasets. On MOSEI, it surpassed the second-best model, ConKI, by a significant margin of 1.15 points, which showed that DFMU can accurately model uncertainty and demonstrate superior performance in fitting high subjective annotation bias samples. Furthermore, the Acc-7 metric of the DFMU on the MOSI exceeded the performance of the best baseline model by 0.79, though the task was particularly a challenging seven-class classification problem, within the context of a regression task. The superiority of DFMU in Acc-7 metric demonstrated that it was excellent performance in predicting labels across the entire range.

4.4 Ablation Study

An ablation study about modalities was conducted first, as shown in Table 3. This finding validated that the text modality plays a dominant role in the MSA. In addition, a series of ablation experiments on subtasks of our proposed distributionbased contrastive learning were conducted. When the three contrastive learning subtasks were removed, the Acc-7 metric of the model decreased by 3.38, 1.85, and 1.56, which showed that among the three sub-tasks, the SWR task showed the highest improvement in performance.

Finally, a series of ablation experiments were conducted on our proposed Uncertainty Regularizaion, Uncertainty Modeling Block and Distribution-based Contrastive Learning. We found that the absence of Uncertainty Regularization and the Uncertainty Modeling Block resulted in a decrease of 3.09

and 4.62, respectively, in the model's Acc-7 scores. Moreover, without the Distribution-based Contrastive Learning, the performance suffered the most, which led to a substantial 5.42 reduction in the Acc-7 metric. The Distribution-based Contrastive Learning helped our model learn better representations of sentiment distributions.

4.5 Hyperparameters of Contrastive Learning

To study the impact of hyperparameters of different uncertainty contrastive learning methods in more detail, we conducted the ablation experiments of three hyperparameters in MOSI. According to Table 4, we found that the cross-modal contrastive learning and Intra-modal Learning methods had little impact on the results, and were insensitive to hyperparameters. On the contrary, sentiment words replacement contrastive learning method was very sensitive to hyperparameters. The value of Acc-7 metric dropped 37.52% when αSWR changed from 0.05 to 0.3. We speculated that this may be because the loss value of this method is very large, which caused the situation.

4.6 ComparisonWith Large Models

To prove the effectiveness of DFMU, we compared it to the pre-trained models and the multimodal large language model, as shown in Table 5. Our model remained ahead in performance although the number of parameters and training data was much smaller than that of large models. It was notable that DFMU exceeded GPT-4V by 7.23 in CMU-MOSI. The weak performance of GPT-4V can be partly attributed to the fact that GPT-4V does not support audio input and the loss of information. The performance of DFMU without audio modality still exceeded GPT-4V in CMU-MOSI by 6.02. The result proved that the necessity of the ability to model the uncertainty and the damage of AU to performance.

4.7 MSAUncertainty Problems Analysis

Subjective Annotation Bias

To verify one of the proposed sources of AU: subjective annotation bias, we used the model to output the AU values corresponding to each sample in MOSI and visualized them. As in Figure 4, we selected representative samples for analysis:

• In sample 1, a person had a consistently highly positive facial expression and the corresponding text. This is the easiest scenario for annotation.

• In sample 2, the person's facial expression remained neutral, but the text was obviously negative. Annotators will disagree on how to annotate the sentiment score when text and facial expressions are inconsistent, causing high sample uncertainty.
• In sample 3, the person's expression switched between frowning and expressionless, and the text is neutral, which caused complex sentiments that are difficult to annotate, causing high uncertainty.
• In sample 4, the person had a highly positive facial expression but with neutral text. Compared to Sample 1, the difficulty of annotation increases, and uncertainty also rises.

Meanwhile, a trend line was plotted in the scatter plot, the polynomial fitting function is used here to calculate the trend line. The line showed a clear negative correlation between the sentiment score and the AU. This is because samples with lower absolute label values are more likely to express complex or ambiguous sentiment, increasing the difficulty of labeling and resulting in subjective annotation bias, which in turn generates AU.

Comparison in High Uncertainty

We added different degrees of noise to the MOSI test data to simulate uncertainty. Specifically, we extract noise vectors from a Gaussian distribution and then multiply these noise vectors by a scaling factor to contaminate the test data, simulating the AUs present in the data. We compared the performance of DFMU and the baselines in the test data and visualized the results as line graphs in Figure 5. By comparing the results, we observed that the performance of all baselines decreased significantly as uncertainty increased. When the scaling factor was set to 9, our model's score only decreased by 7.85%, while the score of ALMT, which cannot model AU, dropped by 28.09%.

Complex Set Relationships of Sentiment

To verify one of the sources of AU: complex set relationships of sentiment, we conducted a visual analysis of the distribution representations of sentiment features in the model. As shown in Figure 6, In the left diagram, samples A B C D preserved the fine-grained set relationships among the features, where samples A B C intersected with one another but do not overlap with D. However, in the right diagram, all relationships degenerated into mere distance relationships, indicating that point representations lost more information compared to distribution representations, particularly for samples with similar label values. This indicated that the loss of sentiment information in point representations also contributed to AU.

5 Conclusion

In this work, we explore two causes for the emergence of Aleatoric Uncertainty in MSA: subjective bias in the annotation process and the complex relationships among sentiment features. To address these issues, we propose DFMU, a Distribution-based Framework for Modeling Aleatoric Uncertainty, which includes an uncertainty modeling module and distribution-based contrastive learning with sentiment word replacement. Our proposed model demonstrates robust performance even under high noise conditions, achieving state-of-the-art results on various popular datasets.

Acknowledgments

This work was supported by the National Natural Science Foundation of China grant under number 62222603, National Training Program of Innovation and Entrepreneurship for Undergraduates (202510561094).

Contribution Statement

The authors of this paper, Chen Tang and Tingrui Shen, have made equal contributions to the paper.