Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities

1 Introduction

The objective of sentiment analysis is to identify human sentiments during communication, which provides a foundation for understanding behaviors and social interactions [Liu, 2022]. Sentiments are expressed through textual content, vocal intonation, and body language, making multimodal sentiment analysis necessary for capturing emotional dynamics more comprehensively [Poria et al., 2017]. Integrating textual, acoustic, and visual information enables richer affective representation and supports broader human-computer interaction applications [Zadeh et al., 2018b].

∗Corresponding author.

Specifically, in real-world deployments, multimodal observations are often missing or unreliable due to environmental conditions, occlusions, and system failures [Wu et al., 2024;

Reza et al., 2024]. Such unreliability may arise from background speech, motion blur, illumination variation, occlusion, or sensor failure, which commonly lead to unstable representations and biased fusion decisions. These situations introduce inconsistent cues and make it difficult to obtain accurate sentiment representations [Yu et al., 2021]. Under modality missing scenarios, many methods generate missing-modality features from available modalities to preserve the multimodal prediction structure [Guo et al., 2024]. However, due to differences in expression mechanisms and sentiment dynamics, generated features may deviate from the true feature space and incur generation bias, as illustrated in Figure 1(A) [Dai et al., 2025], further misleading fusion and drifting predictions toward incorrect semantic regions [Liu et al., 2024].

Even when all modalities are available, multimodal systems still face quality imbalance. Modalities affected by noise, occlusion, or abnormal expressions may be treated as reliable signals during fusion [Arevalo et al., 2017; Mao et al., 2023]. Although such corruptions differ at the input level, they commonly reduce modality reliability and introduce inconsistent sentiment evidence. As shown in Figure 1(B), unreliable sources may be over-weighted in the fused representation, pushing predictions away from the correct sentiment target and causing fusion bias [Baltruˇsaitis et al., 2018].

To address generation bias and fusion bias, we introduce a two-level reference alignment (TLRA) framework that provides controllable cross-modal references at both the feature and sentiment decision levels, as illustrated in Figure 1(C). TLRA uses complete-modality information as stable references to guide feature alignment under modality missing, thereby reducing the deviation of generated or completed features from the true sentiment space. It further models crossmodal consistency at the decision level through referencedriven aggregation, suppressing the influence of unreliable modalities on final prediction. Rather than designing noisespecific correction modules, TLRA treats missing and unreliable modalities as a unified reliability degradation problem and constrains their effects through reference alignment.

The contributions of this work are:

• A TLRA framework that introduces cross-modal references at both feature and sentiment decision levels to address generation bias under modality missing and fusion bias caused by unreliable modalities.
• A reference-constrained feature alignment strategy that leverages complete-modality samples as stable semantic anchors to reduce generation bias and stabilize representations under modality missing.
• A reference-driven decision alignment strategy that models sentiment-level cross-modal consistency with prototype-guided references and reduces the influence of potentially unreliable modalities during fusion.
• Extensive experiments on CMU-MOSI and CMUMOSEI demonstrate that the proposed method outperforms representative methods under various missingmodality scenarios and achieves state-of-the-art performance under complete modalities. 1

1 https://github.com/LiuXY3366/TLRA

2 RelatedWorks

2.1 Sentiment Analysis

Early multimodal sentiment analysis methods commonly assume complete and well-aligned modalities, improving affective modeling through cross-modal attention, feature interaction, and temporal fusion [Liu et al., 2018; Tsai et al., 2019; Yu et al., 2024]. However, real-world modalities can be degraded by noise or missing values, making these methods sensitive to modality stability [Baltruˇsaitis et al., 2018; Zeng et al., 2022; Li et al., 2024]. Existing studies address this issue using cross-modal generation, text-dominant recovery, or robustness training [Neverova et al., 2015; Pham et al., 2019; Lin and Hu, 2023]. Nevertheless, because modalities differ in expression mechanisms and temporal dynamics, recovered features may not align with the true sentiment structure and can produce unstable or conflicting fused representations [Lin and Hu, 2023; Zeng et al., 2022]. Unlike generation-based recovery, our method introduces stable cross-modal references to constrain sentiment representations under missing or quality-imbalanced modality conditions.

2.2 Multimodal Fusion

In multimodal sentiment analysis, fusion performance is strongly affected by modality quality, expression stability, and signal conditions [Liu, 2022]. Prior studies improve robust fusion by estimating modality confidence, adaptively weighting modality contributions, or imposing consistency objectives to reduce uncertainty-induced drift [Arevalo et al., 2017; Zadeh et al., 2018a; Wang et al., 2019; Xie et al., 2024; Han et al., 2021; Hazarika et al., 2020]. In contrast, this work introduces prototype-based decision references to align modality-wise predictions and enhance cross-modal consistency when modality quality varies in practical scenarios.

3 Method

3.1 Overview

To mitigate both generation bias from modality missing and fusion bias from quality imbalance, we develop the TwoLevel Reference Alignment framework (TLRA). TLRA defines cross-modal references at both the representation and decision levels, allowing the model to learn consistent and robust sentiment representations even when only partial or unreliable multimodal inputs are available.

As illustrated in Figure 2, TLRA consists of three key components: modality encoding, representation-level alignment, and decision-level alignment. For textual, acoustic, and visual modalities, TLRA independently encodes available modality signals while constructing sentiment reference structures from complete-modality training samples. The representation-level alignment stage constrains incomplete features into a shared sentiment space to reduce generationinduced representation drift. The decision-level alignment stage further enforces cross-modal consistency by suppressing unreliable modality decisions during fusion, thereby reducing fusion bias and improving prediction stability.

̂

3.2 Modality Encoding

In the modality encoding stage, we independently model the raw signals of each modality based on the presence of potentially missing modalities, creating a structured representation for each modality to serve as input for future alignment. Figure 2(A) depicts that during the modality encoding stage, the TLRA framework only encodes the available modalities (i.e., does not complete or reconstruct missing modalities) at this stage of the TLRA framework, which allows us to perform the alignment step under true conditions of modality missing.

For the textual, acoustic, and visual modalities, we utilize the BERT pretrained language model, the Librosa acoustic feature extractor, and the OpenFace visual analysis system, respectively, to encode the modalities [Devlin et al., 2019; McFee et al., 2015; Baltruˇsaitis et al., 2016]. A multimodal input will result in three modality representations: Hl, Ha, and Hv for the text, audio, and visual modalities, respectively.

During training, complete-modality samples are used as supervision references for constructing reference structures. When one or more modalities are missing, TLRA operates only on available modalities without encoding or reconstructing missing inputs. This design allows the model to distinguish missing information from available evidence and avoids introducing unreliable assumptions at the input level.

3.3 Representation-Level Alignment

Imputed features may deviate from the actual sentiment structure and cause incorrect predictions, mainly due to the lack of stable semantic references for anchoring incomplete inputs. TLRA addresses this issue through referencedriven representation-level alignment, which constrains imputed representations within the semantic space defined by complete modalities, as shown in Figure 2(B).

Dual-Path Encoding Strategy. TLRA uses a dual-path encoding strategy to handle both complete-modality and modality-missing samples. The complete-modality path establishes a stable sentiment space from complete inputs, while the modality-missing path serves as a query and completion path for incomplete inputs. By decoupling reference construction from query-based completion, the model can align missing-modality representations within the same sentiment space without assuming that all modalities are available.

Given modality representations Hm from the encoding stage, where m ∈ M denotes available modalities, the Modal Encoder maps each modality into a shared sentiment space:

The resulting features Fm exhibit stable sentiment structure and support subsequent decision-level alignment.

To support representation completion under modality missing, a Query Encoder is introduced to construct query features for reference retrieval. Each available modality is first projected into a unified representation space via a lightweight 1D convolution, and then processed by the Query Encoder:

All modalities pass through the query path regardless of completeness, producing features that support both completion and alignment. By decoupling reference construction from query-based alignment in a shared semantic space, the dual-path design enables robust alignment without assuming complete inputs. The two-path encoding is implemented with shared lightweight modules to ensure efficiency and stability. Template-Guided Completion. Under modality missing, unconstrained feature generation can produce biased representations deviating from true sentiment structures. To address this, TLRA introduces a template-guided completion mechanism that provides stable semantic references for missing modalities and constrains completion within the sentiment space defined by complete modalities.

Given query features ˜ Fm from the Query Encoding, we apply temporal pooling to obtain a global query vector qm:

TLRA maintains a learnable template library T = {t1, t2, . . . , tK} in the shared sentiment space. The similarity between qm and each template is computed as:

̃

̃

̃

̃

̃

̂

̂

The similarities are normalized via softmax to obtain aggregation weights αk, yielding the template context as:

To adaptively fuse the sample-specific query and the template reference, we compute a consistency weight based on their cosine similarity as:

The completed representation is then obtained as:

The completed representation ˆ Fm serves as sentimentlevel structural compensation rather than reconstruction of the original modality signal. Therefore, the completion process focuses on recovering stable sentiment structure instead of synthesizing raw modality-specific details, which reduces the risk of introducing modality-specific generation noise. By constraining completed features within the shared sentiment space, template-guided completion effectively suppresses generation bias under modality missing and provides stable inputs for alignment and fusion.

Representation-Level Alignment Loss. The objective of representation-level alignment is to enforce structural consistency among intermediate representations within a unified sentiment space under modality missing. Specifically, both query features for observed modalities and completed features for missing modalities are aligned to the modality representations defined under complete inputs.

Let M denote the set of available modalities and Mmiss the set of missing modalities. We define Ω = M∪Mmiss as the set of modalities involved in alignment. Let Fm denote the modality feature produced by the modal encoder, ˜ Fm the query feature, and ˆ Fm the completed feature. The unified representation alignment loss is defined as:

By aligning query features and completed missingmodality features to complete-modality references, this loss suppresses representation drift caused by missing inputs. As a result, different modality combinations are encouraged to follow a consistent sentiment structure, providing more reliable inputs for decision-level alignment.

3.4 Decision-Level Alignment

Although representation-level alignment reduces missinginduced drift, decision inconsistencies may still occur when noisy modalities produce misleading predictions. These unreliable modality-wise predictions can dominate fusion and degrade the final sentiment decision. As shown in Figure 2(C), TLRA filters and aligns modality-wise predictions to suppress unreliable evidence under varying modality conditions and improve decision robustness.

Prototype Construction. For each sentiment class c, we maintain modality-specific prototypes Bcm located within the shared sentiment space for the text, audio, and visual modalities. These prototypes serve as decision-level references and are constructed based upon accumulated training features, capturing stable class structural elements across modalities. In practice, the prototype bank is randomly initialized and then updated online during training, so no extra clustering procedure or external prior is required.

Given the aligned feature Fm of the sample with class label c, we determine the cosine similarity with the associated prototype Bcm, which is utilized to create a mixed prototype:

where αm is determined by the cosine similarity between Fm and Bcm, adaptively balancing sample-specific information and class-level structure to improve robustness.

The mixed prototype B′cm enables gradual updates of the prototype memory with controlled momentum, allowing prototypes to evolve into stable class-level references while remaining robust to sample variations, as illustrated in Figure 3(A). In the binary sentiment setting, each modality maintains one positive and one negative prototype, keeping the memory compact and interpretable while providing explicit class-level anchors. This class-wise design avoids additional prototype-number search and ensures that decision references are directly aligned with sentiment labels. During training, we adopt momentum-based updates with normalization to stabilize cosine similarity, enabling prototypes to progressively evolve from coarse initial references into reliable and discriminative sentiment anchors in practice.

Through this process, class prototypes act as decision-level anchors that provide consistent references across samples and modalities. As the prototype memory is continuously updated during training, it gradually captures stable class-level sentiment structures and helps resolve cross-modal prediction conflicts under noisy and unreliable modality conditions.

Two-Stage Decision-Level Alignment. Since prototype memory gradually evolves from coarse class representations into reliable references, TLRA adopts a two-stage decisionlevel alignment strategy that progressively strengthens crossmodal consistency, as shown in Figure 2(C).

Stage1: Soft Prototype Guidance. At early epochs, prototype memory is not yet stable enough to support hard voting, and strong cross-modal constraints may amplify modality noise. Therefore, prototypes provide continuous sentiment references without explicit voting or hard class decisions. This soft guidance preserves useful modality-specific information while gradually pulling unstable features toward smoother and more reliable prototype structures.

Given the aligned modality feature Fm for each available modality m ∈ M, the model retrieves the corresponding positive and negative prototypes BPm and BNm from the prototype memory and computes their cosine similarities:

The prototype with higher similarity is selected as the pri-

̂

̂

mary sentiment reference for the current modality:

and the maximum similarity is used as a consistency score:

Based on this score, the modality feature is fused with the selected prototype to form a mixed representation F ′m:

This soft guidance preserves reliable modality information while pulling unstable modalities toward smoother prototype structures. The resulting modality representations are then fused to produce the final sentiment prediction.

Stage2: Prototype Voting and Suppression. As training progresses, the prototype memory gradually becomes compact and discriminative. In the second stage, TLRA introduces an explicit prototype voting mechanism to enforce cross-modal decision consistency, as shown in Figure 2(C). Each modality produces a prototype-space decision as:

The modality-wise decisions vm form a voting process, where the majority vote defines the high-confidence sentiment direction. Disagreeing modalities are suppressed and replaced with prototype features consistent with the voted sentiment, preventing unreliable modalities from dominating fusion. The voting mechanism is activated only in the second

̂

̂

stage, allowing the first stage to stabilize prototype formation

before stronger decision-level consistency is enforced. Decision-Level Alignment Loss. Decision-level alignment aims to construct a discriminative and cross-modally consistent sentiment prototype space, and to constrain multimodal fusion decisions to align with ground-truth sentiment labels. To this end, TLRA introduces three loss terms at the decision level: two for prototype structure learning and one for final sentiment prediction to further enhance robustness under modality missing and quality imbalance.

First, to ensure intrinsic discriminability within each modality, the positive (P) and negative (N) sentiment prototypes should be well separated in the prototype space. Let BPm and BNm denote the positive and negative prototypes of modality m. We define an intra-modality separation loss as:

where δ is a margin threshold enforcing clear class boundaries within each modality to enhance separability.

Second, to promote cross-modal consistency, prototypes corresponding to the same sentiment across different modalities are encouraged to align in the sentiment space. The crossmodal prototype alignment loss is defined as:

After prototype construction and alignment, the final sentiment prediction is obtained through the fusion module. A standard task supervision loss is applied:

where ˆ y and y denote the predicted and ground-truth sentiment labels, respectively, for each input sample. The overall decision-level optimization objective is:

Together, these losses enforce intra-modality discriminability, cross-modal consistency, and prediction correctness, enabling TLRA to produce robust sentiment predictions under modality quality imbalance. The loss weighting coefficients are selected via controlled validation, and the model remains stable across a reasonable range of settings, indicating that the performance gain primarily arises from the proposed representation- and decision-level alignment mechanisms rather than sensitive hyperparameter tuning.

4 Experiments

4.1 Experimental Setup, Datasets, andMetrics

Experimental Setup. We trained our model using PyTorch and 1 NVIDIA RTX 4090 GPU. Each modality encoder is initialized with a pretrained model, and alignment and fusion modules were learned end-to-end. Using AdamW with a learning rate of 1×10-4 and a cosine annealing schedule, the batch size is 16 and was trained over 100 epochs. All hyperparameters were kept constant across all experiments, where

λ1 = λ2 = 1.0, λ3 = λ4 = 0.7, and λ5 = 0.2. These coefficients are selected by one-factor-at-a-time validation to favor settings that remain stable across modality combinations rather than relying on exhaustive search. The additional cost of TLRA mainly comes from template retrieval and prototype matching, which scale linearly with the number of templates and prototypes. Since TLRA does not perform input-level modality reconstruction, the dominant computation remains in modality encoding, and the alignment modules introduce only limited overhead. This indicates that TLRA improves robustness without relying on expensive generation or reconstruction modules, making it more practical for deployment under resource-constrained settings.

Datasets. We assessed our methods against two publicly available benchmarks for multimodal sentiment analysis, CMU-MOSI [Zadeh et al., 2016] and CMU-MOSEI [Zadeh et al., 2018c]. For both datasets the videos and modal data were aligned; CMU-MOSI consists of 2199 utterances from 93 video clips, while CMU-MOSEI is a larger dataset containing 23,453 utterances from over 1000 speakers. With aligned multimodal annotations, we were able to perform end-to-end fusion learning with these datasets, as well as evaluate our algorithms under missing-modality settings.

Metrics. The performance metrics for both datasets were assessed using ACC and F1-score (binary sentiment), where samples were assigned to the negative class or the nonnegative class based on standard thresholding of sentiment scores. ACC measures overall classification accuracy and F1score complements ACC by balancing precision and recall.

4.2 Quantitative Evaluations

Missing-Modality Comparison. Table 1 shows a detailed quantitative comparison between our approach under missing-modality scenarios on both CMU-MOSI and CMUMOSEI datasets, in addition to comparing other representative approaches for multimodal sentiment analysis under similar conditions of missing modalities, such as CENET [Wang et al., 2022], TETFN [Wang et al., 2023a], DMD [Li et al., 2023], TFR-Net [Yuan et al., 2021], ALMT [Zhang et al., 2023], LNLN [Zhang et al., 2024], UEFN [Wang et al., 2025c], U-PNS [Chen et al., 2025], MECAM [Wang et al., 2025b], DiCMoR [Wang et al., 2023b], and IMDer [Wang et al., 2023c]. All methods are evaluated using ACC and F1score across all conditions for binary sentiment classification.

Our approach performs better or remains competitive across most modality combinations, showing that TLRA mitigates degradation from incomplete multimodal information. The gains are clearer under severe missing settings, indicating stronger robustness with limited evidence. Representation-level alignment stabilizes incomplete features, while decision-level alignment reduces unreliable modality-wise predictions. Together, they address representation drift and decision inconsistency, producing more reliable sentiment predictions. Since missing-modality and reliability-imbalance settings both reflect modality reliability degradation, TLRA can handle unreliable evidence without relying on specific noise patterns, highlighting its adaptability to real-world conditions and varying data quality.

Full-Modality Comparison. Quantitative performance comparisons for our full-modality evaluation condition can be found in Table 2 where we compare our method to other representative multimodal sentiment analysis methods including ALMT, DMD, LNLN, DLF [Wang et al., 2025a], UEFN, UPNS and M-BERT [Rahman et al., 2020].

Overall, the proposed framework achieves the highest ACC and F1-score on both CMU-MOSI and CMU-MOSEI, demonstrating robustness under complete-modality settings. Beyond performance gains, this suggests TLRA improves multimodal fusion even when all modalities are present. This also indicates TLRA is not limited to missing-modality scenarios; its decision-level reference mechanism improves fusion reliability when modalities exhibit inconsistent quality or conflicting sentiment cues. This further highlights the effectiveness of reference-guided decision alignment in promoting consistent and reliable multimodal predictions.

4.3 Ablation Experiments

We conduct systematic ablation studies on CMU-MOSI, with results summarized in Table 3. By progressively adding modality encoding (ME), representation-level alignment (RLA), and decision-level alignment (DLA) to a baseline, we isolate the contribution of each component.

Results from No. 1 to No. 5 show that RLA consistently improves performance, indicating that stable references effectively mitigate representation drift under modality missing. Results from No. 6 to No. 9 further confirm the effectiveness of DLA; notably, the two-stage strategy yields consistent gains by suppressing unreliable modalities via prototype guidance and voting to enhance cross-modal consistency.

The full model (No. 11) achieves the best performance across all modality settings, demonstrating that jointly aligning representations and decisions is critical for robustness under modality-missing conditions.

4.4 Cross-Dataset Generalization

Table 4 reports the cross-dataset evaluation results, where models are trained on one dataset and tested on the other. This setting is particularly challenging because speakers, topics, and expression styles vary significantly across datasets, often leading to overfitting on dataset-specific patterns. Compared with representative baselines, our method consistently achieves the best performance and exhibits smaller degradation when transferring across datasets. This indicates that TLRA effectively reduces reliance on dataset-specific cues and captures more intrinsic and transferable sentiment structures. As a result, TLRA learns more stable and datasetinvariant sentiment representations, enabling robust generalization under distribution shifts and unseen data conditions.

4.5 Prototype Similarity Analysis

Figure 4 visualizes similarity differences between selected samples and their prototypes across training stages. We se-

Ϭ.Ϭ

Figure 4: Similarity difference visualization between randomly sampled sentiment samples and sentiment prototypes across training stages. Lighter colors indicate improved cross-modal alignment and more consistent sentiment representations.

lect ten positive and ten negative samples and evaluate them at 10, 40, and 70 epochs. In early stages, high contrast indicates modality-specific differences, suggesting that the feature space is dominated by modality variations and inconsistent sentiment representations. As training proceeds, the colors become lighter and more uniform, showing that samples align with their prototypes and converge toward a shared sentiment structure. At later stages, samples show consistent prototype similarity, indicating that the prototype bank has become stable class-level anchors. This trend supports the two-stage strategy, where soft guidance stabilizes the prototype space before voting-based suppression is applied.

5 Conclusion

We present the Two-Level Reference Alignment framework to address decision drift in multimodal sentiment analysis under missing or unreliable modality conditions. TLRA aligns feature representations and final decisions to stable references, mitigating representation deviation and biased fusion caused by incomplete or unreliable evidence. Experiments on CMU-MOSI and CMU-MOSEI show robust performance across missing-modality, full-modality, and cross-dataset settings. These results demonstrate that feature- and decisionlevel reference constraints improve stability, robustness, and generalization under diverse modality degradation scenarios, while reducing sensitivity to dataset-specific sentiment patterns and modality inconsistencies. The framework is general and can be extended to other multimodal learning tasks.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 62172246), the Excellent Young Scientists Fund of the Natural Science Foundation of Shandong Province (No. ZR2024YQ071), the Key Laboratory of Forensic Examination for Sichuan Provincial Universities (No. 2024YB01), and the Fundamental Research Funds for the Central Universities (No. 22CX06037A).