Decoupling and Reconstructing: A Multimodal Sentiment Analysis Framework Towards Robustness

1 Introduction

As an important research direction in artificial intelligence, multimodal emotion recognition aims to achieve more accurate and comprehensive emotional understanding through the integration and analysis of information from different modalities (such as speech, text, vision, etc.)[Liang et al., 2021; Lv et al., 2021a]. With the rapid development of deep learning technologies and the increasing abundance of multimodal data([Zhang et al., 2024b]; [Liu et al., 2023]), significant progress has been made in this field.

Compared to laboratory environments where high-quality data samples can be artificially selected for training, data collected in real scenarios may face varying degrees of missing issues, leading to otherwise well-performing multimodal sentiment classification models to face severe performance loss when dealing with real-world incomplete data.

∗Corresponding author.

Recently, research trends have shifted from laboratory conditions to modeling data from natural scenarios. This shift creates a wider application space for MSA in the real world, despite concerns due to issues such as sensor failure and automatic speech recognition (ASR), which lead to inconsistencies such as incomplete data in real-world deployments. Many influential solutions have been proposed to address the major problem of incomplete data in multimodal sentiment analysis. For example, [Yuan et al., 2021] introduced a transformer-based feature reconstruction mechanism, TFRNet, which aims to improve the robustness of the model in dealing with random deletions in unaligned multimodal sequences by reconstructing the missing data. Zhang introduced a model (LNLN)[Zhang et al., 2024a], the Language Dominated Noise Resistant Learning Network, to improve the robustness of MSA to incomplete data. It aims to enhance the completeness of linguistic mood features, which are considered dominant moods due to their richer emotional cues and supported by other auxiliary moods.

The previous methods have the following problems: the process of reconstructing complete inputs does not take into account the redundancy and complementarity that exists between different modal data, resulting in the model failing to achieve the desired reconstruction effect; at the same time, the inclusion of reconstruction loss may cause the model to pay too much attention to the consistency between the complete data and the missing data after feature extraction, resulting in the degradation of the encoder effect and the failure to effectively extract key features.

To solve the above problems, we propose a feature decoupling-reconstructing approach for multimodal feature fusion. As shown in Figure 1, we first decompose modal features into modal-independent and modal-common features by methods of mutual information-based approach. Then we reconstruct features corresponding to two complete inputs according to the respective properties of the two types of features. We also use a specialized neural network for the output from complete data to guide the supervised feature reconstruction of the model features for the downstream task. The contributions of this work can be summarized as:

• We propose a new approach that is suitable for feature reconstruction to decouple sequence features based on mutual information.
• We propose a missing feature reconstruction method based on decoupled features, which intuitively reflects the redundancy and complementary relationship between different modal data.
• We validate our approach on two widely used multimodal sentiment analysis datasets and compare it with other robust and non-robust fusion methods. The results demonstrate that our approach outperforms other existing models on several metrics and achieves the best overall performance.

2 RelatedWork

2.1 Robust Representation Learning inMSA

Multimodal Sentiment Analysis (MSA) methods can be categorized into Context-based MSA and Noise-aware MSA, depending on the modeling approach[Zhang et al., 2024a]. Most of previous works ([Zadeh et al., 2017]; [Tsai et al., 2019]; [Mai et al., 2020]; [Hazarika et al., 2020]; [Liang et al., 2020]; [Rahman et al., 2020]; [Yu et al., 2021]; [Han et al., 2021]; [Lv et al., 2021b]; [Yang et al., 2022]; [Guo et al., 2022]; [Zhang et al., 2023]; [Zhang et al., 2019]; [Zhang et al., 2021]; [Zhang et al., 2022a];[Zhang et al., 2022b]) can be classified to Context-based MSA. This line of work primarily focuses on learning unified multimodal representations by analyzing contextual relationships within or between modalities. For example, [Zadeh et al., 2017] explore computing the relationships between different modalities using the Cartesian product. [Tsai et al., 2019] utilize pairs of Transformers to model long dependencies between different modalities. [Yu et al., 2021] propose generating pseudo-labels for each modality to further mine the information of consistency and discrepancy between different modalities. Despite these advances, context-based methods are usually suboptimal under varying levels of noise effects (e.g. random data missing). Several recent works ([Mittal et al., 2020];[Yuan et al., 2021];[Yuan et al., 2024];[Li et al., 2025]) have been proposed to tackle this issue.

In concrete terms, [Hazarika et al., 2020] and [Yang et al., 2022] apply feature disentanglement to each modality, modeling multimodal representations from multiple feature subspaces and perspectives. [Yu et al., 2021] and [Liang et al., 2021] explore self-supervised learning and semi-supervised learning to enhance multimodal representations, respectively. [Tsai et al., 2019] and [Rahman et al., 2020] introduce Transformer to learn the long dependencies of modalities. [Zhang et al., 2023] devise a language-guided learning mechanism that uses modalities with more intensive sentiment cues to guide the learning of other modalities. Noise-aware MSA focuses more on perceiving and eliminating the noise present in the data. For example, [Mittal et al., 2020] design a modality check module based on metric learning and Canonical Correlation Analysis (CCA) to identify the modality with greater noise. [Yuan et al., 2021] design a feature reconstruction network to predict the location of missing information in sequences and reconstruct it. [Yuan et al., 2024] introduce adversarial learning to perceive and generate cleaner representations. [Zhang et al., 2024a] proposed LNLN, explored the capability of language-guided mechanisms in resisting noise and provide new. perspectives for the study of MSA in noisy scenarios.

2.2 Multimodal Feature Decoupling

One of the more important features of multimodal tasks, compared to unimodal tasks, is the redundancy and complementarity of the modal information prior. A lot of work has been done to explore the decoupling of modal features into irrelevant classifications and apply them to downstream tasks, starting from the commonalities and differences of information between different modalities. Currently, multimodal feature decoupling can be categorized into two kinds: spatial-based and mutual information-based, among which the spatial-based work is [Hazarika et al., 2020] and [Li et al., 2023], The degree of similarity and dissimilarity of features is measured using the vanilla cosine distances between feature vectors, respectively. And the mutual information-based approach is [Yang et al., 2023] and [Xia et al., 2024]. The former defines similar and dissimilar features by constructing positive and negative examples, and the latter optimizes the loss of mutual information by constructing time-series versions of the upper and lower bounds on the use of mutual information approximations.

Inspired by works on mutual information-based feature decomposition([Yang et al., 2023];[Xia et al., 2024]), the sequence feature decoupling module proposed in this paper employs a similarity measure based on both mutual information and spatial properties, which assumes that similar features have high mutual information between them, while mutual information between dissimilar features should be minimized.

3 The DARModel

3.1 Task Setup

In this paper, we consider three modalities, i.e., language (l), visual(v), acoustic (a). These modalities are represented as Ul ∈ RTl×dl , Uv ∈ RTv×dv , and Ua ∈ RTa×da respectively. Here Tm denotes the length of the utterance, such as number of tokens (Tl), for modality m and dm denotes the respective feature dimensions.

Given these sequences Um∈{l,v,a}, the primary task is to predict the affective orientation of utterance U from either a predefined set of C categories y ∈ RC or as a continuous intensity variable y ∈ R.

3.2 Overview

The general structure of the model is shown in Figure 2. It first obtains incomplete multimodal data through the datamissing operation. Model DAR first uses an alignment layer to adjust the input features of all modalities to the same dimension to ensure data consistency. Then, for each modal input, we use independent modal-common feature encoder and modal-independent feature encoder to obtain modalcommon representation and modal-independent representation of the features. Next, the modal reconstruction module corrects the decomposed two feature reconstructions to restore the feature representation corresponding to the full input. Finally, the feature fusion module utilizes the selfattention mechanism and the cross-attention mechanism to process the two kinds of features, fuse them, and output the classification results through the output layer.

3.3 Input Construction andMultimodal Input

Following the previous method ([Zhang et al., 2024a]), for each modality, we randomly erase changing proportions of information (from 0% to 90%). These pre-processed inputs are represented as sequences, denoted by Um ∈ RTm×dm , m ∈ {l, v, a} representing language, visual and acoustic features respectively where Tm denotes the length of the sequence for modality m (such as number of tokens for m = l), and dm denotes the respective feature dimensions . With obtained Um, we apply random data missing to Um, thus forming the noise-corrupted multimodal input ˜Um .

3.4 Decouple Module

It is essential to standardize the feature representations across modalities for ease of further processing. To achieve this, we apply 1D convolutions followed by a simple nonlinear layers to process the input features. Given features corresponding to complete input data and random missing data be represented as Um ∈ RTm×dm and ˜Um ∈ RTm×dm , m ∈ {l, v, a}. After the alignment operation, the output feature U1m ∈ Rt×d and ˜U1m ∈ Rt×d have unified length of utterance, t and feature dimension d across all modalities, making it suitable for subsequent model processing.

Given the incomplete sequence ˜U1m ∈ Rt×d for modality m, we employ common feature extractors and independent feature extractors to extract the modal-common features ˜Hcom m and modal-independent features ˜Hspec m using the encoding functions.

Similarly, for the complete input corresponding to feature U1m we also use the same encoder to obtain the corresponding modal-common input and modal-independent inputs Hcom m and Hspec m . We reserve two types of features for the generation of restoration features under supervision.

Based on the characteristics of the modal-common and modal-independent features, we aim to ensure that the common features from the same sample across different modalities exhibit high consistency, while the independent features within the same modality show high consistency as well. Simultaneously, we seek to reduce the information redundancy between the two types of features. To achieve this, we define a decoupling loss function Ldecouple as:

Where λ is a hyperparameter, Lre is the restoration loss that reduces the decomposed feature to the original feature and I for mutual information. The mutual information between the two distributions is represented as follows:

where:p(z1, z2) is the joint probability distribution of z1 and z2,p(z1) and p(z2) are the marginal distributions of z1 and z2, respectively.

Specifically, for sets of data in batches B we have:

where i, j represent two different batches of data.

where ˜Hcom m and ˜Hspec m represent the modal-common features and modal-independent features, respectively, m ∈ M and M = {l, v, a}. The objective is to maximize the mutual information between the common features of different modalities for the same sample and the independent features of different batches within the same modality, while minimizing the mutual information between the common and independent features of the same sample.

For the similarity loss, we use the noise comparison lower bounds of the mutual information for optimization; for the dissimilarity loss, we use the CLUB upper bounds of the mutual information for optimization, and we achieve the minimization of decoupling loss by optimizing the upper and lower bounds of the mutual information.

InfoNCE-based Mutual Information Maximization: InfoNCE([Oord et al., 2018]) is a commonly used lower bound for mutual information loss, contrastive methods enhance this by utilizing sample pairs from positive set P and negative set N. The goal is to pull positive pairs closer in the representation space while pushing negative pairs apart. The commonly used InfoNCE loss is defined as:

where: sim(·, ·) is a similarity function, in this paper, we use the cosine similarity, and τ is a temperature parameter. |P| denotes the cardinality of the positive pair set. We maximize the mutual information between positive examples by constructing positive and negative examples, chosen as shown in Figure 3. According to 3a, 3b in Figure 3, we compute the Lcom sim and Lspec sim corresponding to the common and independent features respectively, and add the two together to obtain the final Lsim.

We average the original time series features in the time dimension as the sample features, obtain the corresponding feature z, calculate the InfoNCE as the loss of the lower bound of the mutual information.

CLUB-based MI Minimization: CLUB can effectively optimize the MI upper bound, demonstrating superior advantages in information disentanglement [Cheng et al., 2020]. Given two variables x and y, the objective function of CLUB is defined as:

where qθ is the variational approximation of ground-truth posterior of y given x and can be parameterized by a network θ. We use CLUB to optimize the MI upper bound between the common features ˜Hcom m andmodal-specific features ˜Hspec m . To better measure the mutual information between the two temporal features, we use a combination of a bidirectional lstm([Huang et al., 2015]) and a nonlinear fully connected layer as a variational approximation network qθ, we modify IvCLUB into following:

The approximation network and the main networks are optimized alternatively during training process.

Restoration Loss: To distinguish the differences between ˜Hcom m and ˜Hspec m and mitigate the feature ambiguity, we synthesize the vanilla coupled features ˜U1m in a self-regression manner. Mathematically speaking, for each modality m, we concatenate the features from the other two modalities with ˜Hspec m and exploit a private decoder Dm to produce the coupled feature. Specifically: For modality l:

For the other two modalities, we also use the same way to get the losses Lvre and Lare . Adding up these losses, we get the overall restoration loss Lre:

3.5 Reconstruct Module

We hypothesize that the independent features of a complete modality can be predicted through the corresponding independent features of the missing modality feature, while the common features of a complete modality can be predicted by the common features of all the input missing modalities feature.

To implement this, we propose two distinct feature reconstruction modules for each modality: the Independent Feature correction module and the Common Feature reconstruction module. The Independent Feature reconstruction module takes as input the decoupled independent features and outputs the corrected independent features ˆHspec m . In contrast, the Common Feature Reconstruction module uses the combined common features from all modalities as input and generates the reconstructed features ˆHcom m as output. Finally, after obtaining the two features, we use a specially set up private decoder Dm to reconstruct the coupled complete input U1m.

where θcom denotes the parameters of the common feature reconstruction module Ecom and θspec denotes the parameters of the independent feature reconstruction module Espec.

Finally, we combine reconstructed features with original input features to obtain features for downstream tasks.

To ensure that the reconstructed features are consistent with the common and independent features obtained from the complete input through the encoder, hereafter referred to as the complete common and complete independent features, we construct the alignment loss minimizing the loss between the corrected features and the complete features as following:

3.6 Fusion-Output Module

For the modal-common features, which exhibit relatively similar distributions, we apply a multi-layer self-attention model for further refinement. In contrast, for the modalindependent features, where there are significant distributional differences between features, we employ a crossattention mechanism.

Modal-common Features Fusion Module. Given the modified modal-common feature Hcom fused , we perform feature fusion in the temporal dimension using a multilayer selfattention module for each modal counterpart, while using the features of the last frame of the output of the last layer as the overall feature output hfused.

Modal-independent Features Fusion Module. For modal-independent features, we use a cross-attention mechanism to fuse different modal information. The core of the multimodal transformer is the crossmodal attention unit (CA), which receives features from a pair of modalities and fuses cross-modal information. Take the language modality Hspec fused-L as the source and the visual modality Hspec fused-V as the target, the cross-modal attention can be defined as: QV = Hspec fused-VPq, KL = Hspec fused-LPk, and VL = Hspec fused-LPv, where Pq, Pk, Pv are the learnable parameters, formulated as:

where hspec L→V is the enhanced features from Language to Visual, d means the dimension of QV and KL. For the three modalities in MER, feature of each modality hspec m will be reinforced by the two others and the resulting features will be concatenated. Take visual modality as an example the formula is expressed as follows:

Prediction/Inference. Finally, we splice the obtained fused features and input the nonlinear fully connected layer to generate predictions ˆ y, we also use the bootstrap module to predict the results ˆ yboot using common features generated from the complete information, ensuring that the encoder learns features that facilitate classification.

The task loss Ltask and overall model loss Ltotal are formulated as follows:

where α and β are hyperparameters.

4 Experiments and Analysis

In this section, we provide a comprehensive and fair comparison between the proposed DAR and previous representative MSA methods on MOSI ([Zadeh et al., 2016]) and MOSEI ([Bagher Zadeh et al., 2018]) datasets.

4.1 Datasets

MOSI The dataset includes 2,199 multimodal samples, integrating visual, audio, and language modalities. It is divided into a training set of 1,284 samples, a validation set of 229 samples, and a test set of 686 samples. Each sample is given a sentiment score, varying from -3, indicating strongly negative sentiment, to 3, signifying strongly positive sentiment.

MOSEI The dataset consists of 22,856 video clips sourced from YouTube. The sample is divided into 16,326 clips for training, 1,871 for validation, and 4,659 for testing. Each clip is labeled with a score, ranging from -3, denoting the strongly negative, to 3, denoting the strongly positive.

4.2 Evaluation Settings and Criteria

For each sample in the dataset, we incorporate data from three modalities: language, audio, and visual data. Consistent with previous works ([Zhang et al., 2023]), each modality is processed using widely-used tools: language data is encoded using BERT([Devlin, 2018]), audio features are extracted through Librosa ([McFee et al., 2015]), and visual features are obtained using OpenFace ([Baltrusaitis et al., 2018]). Specifically, for visual and audio modalities, we fill the erased information with zeros. For language modality, we fill the erased information with [UNK] which indicates the unknown word in BERT ([Devlin, 2018]).

Following the previous works ([Zhang et al., 2024a]), we report our results in classification and regression with the average of 3 runs of different seeds and 10 missing rates from 0.0 to 0.9 at 0.1 intervals. For classification, we report the multiclass accuracy and weighted F1 score. We calculate the accuracy of 2-class prediction, 5-class prediction (Acc-5) and 7-class prediction (Acc-7) for MOSI and MOSEI. Besides, Acc-2 and F1-score of MOSI and MOSEI have two forms: negative/non-negative (non-exclude zero) ([Zadeh et al., 2017]) and negative/positive (exclude zero) ([Tsai et al., 2019]1). For regression, we report Mean Absolute Error (MAE) and Pearson correlation (Corr). Except for MAE, higher values indicate better performance for all metrics.

In training process, for hyperparameters, we choose that λ = 0.7, α = 0.1, β = 0.1. On the mosi dataset, we choose the missing rate k = 0.3, and on the mosei dataset, we choose k = 0.4.

Compared with the baseline LNLN([Zhang et al., 2024a]) which uses the best model under different metrics for testing, we use the same model with the smallest overall loss as the optimal model for testing, and at the same time, in order to ensure the stability of the results, we randomly test three times and take the average value as the final result following the baseline settings.

Inaddition, the result of MISA, Self-MM, MMIM, CENET, TETFN, ALMT is reproduced by the authors from open source code in the MMSA([Mao et al., 2022]),which is a unified framework for MSA, using default hyperparameters, LNLN([Zhang et al., 2024a]) model is implemented using the author's open source code and for TFR-Net, We use the results reported in the LNLN article, and since that article uses the best modeling results under the corresponding metrics, we consider this comparison to be fair.

4.3 Robustness Comparison

Table 1 shows the robustness evaluation results on the MOSI and MOSEI datasets. As shown in Table 1, DAR achieves state-of-the-art performance on most metrics, demonstrating the robustness of DAR in the term of different noise effects. For seven categorical metrics on the mosi dataset MAE versus the mosei dataset, our model is able to achieve sub-optimal results. Considering the unpredictability of the impact of stochastic factors on the quality of missing data, and some of the extremes of the data have a huge impact on the overall results, in this case, given the inherent instability of missing data, we can assume that DAR achieves the optimal overall performance on both datasets compared to the other models compared.

Figure 4 shows the performance of all models under two of the most commonly used binomial and multiclassification metrics, non0acc2 and acc5, at different missing rates. The results show that although DAR loses part of its performance compared to other models when facing complete inputs, its performance under other missing rates is significantly improved compared to other models without missing data, and also compared to TFR-Net and LNLN trained with missing data, which proves the effectiveness of our method.

4.4 Ablation Experiment

To evaluate the effectiveness of our proposed approach, we conduct a series of ablation experiments. These experiments systematically remove or modify key components of our model to assess their individual contributions to performance. By comparing the results of these ablations with the full model, we are able to quantify the impact of each design choice. This analysis provides a deeper understanding of the strengths and limitations of our method.

The effect of the ablation experiment is shown in Table 2. The results of the ablation experiments demonstrate the effectiveness of our proposed multimodal fusion framework based on the decomposition-reconstruction idea. Compared to the complete model, eliminating either similarity or dissimilarity loss causes information redundancy in the feature correction reconstruction process, which reduces the performance of the model to varying degrees.

Besides, we also verified the effect of eliminating the alignment loss and bootstrap loss in the incomplete feature reconstruction process on the model effectiveness, and the elimination of the alignment loss increases the uncertainty in the incomplete feature reconstruction process and affects the model performance. While eliminating the bootstrap loss causes the model to focus too much on the effect of the incomplete feature reconstruction, in order to minimize the difference losses between the incomplete input and the complete input after encoding. This leads to the degradation of the encoder's ability to extract features, the reduction of the variability of the extracted features, and ultimately impairing the model's ability.

4.5 Missing Rates Sensitivity Experiment

During the training of the model, we found that the manually selected missing rate of the multimodal data has a critical impact on the training process, and the following demonstrates the specific impact of the missing rate on the model output results. We tested the performance of the model under different missing training sets constructed with different missing rates k. The results are shown in Table 3.

Analyzing the experimental results, it can be seen that the performance of the model appears to increase and then decrease overall as the missing rate increases. After analyzing the results, we believe that too low missing rate will lead to the missing data is not distinct enough from the original complete input data, and the model degenerates into an ordinary multimodal fusion model. In this case,the DAR model is unable to learn the ability of feature reconstruction, while too high missing rate will lead to the features being corrupted seriously, especially for the modal common features, which may lead to the fact that all the modal features corresponding to all modal features are after alignment under too high missing rate. The model is therefore unable to learn the ability to reconstruct complete features from incomplete features.

5 Conlusion

In this paper, we propose a novel method for multimodal sentiment analysis called Decoupled-Adaptive Reconstruction (DAR). The framework uses a reconstruction method based on feature decoupling, and adopts different reconstruction methods for the modal-common features and modal-independent features of the missing data according to their own properties, and achieves a more obvious improvement in the robustness test of the mosi and mosei datasets compared with the existing methods. In addition, we validate the effectiveness of our proposed feature decomposition-reconstruction framework through ablation experiments, showing that our method can alleviate problems such as information redundancy in the feature reconstruction process.

Finally, we explore the performance of the trained models with different levels of data missing rates, and the results show that choosing the appropriate data missing rate has an extremely important impact on the robust performance of the models. In this experiment, we only discuss the case of the same missing rate for multiple modalities, however, in practice, due to the different quality and noise immunity of different modalities, choosing different missing rates for different modalities or using methods that can adapt the missing rate is a more promising direction for future improvement.

Acknowledgments

This research was partially supported by the National Natural Science Foundation of China (Grants No.62406303), Anhui Provincial Natural Science Foundation (No. 2308085QF229), Anhui Province Science and Technology Innovation Project (202423k09020010) and the Fundamental Research Funds for the Central Universities (No. WK2150110034).