GSDNet: Revisiting Incomplete Multimodality-Diffusion Emotion Recognition from the Perspective of Graph Spectrum

1 Introduction

Multimodal emotion recognition in conversations (MERC) aims to build an emotion recognition model with cross- Recently, graph neural networks (GCNs) and diffusion generative models (DGMs) have shown outstanding performance in multiple tasks such as vision, and language [Hu et al., 2021], [Jo et al., 2022]. The core of GCNs is message passing based on graph structure to establish complex dependencies. The core of DGMs is forward denoising and backward denoising to learn the original distribution of data. Considering the potential of GCNs and DGMs, some researchers have tried to apply GCNs and SGMs to modality completion. They alleviate the impact of modality loss on multimodal emotion recognition performance from different perspectives. For GCNs, [Lian et al., 2023] applied graph completion networks to model semantic dependencies between different modal data to achieve modal recovery. For DGMs, [Wang et al., 2024] proposed modal diffusion to learn the original distribution of data to achieve modal recovery. All of the above methods have shown significant results because they have strong modeling capabilities in their respective dimensions. Specifically, GCNs and DGMs respectively model the dependence and distribution between multi-modal data to achieve modal recovery. Generally, fusing complex semantic information between multi-modal features and capturing the original distribution of multi-modal data are crucial for emotion recognition performance in missing modalities. This inspires us to combine these two dimensions to obtain more powerful modal recovery capabilities.

∗Corresponding author

However, unlike visual data, which has dense structural information, the structure of graph data is generally sparse, causing the data generated by the graph diffusion model to be unable to retain the topological information of the original graph. As shown in Fig. 1, the image perturbed by Gaussian noise still retains recognizable numerical patterns and local structural information in the early and middle stages of forward diffusion. For example, even if the details of the image are gradually covered by noise, its global contour and edge information can still be captured by the model to a certain extent. This enables the model to effectively restore the content of the image using this residual information during the reverse diffusion process. However, during the forward diffusion of graph data, the topological structure of the graph adjacency matrix is rapidly lost and a dense noise matrix is formed. Intuitively, the diffusion method of inserting Gaussian noise into the graph adjacency matrix seriously undermines the ability to learn the graph topology and feature representation. From a theoretical perspective, running diffusion over the entire space of the adjacency matrix will cause the signal-to-noise ratio (SNR) to drop rapidly and approach zero. Since the SNR is basically zero, the scoring network will not be able to effectively capture the gradient information of the original distribution during training.

To overcome these problems, we propose a novel Graph Spectral Diffusion Network (GSDNet) to strictly restrict the diffusion of Gaussian noise to the spectral space of the adjacency matrix. Specifically, we perform eigendecomposition on the adjacency matrix, decomposing it into eigenvalue matrix and eigenvector matrix and adding Gaussian noise to the eigenvalues without interfering with the eigenvectors. On the one hand, by operating the eigenvalues in the spectral space, the direct destruction of the local structure of the adjacency matrix can be effectively avoided, ensuring the generated graph still conforms to the global semantics. On the other hand, this constrained diffusion process can more naturally capture and preserve the topological information of the graph, ensuring that the generated graph has consistent spectral features. Overall, our contributions are as:

• We design a novel modality completion model, the Graph Spectral Diffusion Network (GSDNet), which can simultaneously model the dependencies between multimodal features and the distribution of original data to obtain powerful modality recovery capabilities.
• We strictly limit the diffusion of Gaussian noise in the spectral space of the adjacency matrix to avoid the destruction of the graph structure and ensure that the graph data generated by GSDNet still conforms to the global semantics.
• We conduct extensive experiments on multiple realworld datasets to demonstrate that our GSDNet outperforms state-of-the-art methods for conversational emotion recognition in incomplete multimodal scenarios.

2 RelatedWork

2.1 Incomplete Multimodal Learning

In practical applications, missing modalities are an inevitable problem. To address this challenge, an effective approach is to find a low-dimensional subspace that can be shared by all modalities, in which the correlation between different modalities is maximized. However, strategies based on shared lowdimensional subspaces often ignore the complementarity between heterogeneous modalities. To overcome this shortcoming, another more effective approach is to restore the missing modality through the existing modality. This process not only requires inferring the content of the missing modality based on the features of the known modality but also ensures that the restored modality can work together with other modalities. Existing modality restoration methods can be divided into several types, including zero-based restoration [Parthasarathy and Sundaram, 2020], average-based restoration [Zhang et al., 2020], and deep learning-based restoration [Pham et al., 2019]. Since zero-filling and average-based restoration methods do not use any supervised information, the data they restore often have a significant gap with the original data. In contrast, deep learning-based methods, with their powerful feature learning capabilities, can more accurately estimate the missing modality. For example, Tran et al. [Tran et al., 2017] used a cascaded residual autoencoder to restore the missing modality, and the network's residual learning mechanism made the restoration effect more accurate. In addition, some researchers have proposed deep learning methods based on cross-modal restoration strategies, using cycle consistency loss to ensure the matching degree between the restored modality and the original modality [Zhao et al., 2021]. Other studies use graph neural networks (GNNs) to solve the modality restoration problem. For example, Lian et al. [Lian et al., 2023] introduced a graph neural network framework and combined the relationship between nodes and edges to enhance the correlation between modalities.

2.2 Score-based Generative Models

Score-based generative models (SGMs) estimate the probability distribution of data by parameterizing the score function [Song and Ermon, 2019], [Song and Ermon, 2020], [Song et al., ]. Specifically, SGMs model the scoring network s(x; θ) through learnable parameters θ, thereby training the model to estimate ∇x log p(x). Unlike likelihood-based generative models (e.g., regularized flows [Kingma and Dhariwal, 2018]), score-based generative models do not require regularization of the generation process. Specifically, in likelihoodbased methods, model training usually relies on maximizing the likelihood function, which means that regularization terms need to be introduced to prevent overfitting and ensure model stability. In contrast, SGMs estimate the gradient of the data distribution by optimizing the score function. This approach usually does not require explicit regularization and reduces the complexity of model training. In addition, SGMs only need to focus on estimating the gradient of the data distribution by learning the score function, avoiding the complexity of accurately modeling the entire distribution process.

3 Preliminary Information

3.1 Score-based Generative Models

SGMs are efficient generative models that can generate highquality data and model complex data distribution. Given an input X ∈ Rd and a complicated data distribution D, a forward noising process can be obtained through a stochastic differential equation (SDE) as follows:

where σt the diffusion coefficient, B represents the Brownian motion.

Assuming that pt is a probability density function, the reverse denoising process can be established through the reversed time SDE as follows:

where d¯ t = -dt is the negative infinitesimal time step, ¯B is the reversed time Brownian motion.

In the reverse denoising process, the scoring network s(X(t), t; θ) provides gradient information for the current noise sample Xt, indicating how to adjust the value of the sample so as to gradually restore the original data distribution. During the training process, the scoring network is to minimize the gap between the model estimated score function and the true score function through the score matching loss function as follows:

where U(0, T) is a uniform distribution over [0, T]. Given a well-trained scoring network sθ∗ , we can generate realistic data by solving the learned reverse-time SDE as follows:

where π is the prior information of the data.

3.2 Score-based Graph Generative Models

In the graph generation model, given a graph G(X,A) ∈ Rn×d × Rn×n with n nodes, where X ∈ Rn×d is the feature vectors of each node with dimension d , and A ∈ Rn×n is the connection relationship between nodes. The goal of the graph generation model is to learn the underlying data distribution of the graph, A standard Graph Diffusion Score-based Model [Jo et al., 2022], [Luo et al., 2023] gradually generates a perturbation graph through a diffusion process and learns the generation process of the graph through a scoring network. Specifically, given each graph sample (X, A), a forward noising process is obtained through an SDE as follows:

where σX,t, and σA,t the diffusion coefficient, BAt , and BXt represent the Brownian motion.

Assuming that pt is a probability density function, the reverse denoising process can be established through the reversed time SDE as follows:

where ¯BAt , and ¯BXt represent the reversed time Brownian motion.

Given G0, the joint probability distribution of Xt and At can be simplified to the product of two simpler distributions [Jo et al., 2022], so that the objective function of denoising score matching can be simplified in form as follows:

4 ProblemDefinition

Assume that the set {X1,X2, . . . ,XM} represents the M modalities, where Xk represents the input of the k-th modality. We introduce a binary indicator α ∈ {0, 1} to identify the availability status of each modality. If the k-th modality is missing, let αk = 0; conversely, if the k-th modality is available, let αk = 1. We can define a set of missing modalities Im = {k|αk = 0}. In this incomplete modality scenario, the goal is to recover these unobserved modalities to make up for the missing information. The process of modal recovery usually needs to rely on the existing observed modal information Io = {k|αk = 1}, and complete it by modeling the correlation between modalities.

Our main idea is to recover the missing emotion modality Im from its latent distribution space conditioned on the observed modality Io. We use the observed modality Io as a semantic condition to guide the generation of the missing modality, ensuring that the recovered modality data is consistent and relevant to the real data. Formally, we denote the data distribution of the missing modality as p(Xm) and the data distribution of the available modality as p(XIo ). Our ultimate goal is to sample the missing modality data from the conditional distribution p(Xm|XIo ). Inspired by graph completion networks [Lian et al., 2023] and diffusion modality generation [Wang et al., 2024], we combine the advantages of GNNs and diffusion models to simultaneously model the complementary semantic information between modalities and reconstruct high-quality missing modality features.

However, directly adding Gaussian noise to the adjacency matrix may seriously destroy the local structure of the graph, resulting in an unreasonable generated adjacency matrix. To overcome these problems, we propose to strictly restrict the diffusion of Gaussian noise to the spectral space of the adjacency matrix. On the one hand, by operating the eigenvalues in the spectral space, the direct destruction of the local structure of the adjacency matrix can be effectively avoided, ensuring that the generated graph still conforms to the global semantics. On the other hand, this constrained diffusion process can more naturally capture and preserve the topological information of the graph, ensuring that the generated graph has good connectivity and consistent spectral features.

Specifically, we consider a multi-step diffusion model to gradually construct the conditional distribution by perturbing Xm and Λm, where Am = UmΛmUm, U are the eigenvectors and Λ is the diagonal eigenvalues. In the t-th step, the conditional transfer distribution of the modal features and the adjacency matrix can be expressed as pt(Xm(t)|XIo (0)) and pt(Λm(t)|ΛIo (0)) and can be approximated as follows:

According to the score-based diffusion model [Jo et al., 2022], we calculated the conditional transition probability

where XIo (t) and ΛIo (t) is a random sample from pt(XIo (t))|XIo (0)) and pt(ΛIo (t))|ΛIo (0)), respectively. Eq. 9 is held because:

Finally, we derive the score-matching objective as follows:

5 The ProposedMethod

5.1 Modality Encoder

Since the original features of text, audio, and video modalities usually have significant dimensional differences, directly using the original features of the modalities to recover missing modalities may lead to difficulties in semantic alignment or even introduce noise.To ensure that the unimodal sequence representations of the three modalities can be mapped to in the same feature space, we input these modalities into a onedimensional convolutional layer to achieve feature alignment:

where lm represents the size of the one-dimensional convolution kernel corresponding to the m-th modality. N represents the number of utterances in the conversation. d represents the dimension of the common feature space.

To make full use of the position and order information in the sequence, we introduced position embedding when processing the sequence after convolution:

where pos represents the index of the sequence, and dimension i represents the index of the feature dimension.

Overall, we feed position embeddings into a convolutional sequence as follows:

5.2 MissingModality Graph Spectral Diffusion Network

We train two scoring networks sθ and sϕ to model the distribution of missing modalities m ∈ Im, respectively. Similar to score-based diffusion [Jo et al., 2022], the corresponding inverse-time SDE can be derived as follows:

The core of Eq. 15 is to guide the inverse diffusion process through the trained score network, gradually transforming random noise into missing modal data consistent with the true distribution. We assume that the language modality Xl, the visual modality Xv, and the corresponding diagonal eigenvalues Λl and Λv are observed, while the acoustic modality Xa and the corresponding diagonal eigenvalues Λa are missing. Our goal is to model the missing acoustic modality Xa and Λa, and recover their data through the score network sθ and sϕ conditioned on Xl, Xv, Λl and Λv as follows:

where ∆t is a discrete time step size and ϵt ∼ N(0, I). After enough iterations, we can gradually guide the noise data to approach the target distribution and finally obtain the restored acoustic modal data X′a and the corresponding diagonal eigenvalues Λ′a . To generate more refined acoustic modalities, we input the restored acoustic data xa into a specially designed acoustic modal reconstruction module DX and diagonal eigenvalues reconstruction module DΛ to obtain the final reconstructed acoustic modal ˆXa = DX(X′a ) and diagonal eigenvalues ˆΛa = DΛ(Λ′a ). We define a reconstruction loss function Lrec to measure the difference between the reconstructed data and the original target modal data under any missing patterns as follows:

Therefore, the loss of the missing modality graph diffusion network is as follows:

5.3 Multimodal Fusion and Prediction

The recovered data and the observed available data are combined to obtain the complete multimodal data H and the adjacency matrix A. To achieve the fusion of multimodal data, we use GCN to capture the complementary semantic information between the modalities as follows:

whereW(l) is the learnable weight matrix of the l-th layer.

To train the entire model, we combine the losses of the above reconstruction and prediction tasks into a joint optimization objective function as follows:

where β is a hyperparameter.

6 Experimental Database and Setup

6.1 Datasets

We conduct extensive experiments on two MERC datasets to conduct experiments, including CMU-MOSI [Zadeh et al., 2016], and CMU-MOSEI [Zadeh et al., 2018]. On the two datasets, we extract the lexical modality features via pretrained RoBERTa-Large model [Liu et al., 2019] and obtain a 1024-dimensional word embedding. For visual modality, each video frame was encoded via DenseNet model [Huang et al., 2017] and obtain a 1024-dimensional visual feature. The acoustic modality was processed by wav2vec [Schneider et al., 2019] to obtain the 512-dimensional acoustic features.

6.2 Baselines

We compare our proposed method GSDNet to the stateof-the-art incomplete learning methods, including MCTN, MMIN [Zhao et al., 2021], GCNet [Lian et al., 2023], DiCMoR [Wang et al., 2023a], IMDer [Wang et al., 2023b].

7 Results and Discussion

7.1 Comparison with the state-of-the-arts

Tables 1 and 2 lists the quantitative results of the different missing modalities and the random missing ratio on CMUMOSI and CMU-CMSEI datasets, showing the performance of different methods under the missing modal. Specifically, GSDNet achieved the best results on the two datasets, verifying its superiority in dealing with modal missing. The performance improvement of GSDNet may be attributed to its ability to explicitly restore the missing modality, which not only helps to restore the lost information but also provides additional complementary information for MERC. In addition, GSDNet has a significant advantage in maintaining consistency between the restored modality and the original modality. This distribution consistency ensures that the information fusion between different modalities is smoother and more accurate, further improving the overall performance of the model. Compared with other MERC methods, the performance degradation of GSDNet decreases as the modal missing rate increases. In practical applications, when the modal missing rate is high, most recovery-based models will experience significant performance degradation.

7.2 Ablation study

We conduct ablation experiments on the CMU-MOSI and CMU-MOSEI datasets. The results in Table 3 show that GSDNet consistently outperforms all variants. Removing the frequency diffusion degrades the performance, which highlights the role of frequency diffusion in capturing the distribution of multimodal data.

7.3 Visualization of Embedding Space

Fig. 3 shows the distribution of the restored data and the original data in the feature space obtained by different restoration methods under the condition of fixed missing modalities. In order to compare these distributions more intuitively, we use t-SNE dimensionality reduction technology on the CMUMOSEI dataset to project the high-dimensional features into two-dimensional space for visualization. As can be seen from Fig. 3, the modal data restored by GSDNet is closest to the distribution of the original data, which shows that GSDNet can better maintain the original feature distribution of the data when restoring the missing modalities. In contrast, there is a clear difference between the distribution of the restored data of other methods and the original data, especially in some local areas, the degree of overlap of the distribution is low.

7.4 Imputation Performance

Fig. 4 shows the interpolation results of different methods under different missing rates. By comparing the performance of baseline models under different missing rates, our proposed GSDNet always outperforms other baseline methods in the CMU-MOSI and CMU-MOSEI datasets and all missing rate conditions. Specifically, GSDNet not only shows strong interpolation performance under low missing rates but also has more outstanding performance advantages under high missing rates. The experimental results show that speaker dependency and data distribution consistency play a vital role in data interpolation tasks. Most baseline methods often ignore the synergy of these dependencies, which limits their interpolation performance when dealing with missing data. In contrast, GSDNet can use the speakers relationship to perform more accurate interpolation while maintaining data distribution consistency through the graph diffusion model, so that GSDNet can always maintain relatively good performance under various missing rates.

8 Conclusions

In this paper, we introduce a novel GSDNet, which maps Gaussian noise to the graph spectral space of missing modalities and recover the missing data according to original distribution. GSDNet only affects the eigenvalues of the adjacency matrix instead of destroying the adjacency matrix directly, which can maintain the global topological information and important spectral features during the diffusion process. Extensive experiments have demonstrated that GSDNet achieves state-of-the-art emotion recognition performance in various modality loss scenarios.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No. 62372478), the Research Foundation of Education Bureau of Hunan Province of China (Grant No. 22B0275), and the Hunan Provincial Natural Science Foundation Youth Project (Grant No. 2025JJ60420).