RAMer: Reconstruction-based Adversarial Model for Multi-party Multi-modal Multi-label Emotion Recognition

1 Introduction

Emotion recognition from videos is crucial for advancing human-computer interaction and social intelligence. Multimodal multi-label emotion recognition (MMER) leverages visual, textual, and acoustic signals to identify multiple emotions (e.g., happy, sad) simultaneously [Zhang et al., 2020; Zhang et al., 2021a]. Conventional MMER methods, as shown in Figure 1(a), typically focus on monologue or dyadic settings, assuming all modalities are fully available. However, real-world conversations often involve multiple participants (i.e., multi-party) with incomplete modality data for

∗ Corresponding author.

A

Figure 1: (a) shows the conventional approach for MMER in monologue and dyadic conversations with complete modalities in a uniform representation; (b) depicts our approach for multi-party conversations with incomplete modalities, reconstructing and projecting them into both specificity and commonality representations.

non-speakers who always lack acoustic and textual signals. Multi-party MMER, a more complex and practical setting, introduces three key challenges. Firstly, handling incomplete modalities is a significant challenge, which requires robust methods to reconstruct or infer missing information. Most existing works [Zhang et al., 2021a; Zhang et al., 2022; Ge et al., 2023] assume complete modality access and encode each modality independently, overlooking missing data. While some methods [Ghosal et al., 2019; Hu et al., 2021] leverage speaker-aware context modeling, their performance degrades in multi-party settings where nonspeakers often lack critical modalities. Secondly, representing diverse modalities effectively remains challenging, often requiring techniques that can not only integrate disparate information but also reconstruct rich, complete representations from potential modalities. Current fusion strategies, such as aggregation-based methods (e.g., concatenation, averaging)[Shen et al., 2020] and hybrid approaches [Manzoor et al., 2023], project modalities into a shared subspace, often neglecting their unique characteristics and reducing discriminative ability. Recent methods [Zhang et al., 2022] attempt to separate modality-specific and shared features but often suffer from information loss due to inadequate handling of inter-modal correlations. Similarly, methods preserving modality-specific information [Peng et al., 2023] may overlook cross-modal commonalities, limiting their ability to fully capture inter-modal relationships. Finally, multi-label learning presents challenges in modeling robust label correlations and capturing complex interdependency between modalities and labels. Existing approaches [Cisse et al., 2013; Ma et al., 2021] often fail to fully exploit collaborative label relationships. Moreover, emotions vary across modalities, and different emotions rely on distinct modality features, further complicating the task.

To address these issues, we propose RAMer, a novel framework designed to tackle the challenges of the Multiparty MMER problem. RAMer integrates multimodal representation learning with multi-label modeling to effectively handle incomplete modalities in multi-party settings.

As illustrated in Figure 1(b), RAMer addresses the challenge of multi-party MMER by following techniques. To address the challenge of incomplete modalities, we propose an auxiliary task that incorporates external knowledge, such as personality traits, to complement the existing modalities. Leveraging this, we employ modality-level attention mechanisms to capture both inter- and intra-personal features. A reconstruction-based network is utilized to recover and enrich the features of any modality by leveraging information from the other modalities.

To represent diverse modalities effectively and capture discriminative features, we design an adversarial network that extracts commonality across modalities while amplifying the specificity inherent to each one. This helps ensure minimal information loss during the fusion process.

Additionally, to model robust interconnections between modalities and labels, we propose a novel modality shuffle strategy. This strategy enriches the feature space by shuffling both samples and modalities, based on the commonality and specificity of the modalities, improving the model's ability to capture label correlations and modality-to-label relationships.

In summary, the contributions of this work are:

• A Novel Model for the Multi-party MMER Problem. We present RAMer, a new framework that centrally integrates feature reconstruction within an adversarial learning paradigm. RAMer adeptly captures both commonality and specificity across modalities, crucially utilizing robustly reconstructed features to significantly improve emotion recognition, especially even with incomplete modality data.
• Optimization Techniques. To enhance the robustness of multi-party emotion recognition, RAMer employs contrastive learning to enrich reconstructed features and integrates a personality auxiliary task to capture modalitylevel attention. We also propose a stack shuffle strategy, enhancing the modeling of label correlations and modality-to-label relationships by leveraging the commonality and specificity of different modalities.
• Extensive Experiments. We conduct comprehensive experiments on three benchmarks, i.e., MEmoR, CMUMOSEI, and M 3ED, across various conversation scenarios. Results show that RAMer surpasses existing approaches and achieves state-of-the-art performance in both dyadic and multi-party MMER problems.

2 RelatedWork

Multi-modal RepresentationLearning. Emotion recognition has progressed from uni-modal approaches [Huang et al., 2021; Saha et al., 2020] to multi-modal methods [Lv et al., 2021; Zhang et al., 2022] that exploit complementary features across modalities. While uni-modal approaches often face recognition biases [Huang et al., 2021], multi-modal learning has gained significant attention, with a key challenge being the effective integration of heterogeneous modalities. Early fusion methods, such as concatenation [Ngiam et al., 2011a], tensor fusion [Liu et al., 2018a], and averaging [Hazirbas et al., 2017], struggle with modality gaps that hinder effective feature alignment. To address this, attentionbased methods [Ge et al., 2023; Tsai et al., 2019] leverage cross-attention mechanisms to dynamically align features in the latent space, while contrastive learning [Chen et al., 2020; Peng et al., 2023] further improves robustness. However, most attention-based methods merge modalities into a joint embedding, often overlooking modality-specific features.

Multi-label Emotion Recognition in Videos. MMER involves assigning multiple emotion labels to a target person in the video. Early methods treated multi-label classification as independent binary tasks [Boutell et al., 2004], but advancements explore label correlations using techniques like Adjacency-based Similarity Graph Embedding(ASGE) [You et al., 2020], Graph Convolutional Network (GCN) [Chen et al., 2019], and multi-task pattern [Tsai and Lee, 2020] to explore label correlations. Some noteworthy strategies [Zhang et al., 2021b; Zhang et al., 2022] focus on modeling labelfeature correlations through label-specific representations enabled by visual attention [Chen et al., 2019] and transformers [Zhang et al., 2022]. Beyond monologue settings, MMER in conversations has gained interest, using GCN [Ghosal et al., 2019] and memory networks [Hazarika et al., 2018] to model dynamic speaker interactions. However, multi-party MMER is especially challenging, as it requires recognizing emotions for multiple speakers with incomplete modalities. Additionally, the influence of individual personalities on label-feature correlations remains underexplored.

Adversarial Training. Adversarial training (AT) [Goodfellow et al., 2014], involves two models: a discriminator that estimates the probability of samples, and a generator that creates samples indistinguishable from actual data. This setup forms a minimax two-player game, enhancing the robustness of the model. The technique has since been adapted for CV and NLP applications [Wang et al., 2017]. For instance, Miyato et al. [Miyato et al., 2016] extended AT to text categorization by introducing perturbations to word embeddings. Wu et al. [Wu et al., 2017] applied it within a multi-label learning framework to facilitate relationship extraction. Additionally, AT has been used to learn joint distributions between multi-modal [Tsai et al., 2018]. More recently, Ge et al. [Ge et al., 2023] applied AT to reduce modal and data biases in MMER tasks. However, Zhang et al. [Zhang et al., 2022] implemented AT to extract multi-modal commonality and diversity, but suffered a significant loss of modality information due to inadequate cross-modal information fusion.

3 ProblemFormulation

In this section, we introduce the notations used and formally define the Multi-party Multi-modal Multi-label Emotion Recognition (Multi-party MMER) problem.

Notations. We use lowercase letters for scalars (e.g., v), uppercase letters for vectors (e.g., Y ), and boldface for matrices (e.g., X). A data sample is represented by the tuple (V, Pt, Sr , Et,r), where:

• V = ({Pi}Ti=1 , {Sj}Rj=1) is a video clip containing T persons and R semantic segments.
• {Pi}Ti=1 refers to the set of target persons, and {Sj}Rj=1 represents the target segments, each annotated with an emotion moment.
• Et,r denotes the labeled emotion for person Pt in Sr.

Each sample involves modalities such as visual (v), acoustic (a), textual (t), and personality traits (p).

For each modality m ∈ {v, a, t, p}, the corresponding features are represented as (X 1 ,X 2 , · · · ,X m), where X k ∈

Rlk×dk represents the feature space of the k-th modality. Here: lk denotes the sequence length, and dk denotes the dimension of the modality. Let Y = {y1, y2, · · · , yζ} represent a label space with ζ possible emotion labels.

Multi-party MMER Problem. Given a training dataset D = {X{1,2,··· ,m} τ , Yτ }N τ=1 with N data samples, where: (1) Xm τ ∈ X m represents the features for each modality m in sample τ, and (2) Yτ = {0, 1}ζ is a multi-hot vector indicating the presence (1) or absence (0) of emotion labels, where Y υ τ = 1 indicates that sample τ belongs to class υ, and Y υ τ = 0 otherwise. The goal of the Multi-party MMER problem is to learn a function F : X 1 ×X 2 ×· · ·×X m ↦→ Y that predicts the target emotion Et,r for person Pt in segment Sr, leveraging contextual information from multiple modalities.

Discussion. It is important to note that the target person Pt may have incomplete modality information, meaning they may not simultaneously possess visual, textual, or acoustic representations. This introduces uncertainty in the modality of the target segment Sr, making the prediction task more challenging.

4 Methodology

Figure 2 shows the framework of RAMer, which consists of three components: auxiliary uni-modal embedding, reconstruction-based adversarial Learning, and stack shuffle feature augmentation.

4.1 Auxiliary Uni-modal Embedding

To extract contextual information from each modality, we employ four independent transformer encoders [Vaswani et al., 2017], each dedicated to a specific modality m.Each encoder consists of nm identical layers to ensure consistent and deep representation. For multi-party conversation videos with T participants with incomplete modalities, we introduce an optional auxiliary task leveraging personality to complement missing modalities. Specifically, we concatenate personality embedding X p with each modality X m ∈ {v, t, a} to enrich the feature space. We then apply the scaled dot-product attention to compute inter-person attention across the person dimension within each segment, and intra-person attention along the segment dimension for each individual [Shen et al., 2020]. This modality-level attention mechanism is designed to enhance the model's emotion reasoning ability by effectively capturing both interpersonal dynamics and temporal patterns within the data. In this way, we obtain personality enhanced representation X mα ∈ Rl×d .

4.2 Reconstruction-based Adversarial Learning

The second component leverages multiple modalities by capturing inter-modal commonalities while preserving modalityspecific features. To address the limitations of adversarial networks [Goodfellow et al., 2014; Zhang et al., 2022], which can result in information loss and difficulty in learning modality-label dependencies, we introduce a reconstruction-based approach. It employs contrastive learning to learn modality-independent but label-relevant representations while reconstructing missing modalities during training to enhance robustness.

Adversarial Training. To balance modality specificity and commonality, we adopt adversarial training to extract discriminative features. The uni-modal embeddings X mα are fed to three fully connected networks fm to extract specificity Sm,m ∈ {v, a, t}. In parallel, X mα are also passed through a reconstruction network, which is coupled with a contrastive learning network, followed by a generator G(·; θG) to derive the commonality Cm. Both specificity and commonality are then passed through linear layers with softmax activation in the discriminator D(·; θD) that is designed to distinguish which modality the inputs come from. The generator captures commonality C m by projecting different reconstructed embeddingX mγ into a shared latent subspace, ensuring distributional alignment across modalities. Consequently, this architecture encourages the generator G(·; θG) to produce outputs that challenge the discriminator D(·; θD) by obscuring the source modality of Cm. The generator and discriminator are jointly trained in a game-theoretic setup to enhance feature robustness against modality-specific biases. Both the commonality adversarial loss LC and the specificity adversarial loss LS are calculated by cross-entropy loss as,

where U ∈ {U v ,U t ,U a } represents the ground truth label corresponding to the discriminator's input.

In the shared subspace, it is advantageous to employ a unified representation of various modalities to facilitate multilabel classification. This representation is designed to eliminate redundant information and extract the elements common to the different modalities, thereby introducing a common semantic loss defined as,

(3)

where ˆ yυ,m τ is predicted with Cm and yυ τ is the ground-truth label. To encode diverse aspects of multi-modal data, we introduce an orthogonal loss Lorth that encourages the commonality Cm and specificity Sm subspaces to remain distinct by minimizing their overlap.

where ∥·∥F is Frobenius norm. Hereby, the objective of adversarial training Ladv is

where λa, λo and λc are trade-off parameters.

Multi-modal Feature Reconstruction. To reconstruct the features of any modality by leveraging information from the other modalities. We employ a reconstruction network that is composed of modality-specific encoders εm, decoders dm, and a two-level reconstruction process utilizing multi-layer linear networks g(·). Given input X mα from different modality, three encoders εm that consist of MLPs are utilized to project X mα into latent embeddingZm α within the latent space S z . Subsequently, three corresponding decoders dm transform these latent vectors into the decoded vectors ˜ X mα . At the first level of reconstruction network, the intrinsic vector ˜Dm that derived from contrastive learning network and semantic features ˜ X {v,t,a}\m α are concatenated to form the input, which is processed to produce X mβ used for the second-level reconstruction network. Hereby, the reconstruction network can be formulated as,

The obtained three embeddingX mα ,X mβ , andX mγ from three distinct feature spaces are fed into fully connected network followed by max pooling. We can formulate the reconstruction loss Lrec and classification loss Llsr cls as,

where ∥·∥F is the Frobenius norm, λα,β,γ are trade-off parameters, LB is the binary cross entropy (BCE) loss.

To capture the feature distributions of different modalities and use them to guide the restoration of incomplete modalities, intrinsic vectors ˜Dm obtained through a supervised contrastive learning network [Khosla et al., 2020] are incorporated into the reconstruction network. The encoders εm project input X mσ to contrastive embeddings Zm σ ,

σ ∈ {α, β, γ}. Given a contrastive embedding set Z = {Z{v,t,a} σ }, an anchor vector zi ∈ Z and assuming the prototype vector updated during the training process based on the moving average is µmj,k , where modality m ∈ {v, t, a}, label category j ∈ [ζ], label polarity k ∈ {pos, neg}, then the intrinsic vector ˜Dm can be derived from:

The loss of contrastive learning network is defined as,

where P(i) is the positive set, η ∈ R+ is a temperature parameter, and A(i) = Z \ {i}.

4.3 Stack Shuffle for Feature Augmentation

To construct more robust correlations among labels and model the complex interconnections between modalities and labels, we propose a multi-modal feature augmentation strategy that incorporates a stack shuffle mechanism. After obtaining the commonality and specificity representations, we perform sample-wise and modality-wise shuffling processes sequentially on a batch of samples. To strengthen the correlations between labels, we first apply a sample-wise shuffle. The features derived from Cm and Sm are split into k stacks along the sample dimension, with the top elements of each stack cyclically popped and appended to form new vectors. Next, a modality-wise shuffle is introduced to help the model capture and integrate information across different modalities. For each sample, features are divided into stacks along the modality dimension, and iterative pop-and-append operations are applied. Finally, the shuffled samples V are used to finetune the classifier cζ with the binary cross-entropy (BCE) loss.

Combing the Eq.(5), Eq.(7) ∼ Eq.(11) and Eq.(12), the final objective function L is formulated as,

where λr, λs are trade-off parameters.

5 Experiments

5.1 Experimental Setup

Datasets and Metrics. We evaluated RAMer on three benchmark datasets: MEmoR [Shen et al., 2020], a multiparty conversation dataset that includes personality, and CMU-MOSEI [Zadeh et al., 2018] and M 3ED [Zhao et al., 2022], which are dyadic conversation datasets that do not include personality information. The evaluation is conducted under the protocols of these datasets. For CMU-MOSEI and M 3ED, we employed four commonly used evaluation metrics: Accuracy (Acc), Micro-F1, Precision (P), and Recall (R). For MEmoR, we followed the benchmark's protocol and used Micro-F1, Macro-F1, and Weighted-F1 metrics.

Baselines. For the MEmoR dataset, we compare RAMer with multi-party conversation baselines, including MDL, MDAE [Ngiam et al., 2011b], BiLSTM+TFN [Zadeh et al., 2017], BiLSTM+LMF [Liu et al., 2018b], DialogueGCN [Ghosal et al., 2019], DialogueCRN [Hu et al., 2021], and AMER [Shen et al., 2020]. We further assess its robustness against recent dyadic models, including CARAT [Peng et al., 2023] and TAILOR [Zhang et al., 2022]. For the CMU-MOSEI and M 3ED datasets, we test three categories of methods. 1) Classic methods. CC [Read et al., 2011], which concatenates all available modalities as input for binary classifiers. 2) Deep-based methods. MLGCN [Chen et al., 2019], using Graph Convolutional Networks to map label representations and capture label correlations. 3) Multi-modal multi-label methods. These include MulT [Tsai et al., 2019] for cross-modal interactions, MISA [Hazarika et al., 2020] for learning modalityinvariant and modality-specific features, and methods like MMS2S [Zhang et al., 2020], HHMPN [Zhang et al., 2021a], TAILOR [Zhang et al., 2022], AMP [Ge et al., 2023], and CARAT [Peng et al., 2023].

5.2 Comparison with the State-of-the-Art

We present the performance comparisons of RAMer on the MEmoR, CMU-MOSEI, and M3ED datasets in Table 1, Table 2, and Table 3, respectively, with following observations.

1) On the MEmoR dataset, RAMer outperforms all baselines by a significant margin. While TAILOR achieves a high weighted-F1 score in the fine-grained setting, its overall performance is weaker due to biases toward frequent and easierto-recognize classes. RAMer consistently delivers strong results across all settings, demonstrating its ability to learn more effective representations. 2) On the CMU-MOSEI and M3ED datasets, RAMer surpasses state-of-the-art methods on all metrics except recall, which is less critical compared to accuracy and Micro-F1 in these contexts. 3) Deep-based methods outperform classical ones, highlighting the importance of capturing label correlations for improved classification performance. 4) Multimodal methods like HHMPN and AMP significantly outperform the unimodal ML-GCN, emphasizing the necessity of multimodal interactions. 5) Models optimized for dyadic conversations, such as CARAT, experience a notable performance drop in multi-party settings with incomplete modalities. In contrast, RAMer excels in both scenarios, achieving substantial improvements in Micro-F1 and Macro-F1 scores on the MEmoR dataset.

5.3 Ablation Study

To better understand the importance of each component of RAMer, we compared various ablated variants.

As shown in Table 4, we make the following observations:

• The specificity and commonality enhance MMER performance. Variants (1), (2), and (3) show lower Micro-

(a) w/oAdversarial Training (b) w/ Adversarial Training

(d) w/ RN and CLN

Table 3: Performance Comparison on the M3ED dataset

F1 than variant (11). This indicates that jointly learning specificity and commonality yields superior performance, underscoring the importance of capturing both modality-specific specificity and shared commonality.

• Contrastive learning benefits the MMER. The inclusion of loss functions Lscl in adversarial training leads to progressive performance improvements, as evidenced by

the superior results of (4).

• Feature reconstruction net benefits MMER. Variants (5), (6), (7) are worse than (11), and (8) shows an 0.045 decrease in Micro-F1, which indicates that feature reconstruction can improve model performance. When the entire reconstruction process is omitted, the performance of (8) declines even more compared to (6) and (7), confirming the effectiveness of multi-level feature reconstruction in achieving multi-modal fusion.
• Changing the fusion order leads to poor performance, variants (9) and (10) perform worse than (11). It validates the rationality and optimality of feature fusion.

5.4 Qualitative Analysis

Visualization of Learned Modality Representations

To evaluate the effectiveness of reconstruction-based adversarial training, we use t-SNE to visualize the commonality and specificity representations learned in the aligned CMUMOSEI dataset. In figure 3(a), without adversarial training, specificity and commonality are loosely separated, but their distributions overlap in certain areas, such as the lower-right corner. In contrast, Figure 3(b) shows clearer separation of commonality and specificity, forming distinct boundaries and effectively distinguishing emotions across modalities. Figure 3(c) demonstrates that without the reconstruction net (RN) and contrastive learning net (CLN), modality-wise embeddings are distinguishable, but emotion labels within the same modality remain intermixed. In contrast, Figure 3(d) shows clear separation across both modalities and emotions, indicating that the reconstruction-based module improves representation distinctiveness. Overall, RAMer accurately captures both the commonality and specificity of different modalities.

Visualization of Modality-to-Label Correlations

To explore the relationship between modalities and labels, we visualized the correlation of labels with their most relevant modalities. As shown in Figure 4, regardless of the presence of adversarial training, different emotion label is influenced by different modalities. For instance, surprise is predominantly correlated with the acoustic modality, while anger is primarily associated with the visual modality. This indicates that each modality captures the distinguishable semantic information of the labels from distinct perspectives.

Case Study

To demonstrate RAMer's robustness in complex scenarios, Figure 5 shows an example of MMER on the MEmoR dataset where specific target persons have incomplete modality signals. The top three rows display different modalities from a video clip, segmented semantically with aligned multimodal signals. Key observations include: 1) The target moment requires recognizing emotions for both the speaker (e.g., Howard) and non-speakers (e.g., Penny and Leonard). While the speaker typically has complete multi-modal signals, non-speakers often lack certain modalities. TAILOR, limited by missing modalities, yields partial predictions as its self-attention mechanism struggles to align labels with missing features. 2) Limitations of single or incomplete modalities. A single modality, such as text, is often insufficient for accurate emotion inference (e.g., only Howard's Joy is detectable from text alone). Although CARAT attempts to reconstruct missing information, it fails to capture cross-modal commonality, leading to incorrect predictions. 3)Inter-person interactions and external knowledge (e.g., personality traits) play important roles. Inter-person attention helps compensate for missing data, while personality-aware reasoning improves emotion inference across participants, highlighting the synergy between user profiling and emotion recognition. Experimental results demonstrate that RAMer achieves superior robustness and effectiveness in complex real-world scenarios.

6 Conclusion

In this paper, we proposed RAMer, a framework that refines multi-modal representations using reconstruction-based adversarial learning to address the Multi-party Multi-modal Multi-label Emotion Recognition problem. RAMer captures both the commonality and specificity across modalities using an adversarial learning module, with reconstruction and contrastive learning enhancing its ability to differentiate emotion labels, even with missing data. We also introduce a personality auxiliary task to complement incomplete modalities, improving emotion reasoning through modality-level attention. Furthermore, the stack shuffle strategy enriches the feature space and strengthens correlations between labels and modalities. Extensive experiments on three datasets demonstrate that RAMer consistently outperforms state-of-the-art methods in both dyadic and multi-party MMER scenarios.

Acknowledgments

We thank anonymous reviewers and scholars for their insightful comments and suggestions which helped us significantly improve an earlier draft of this paper. This paper is supported by NSF of China (62402409), Guangdong provincial project (2023CX10X008), Guangdong Basic and Applied Basic Research Foundation (2023A1515110545), Guangzhou Basic and Applied Basic Research Foundation (2025A04J3935), and Guangzhou-HKUST(GZ) Joint Funding Program (2025A03J3714).