Joint Multimodal Aspect Sentiment Analysis with Aspect Enhancement and Syntactic Adaptive Learning

1 Introduction

In the present context of information society, multimodal data plays a pivotal role in understanding human sentiments [Zhang et al., 2022a; Zhou et al., 2021; Yu et al., 2022]. Multimodal Aspect-Based Sentiment Analysis (MABSA), an essential AI research area, combines text, image, audio, or other modalities to accurately identify and interpret sentiments, transcending the limitations of single-modal analysis and offering a comprehensive reflection of complex sentiment expressions [Xu et al., 2019; Zhu et al., 2023; Das and Singh, 2023]. The proliferation of social media have identified two main challenges in multimodal sentiment analysis. 1) How to adequately consider aspect relevance between multimodal input data? As shown in Fig. 1, the aspect "Oklahoma City Thunder" in the text is consistent with "region2" in the image, and "region3" in the image is the celebratory gesture of "thump" in the text. This suggests that the sentiment described in text can be reinforced or modified by the object in the image. By analyzing their interconnections, the sentiment or characteristics of the aspect can be captured and understood more accurately. 2) How to effectively capture the syntactic structure and semantic association between different words in the text? The arrows in Fig. 1 show the dependencies of different nodes. Capturing semantic relationships between different nodes helps to understand text semantics and perform sentiment analysis.

∗Contact Author

To address the above issues, we propose a joint multimodal aspect sentiment analysis method based on aspect enhancement and syntactic adaptive learning, which fully considers the syntactic and semantic relationships between aspects and different nodes. First, we utilize RoBERTa [Liu et al., 2019] and ViT [Dosovitskiy et al., 2020] to encode text and image respectively. And we propose an aspect enhancement pretraining task to enable the model pay more attention to the aspect in text and image. Secondly, in order to fully consider the local and global information between nodes, we construct the syntactic dependency graphs of first-order nodes, secondorder nodes and even global syntactic structure. In addition, we design a multi-channel adaptive graph convolutional network to mine the interaction features between different dependency graphs and multimodalities. Finally, the detector is used to predict the results of JMASA, MATE and MASC tasks. Our main contributions are as follows:

• We propose an AESAL model that can fully consider aspects and the syntactic associations between different nodes for JMASA, MATE and MASC tasks.
• We introduce an aspect enhancement pre-training task that constructs aspect-based positive and negative examples to enhance the sensitivity of the model to aspect term in multimodal data.
• We design the syntactic adaptive learning mechanism, which consists of syntactic dependency graph and multichannel adaptive graph convolutional network. The syntactic dependency graphs based on different syntactic distances are used to synthesize global and local associations. The multi-channel adaptive graph convolutional network is used to enhance the node representation of the dependency graph.
• Experimental results on two benchmark datasets show that the AESAL model achieves the state of-the-art performance.

2 RelatedWork

In the field of multimodal sentiment analysis, some of the pre-training efforts to study specific tasks have achieved impressive results. For example, [Liu et al., 2023a] introduce a unified alignment pre-training framework into the vanilla pretrain-finetune pipeline, that has both instance and knowledge-level alignment. [Ling et al., 2022] propose a task-specific Vision-Language Pre-training framework for MABSA (VLP-MABSA), which is a unified multimodal encoder-decoder architecture for all the pre-training and downstream tasks. [Liu et al., 2023b] propose an entityrelated unsupervised pre-training with visual prompts for sentiment analysis. [Li et al., 2021b] adopt supervised contrastive pre-training on large-scale sentiment annotated corpora retrieved from in-domain language resources. Unlike the above work, we design aspect enhancement pre-training task to improve the learning ability of the model for aspect term.

Recently, various approaches have been proposed to model the semantic relations between aspects and their contexts to capture opinion expressions. [Zhang et al., 2022b] propose SSEGCN, which can not only learn the aspect-related semantic correlation, but also learn the global semantics of the sentence. [Liang et al., 2022] give a graph convolutional network construction scheme for graph based on dependency tree and affective commonsense knowledge. [Tian et al., 2021] offer a method for explicitly utilizing dependency types. [Li et al., 2021a] provide a dual graph convolutional network (DualGCN) model that considers the complementarity of syntax structures and semantic correlations simultaneously. However, these approaches usually ignore the relationship between syntactic distance and syntactic semantics. To address this issue, we construct syntactic distance-based syntactic dependency graph to complement the semantic information of sentences.

Based on this, several works have extended the GCN and GAT models with syntactic dependency tree and developed several excellent models based on multimodal sentiment analysis [Zhang et al., 2022b; Tian et al., 2021; Zhao et al., 2020; Chen et al., 2020]. [Zhou et al., 2023] propose an Aspectoriented Method (AoM) which can detect aspect relevant information from perspectives of both semantics and sentiment. [Yang et al., 2022] design a multitask learning architecture named Cross-Modal Multitask Transformer (CMMT) for the End-to-End sentiment analysis. [Ju et al., 2021] raise a multi-modal joint learning approach with auxiliary crossmodal relation detection for multi-modal aspect-level sentiment analysis. [Ling et al., 2022] present a unified multimodal encoder-decoder architecture for MABSA. Inspired by the above work, we design multi-channel adaptive graph convolutional network for multimodal sentiment analysis to maintain the uniqueness of each modality while fusing the correlations between different modalities.

In this paper, we propose a joint multimodal method to sentiment analysis based on aspect enhancement pre-training and syntactic adaptive learning. It enables the model to fully consider the aspect terms in the input, while effectively utilizing the dependencies between syntaxes.

3 Methodology

The overview of AESAL is given in Fig. 2. In this section, we describe the AESAL model, which consists of five main components: feature extraction, aspect enhancement pre-training, syntactic dependency graph, multi-channel adaptive graph convolutional network and prediction. Next, the components of AESAL are described separately in the remaining sections.

3.1 Feature Extractor

The multimodal data contains text sequence T = {t1, t2, ..., tn} and image V . Our goal is to extract all aspect terms A = {a1, a2, ..., am} with the corresponding sentiment polarity S = {s1, s2, ..., sm}, where n represents the length of text, m represents the number of aspects contained in the text, ai represents the ith aspect term, and si represents the sentiment polarity corresponding to the ith aspect term. In addition, si ∈ {POS,NEU,NEG}, where POS, NEU and NEG standing for the sentiment of positive, neutral and negative, respectively.

We use RoBERTa [Yu and Jiang, 2019] as text encoder to extract the hidden context representation Ht = {ht1 , ht2 , ..., htn }, and utilize ViT [Hu et al., 2019] as image encoder to extract the hidden image representation Hv = {hv1 , hv2 , ..., hvn }. In addition, the image encoding needs to realize the alignment with the text shape by MLP. Here, Ht and Hv ∈ Rn×d , d represents the hidden state dimension.

3.2 Aspect Enhancement Pre-training

In order to enhance the sensitivity of the model to aspect while extracting features, we designed the aspect enhancement pre-training task, as Fig. 3. For any sample containing (T, V, A, S) quaternion, we use the special symbol "[Mask]" to replace the aspect A in the text sequence T to get the aspect-free text sequence N, as shown in Equation (1). We consider (T, V ) as positive samples and (N, V ) as negative samples to construct a new dataset (S, V,L) to improve the model learning aspect. Here, S ∈ {T,N}, S stands for texts, V stands for images, L stands for positive and negative sample labels, and L = 1 means that the aspect terms in the image match the text, otherwise 0.

In the pre-training period, we first use the feature extraction module to obtain text embedding and image embedding. Then, the text embedding and image embedding are concatenated, and the probability distribution associated with the text and image is output through the softmax layer. Finally, we use cross-entropy loss to train our aspect enhancement pretraining task.

3.3 Syntactic Dependency Graph

Traditional syntactic dependency graph are constructed according to the connection relationship of each node in the syntactic dependency tree, which focuses only on local relationship and does not consider the deeper relationship of syntax. In order to effectively utilize the global and local dependencies between different nodes, we construct the dependency graphs based on syntactic distance, which consider both syntactic direct and indirect associations.

In this section, we use spacy 1 to obtain syntactic dependency tree, as shown in Fig. 4. A syntactic dependency tree is considered as an undirected graph, where nodes represent words and edges represent the existence of a direct relationship between two words in the syntactic structure. First, we define the distance between any two nodes i and j in the graph as d(i, j). Assuming that there are a total of p paths between nodes i and j, we use the Breadth First Search (BFS) algorithm to find the shortest path dmin of nodes i and j, as in Equation (2). Secondly, We believe that the relationship between two distant nodes is relatively weak and there is no need for aggregation. Thus we filter the information with syntactic distance greater than k, and keep the information with syntactic distance less than or equal to k. Thus, we give more weight to the edges between two nodes with closer distance and get the dependency matrix M k , as shown in Equation (3).

Considering the different perceptual scopes at different syntactic distances, we set different thresholds for multiple (h) dependency matrices to focus on the global correlation information of the sentence and different degrees of local correlation information in Equation (4).

1 https://spacy.io/

M 1 captures first-order nodes dependencies, M 2 captures second-order nodes dependencies, and M h captures global nodes dependencies.

3.4 Multi-Channel Adaptive Graph Convolutional Network

To preserve uniqueness and capture correlations across modalities and syntactic distances, we design a multi-channel adaptive graph convolutional network. It adaptively adjusts the adjacency matrix based on syntactic dependency graphs at different distances (Sec. 3.3), updating multimodal information in parallel to focus more on aspects and viewpoints.

We use the multi-head cross-attention mechanism to construct the text-to-text multi-head attention matrix P t2t , textto-image multi-head attention matrix P t2v , and image-to-text multi-head attention matrix P v2t respectively, and obtain the multi-channel attention matrix P = (P t2t , P t2v , P v2t ). It aims to capture the interrelationships that exist among textto-text, image-to-image, and image-to-text in a multidimensional way.

First, we define the attention matrix function AF as shown in Equation (5). We define the multi-head attention matrix function MAF in Equation (6), and get the multi-channel attention matrix P in Equation (7) to match the multi-head dependency matrix M.

Here, Q and K represent query vector and value vector respectively. W Q and W K are learnable parameters, dk represents the dimension of K.

Second, we use the multi-head dependency matrix M to mask the multi-channel attention matrix P with different syntactic distances to obtain the neighbor matrix A = (At2t ,At2v ,Av2t ) as shown in Equation (8). At this point, our adjacency matrix A contains syntactic dependencies at different distances and contains learnable parameters. Thus, we can both realize multi-level information aggregation from local to whole at the syntactic level and adaptively adjust the filtering of aspect-independent information.

where ⊙ denotes element-wise multiplication.

Finally, we feed the text-hidden state Ht , image-hidden state Hv with the adjacency matrix A into the graph convolutional network to obtain features Hgt2t , Hgt2v and Hgv2t as in Equation (9). And the convolutional features Hgt2t , Hgt2v , Hgv2t take the mean value as our multimodal fusion feature Hfusion , as in Equation (10).

3.5 Prediction

We use a decoder consisting of a two-layer linear neural network with an activation function for task prediction, and a loss function using cross-entropy loss with the following equation:

where W1,W2, b1, b2 are the learnable parameters, ˆ y is the subtask prediction result, y is the subtask true label, L is the final loss.

4 Experiment

We compare our model with numerous methods on three tasks, including JMASA, MATE and MASC.

4.1 Experimental Settings

Datasets. We conduct experiments on two public Twitter datasets [Yu and Jiang, 2019] (i.e., Twitter-2015 and Twitter2017). As shown in Table 1, sentences with multiple aspects make up a significant portion of both datasets. We use these datasets for aspect enhancement pre-training and subsequent experiments.

Implementation Details. We implement our method under Linux system, CUDA version 10.2, Pytorch version 1.12.0, Python version 3.9, and NVIDIA GeForce RTX 3090. In addition, we set the Learning rate to 2e-5, the dropout to 0.1, hidden size to 768.

Evaluation Metrics. We evaluate the performance of our model on JMASA task and MATE task by Micro-F1 score (F1), Precision (P) and Recall (R), while on MASC task we use Accuracy (Acc) and F1 following previous studies.

4.2 Baselines

We compare AESAL with four types of methods listed below. Methods for textual ABSA. 1) SPAN [Hu et al., 2019] is a span-based extraction-categorization framework that extracts multiple opinions directly from sentences and then categorizes them. 2) D-GCN [Chen et al., 2020] performs the task following the sequence tagging paradigm and models the dependencies between input words with an appropriate architecture. 3) BART [Yan et al., 2021] is a pre-trained model to solve seven ABSA subtasks.

Methods for JMASA. 1) UMT-collapse [Yu et al., 2020], OSCGA-collapse [Wu et al., 2020b] and RpBERT-collapse [Sun et al., 2021] use the same visual feed to collapse the markers. 2) UMT+TomBERT, OSCGA+TomBERT are two pipeline methods. 3) JML [Ju et al., 2021] is a multimodal joint approach to simultaneously handle the aspect terms extraction and sentiment classification. 4) VLPMABSA [Ling et al., 2022] is a unified multimodal encoderdecoder architecture for all the pre-training and downstream tasks. 5) CMMT [Yang et al., 2022] is a multi-task learning framework to extract aspect-sentiment pairs from a pair of sentence and image. 6) AOM [Zhou et al., 2023] is an aspectoriented network to mitigate the visual and textual noises from the complex image-text interaction.

Methods for MATE. 1) RAN [Wu et al., 2020a] is a novel approach which uses object and text features as the input on MATE task. 2) UMT [Yu et al., 2020] is a unified architecture to alleviate the bias of the visual context in multimodal named entity recognition. 3) OS-CGA [Yan et al., 2021] is an object-aware neural model that combines visual and textual representations into entities predicting.

Methods for MASC. 1) ESAFN [Yu et al., 2019] is an entity-sensitive attention and fusion network for MASC. 2) TomBERT [Yu and Jiang, 2019] is a target-oriented multimodal sentiment classification method. 3) CapTrBERT [Khan and Fu, 2021] is a two-stream model that translates images in input space.

4.3 Main Results

In this section, we show the excellent performance of our method compared with SOTAs.

Performance on JMASA. The results of JMASA are shown in Table 2. First, our model far outperforms all text-based models, which means that utilizing multimodal information is beneficial for the JMASA task. Second, our model outperforms other multimodal aspect sentiment analysis methods on every metric. In particular, compared to the suboptimal model (AoM), the precision improves by 0.8%, recall improves by 1.1%, and F1 score improves by 0.9% on the Twitter-2015 dataset; on Twitter-2017, P, R, and F1 increased by 1%, 3.8%, and 2.3%, respectively. This demonstrates the effectiveness of our model on the JMASA task.

Performance on MATE. Table 3 shows the performance of MATE. Our model still achieves optimal results. Specifically, on the Twitter-2015 dataset, compared to the suboptimal result on AoM, P improves by 5.6%, R improves by 2.7%, and F1 improves by 4.2%; on the Twitter-2017 dataset, compared to the suboptimal result on CMMT, P improves by 0.9%, R improves by 2.5%, and F1 improves by 1.6%. It proves that AESAL is the most capable of detecting aspect term from Table 4: Results of different methods for MASC. Our model AESAL achieves the current optimum of MASC on a slightly larger dataset. * denotes the results from [Zhou et al., 2023].

image and text.

Performance on MASC. As shown in Table 4, AESAL performs best on Twitter-2017, with 2.4% improvement in Acc and 0.9% improvement in F1 compared to the second best AoM. AoM performs slightly better than us on Twitter-2015, outperforming us by 0.1% on Acc and 0.7% on F1. This may be related to the number of aspect terms in the dataset. There are fewer aspect terms in the Twitter-2015 data than in Twitter-2017, so our aspect-enhanced pre-training may not have been utilized to its maximum capacity.

4.4 Ablation Study

In this section, we compare the variants of the AESAL in terms of the following five components to demonstrate the effectiveness of AESAL framework: Image (Img), Aspect Enhancement Pre-training (AE), Indirect Relation Dependency Graph (IR), Syntactic Adaptive Learning (SAL), Text2Text Convolution Channel (T2T).

W/o Img is a variant of AESAL that removes image informa-

tion from multimodal data and utilizes only text information.

W/o AE is a variant of AESAL without aspect enhancement pre-training module.

W/o IR is a variant of AESAL without the syntactic dependency graph of indirect relations, using only the syntactic dependency graph of neighboring nodes.

W/o SAL is a variant of AESAL without syntactic adaptive learning and utilizes only cross-attention to fuse multimodal information for prediction.

W/o T2T is a variant of AESAL without the convolution channel of Text2Text and considers only image-text, textimage related channels.

The results of the ablation experiments for the JMASA, MATE and MASC tasks are given in Table 5. Based on the JMASA task, the variant without AE and the variant without IR have the worst results. This shows the importance of the aspect enhancement pre-training module and the syntactic dependency graph of indirect relation in our model. Based on the MATE task, the variant without SAL and the variant without IR have the worst results. It thus illustrates the importance of syntactic adaptive learning and indirect relations between words for the MATE task, and they help the model to better understand and learn syntactic dependencies. Based on the MASC task, the variant without the convolution channel of Text2Text has the lowest results, reflecting the importance of textual information in the MASC task. Generally speaking, textual information expresses sentiments more accurately, while images only serve an auxiliary role.

4.5 Case Study

To further demonstrate the effectiveness of AESAL, we present two test examples with predictions from different methods. As shown in Fig. 5, for example (a), although D-GCN can accurately detect two aspect terms of groundtruth, it gives the wrong sentiment prediction of aspect term "lionelmessi". This may be due to the lack of syntactic adaptive learning mechanism that fail to adequately capture syntactic and semantic information. However, OSCGA-collapse and our AESAL methods correctly predict the aspects and its corresponding polarity. For example, in (b), OSCGAcollapse incorrectly detects the aspect as "ridgetown dhs tractor day", but the correct aspect term is "ridgetown dhs". This is mainly because of the lack of aspect enhancement pretraining task. In both cases, our AESAL model correctly extracts all aspects and categorizes sentiments, demonstrating the excellence of AESAL on the MATE, MASC, and JMASA tasks.

5 Conclusion

In this paper, we propose a joint multimodal aspect sentiment analysis method based on aspect enhancement and syntactic adaptive learning. First, we construct the aspect enhancement pre-training task to enable the model adequately learn the aspect of the multimodal input data. Second, we design different syntactic dependency graphs of first-order nodes, secondorders nodes and even global nodes simultaneously, so that the model captures the global and local information in the text. After that, we enhance the node representation by utilizing the multi-channel adaptive graph convolutional network. Finally, we execute the MATE, MASC, and JMASA tasks, respectively. Experimental results on two widely available datasets demonstrate the effectiveness of our method.

Acknowledgments

This work was supported in part by the National Science Foundation of China under Grant 62072365, in part by the Science Foundation of Distinguished Young Scholars of Shaanxi under Grant 2022JC-48, in part by the Aviation Science Foundation 2023M071070002, in part by the Key Research and Development Program of Shaanxi under Grant 2022GY-332, 2023-YBGY-230 and 2024GX-YBXM-533, in part by the Innovation Capability Support Plan of Shaanxi under Grant 2022PT-33, in part by the Xi'an Science and Technology plan Key industrial chain technology research project under Grant 23ZDCYJSGG0007, and in part by the Xi'an Science and Technology plan Key industrial chain, key core technology research project under Grant 23LLRH0022, and Qinchuangyuan Construction of Two Chain Integration Important Project 23LLRHZDZX0006. Any opinions, findings, and conclusions expressed here are those of the authors and do not necessarily reflect the views of the funding agencies.