Sentiment-enhanced Multi-hop Connected Graph Attention Network for Multimodal Aspect-Based Sentiment Analysis

1 Introduction

With the rapid development of internet, multimodal data are playing an increasingly important role in social events and user attitude expression. Multimodal aspect-based sentiment analysis (MABSA) has garnered increasing attention from researchers due to its ability to accurately discern the sentiment inclinations of various aspects within multimodal data. It typically encompasses three subtasks: joint multimodal aspectbased sentiment analysis (JMASA), multimodal aspect term extraction (MATE), and multimodal aspect sentiment classification (MASC).

∗Contact Author

Early MABSA studies [Ju et al., 2021; Yu et al., 2022; Li et al., 2024b] enhanced the integration of cross-modal information through modality alignment techniques. For instance, JML [Ju et al., 2021] utilized cross-modal relation detection, while DTCA [Yu et al., 2022] adopted a dualencoder transformer with cross-modal alignment. However, these methods overlooked the fine-grained associations between aspects and opinions. To address this issue, AoM [Zhou et al., 2023] introduced aspect-oriented attention modules and graph convolutional networks, and AETS [Zhu et al., 2025] developed an aspect-enhanced module to increase sensitivity to aspects, but their performance suffers in scenarios with sparse data. Current research improves sentiment detection by parsing the syntactic structure of text. AESAL [Zhu et al., 2024] proposed an adaptive syntactic learning mechanism relying on word distance to evaluate semantic relationships, overlooking the differences in syntactic structure connectivity, which may affect the understanding of text semantics.

While these studies have demonstrated promising performance, our analysis reveals two challenges in MABSA: 1) In multimodal multi-sentiment scenarios, the sentiment polarity of aspect terms can be nuanced and complex, challenging models to make accurate judgments with limited data. As shown in Fig. 1 a⃝, there are three aspects "Barcelona", "La Liga" and "Suarez", with corresponding sentiments of "positive", "neutral" and "negative". However, due to the influence of "Sink", "Suarez" is misjudged as negative. Integrating visual information (Fig. 1 b⃝) in pretraining tasks for sentiment enhancement can significantly improve the model's ability to accurately distinguish sentiments, especially in data-limited multi-sentiment task. 2) Differences in syntactic relationships revealed by syntactic connectedness have different effects on MABSA. As shown in the syntactic dependency tree in Fig. 1 c⃝, syntactic connectivities like "Crowned-Champions", "ChampionsLiga" and "Champions-Sink" are involved in multiple syntactic relationships, demonstrating strong syntactic connectivity and encompassing significant connections with aspect and sentiment words. This intricate syntactic structure is crucial for the precise identification of aspects and sentiment analysis. In contrast, the connectivity of "Liga-La", "Sink-Suarez" is relatively lower, with the syntactic relationships being of lesser importance. Different syntactic relationships have a certain impact on the understanding of the text. Therefore, it is crucial to pay full attention to the importance of multiple sentiments and syntactic relations for MABSA task.

To address these issues, we propose a sentiment-enhanced multi-hop connected graph attention network (MCG) for MABSA. The method enhances the sensitivity of model to sentiments and fully considers the importance differences driven by syntactic structure through a multi-hop connected graph attention mechanism. Specifically, we firstly construct sentiment-enhanced pre-training based on positive and negative samples to increase the diversity and complexity of training samples, thereby improving the ability to distinguish multiple sentiments. Secondly, we constructed a multi-hop connected syntactic dependency graph to capture the importance of different syntactic relationships, and utilized the attention mechanism of the multi-hop connected graph to effectively highlight key relationships in the syntactic structure. Our contributions are as follows:

• We are the first to consider the impact of syntactic relation differences based on connectivity for MABSA task and propose a sentiment-enhanced multi-hop connected graph attention network that analyzes the importance of key syntactic relations.
• We design a sentiment-enhanced pre-training task that expands the diversity of training samples and enhances the discriminative ability for multiple sentiments.
• We introduce a multi-hop graph attention mechanism to capture sentence structure and semantics by constructing syntactic dependency graphs and focusing on important syntactic relations for deeper text understanding.
• Experimental results on three tasks (JMASA,MATE and MASC) across two benchmark datasets indicate that MCG achieves state-of-the-art performance.

2 RelatedWork

Multimodal Aspect-Based Sentiment Analysis is a task that is both challenging and requires multiple processing, necessitating models that can deeply understand multimodal data and effectively extract sentiment cues [Ye et al., 2022; Zhang et al., 2023]. Early research primarily relied on attention mechanisms to achieve alignment between different modalities. For instance, JML [Ju et al., 2021] optimizes the utilization of visual information by constructing an auxiliary text-image relation detection module and employed a hierarchical framework to facilitate multimodal interaction between MATE and MASC. DTCA [Yu et al., 2022] introduces two auxiliary tasks to enhance cross-attention performance and aligned text and image modalities by minimizing the Wasserstein distance between them. VLP-MABSA [Ling et al., 2022] innovatively transformes MABSA into a text generation task, incorporating, in addition to conventional masked language modeling and masked region modeling tasks, pretraining tasks such as text aspect-opinion extraction, visual aspect-opinion generation, and multimodal sentiment prediction, to more finely identify aspects, opinions, and their crossmodal consistency.

Existing research focuses on aspect enhancement and syntax mining. CMMT [Yang et al., 2022] uses two auxiliary tasks to learn intra-modal representations for aspects or sentiment perception and introduces a text-guided cross-modal interaction module to dynamically control the contribution of visual information to each word's representation. AoM [Zhou et al., 2023] designs an aspect-oriented attention module to select text labels and image patches relevant to aspect semantics simultaneously, incorporating sentiment embeddings into aspect semantics. Atlantis [Xiao et al., 2024] enhances multimodal data with visual aesthetic attributes. AESAL [Zhu et al., 2024] develops an aspect enhancement module to learn aspect correlations in multimodal input data and a syntaxadaptive learning mechanism for syntactic relationships.

Despite the significant progress made, previous studies have overlooked the handling of diverse sentiments and different syntactic relationships. To address these issues, this paper introduces the MCG model, which enhances the sentiment discriminative capability of model through contrastive sentiment pre-training and employs the multi-hop connected graph attention mechanism to capture the significance of different syntactic relationships within syntactic structures.

3 Method

In this section, we first introduce the task formulation, followed by a detailed description of the proposed sentimentenhanced multi-hop connected graph attention network (MCG). The overall framework is shown in Fig. 2, and the network consists of the following components: feature extraction, sentiment-enhanced pre-training, multi-hop connected syntactic dependency graph, multi-hop connected graph attention, and the prediction.

3.1 Task Formulation

We assume the dataset D = {(Ti, Vi,Ai, Si)Ki=1 } , which contains K samples. Each sample x ∈ D includes the text T = {t1, t2, · · · , tn}, the image V ∈ R3×H×W , aspect terms A = {a1, a2, · · · , am} and their corresponding sentiments S = {s1, s2, · · · , sm}, where n represents the number of

Multi-hop Connected Syntactic Dependency Graph

words, 3, H, W denote the number of channels, height, width of the image, and m represents the number of aspects and sentiments. si ∈ {POS,NEU,NEG}, POS,NEU,NEGdenote positive, neutral, and negative, respectively. We define the input and output for the three sub-tasks MATE, MASC, and JMASA as follows:

3.2 Feature Extraction

Given the superior performance of RoBERTa [Liu et al., 2019] for text representation and ViT [Dosovitskiy et al., 2020] for visual representation, we utilize RoBERTa and ViT to encode text and images, respectively. Specifically, we append the special tokens [cls] and [sep] to the beginning and end of the text to indicate the start and end of a sentence, with [cls] also used to mark the beginning of the image. In Equations (1) and (2), we feed the text and image into RoBERTa and ViT to derive the hidden layer states and for text and image. During this process, we use an MLP to adjust the dimensions of image to align with the text representation.

Ht ,Hv ∈ Rn×d , n denotes the number of words and d denotes the hidden state dimension.

3.3 Sentiment-enhanced Pre-training

In order to improve the discriminative capability of model in the domain of multiple sentiments recognition, we devise a contrastive sentiment-enhanced pre-training task, with the specific architecture depicted in Fig. 3.

For each training sample x, we employ the following strategy for sample processing. First, we append the aspect term ai and its corresponding sentiment label si to the end of the text, forming a positive training sample P by inserting a special delimiter "<s>". Next, we randomly select a non-real sentiment label si from a collection of labels, which differs from the real sentiment label si (where si ∈ {POS,NEU,NEG} and si ̸= si), and combine it with the aspect term ai to generate a negative training sample N. During this process, we treat (P, V ) as a positive training pair and (N, V ) as a negative training pair. Subsequently, the positive and negative training pairs are input into the feature extraction module to obtain text embeddings and image embeddings. These two embedding vectors are then concatenated, and a cross-entropy loss function L∫ is employed to increase the distance between the positive and negative training pairs as in Equation (3).

B represents the true label of the sample pair, and ˆB is the probability that the model predicts the sample pair as positive sample.

The method of randomly selecting a non-real sentiment label as a negative example helps to balance positive sentiment information and noise (including irrelevant or misleading information), thereby encouraging the model to focus on the key differences between positive and negative training pairs. This design strategy significantly improves the performance in two aspects: Firstly, as a data augmentation technique, it increases the complexity and diversity of the dataset. Secondly, by enhancing the discrimination between positive and negative training pairs, it enhances the ability of model to discriminate between ambiguous sentiments and multiple sentiment categories.

3.4 Multi-hop Connected Syntactic Dependency Graph

The syntactic dependency graph highlights syntactic relationships in sentences. AESAL [Zhu et al., 2024] evaluates these relationships by node proximity, suggesting that closer nodes are more significantly associated. However, in semantic comprehension, nodes with more connections are often more crucial [Borge-Holthoefer and Arenas, 2010], due to their involvement in multiple syntactic relationships and higher connectivity. Thus, we build a multi-hop syntactic dependency graph to analyze connectivity, capturing variations in syntactic relationships for a deeper semantic understanding of text.

We represent the significance of different syntactic relationships in the syntactic dependency tree through the connectivity of the graph. Specifically, we first use the spaCy 1 library to obtain the syntactic dependency tree and convert it into an undirected graph G. The adjacency matrix M ∈ Rn×n of graph G represents its connectivity, where n is the number of words. M k represents the number of paths of length k, and M k ij represents the number of paths of length k from node i to node j. The proof is given in Theorem 1.

Theorem 1. For any graph G with adjacency matrix M, the element M k ij represents the number of paths of length k from vertex i to vertex j.

We perform kth power operations on the syntactic dependency graph and normalize it according to the following Equations (4) and (5) to obtain the weights of syntactic relationships:

1 https://spacy.io/

Here, I is the identity matrix, D is the degree matrix of M + I, and ˜D is the diagonal matrix. ˜ M 1 , ˜ M 2 , ..., ˜ M k represent the weights of syntactic dependency relationships from first-hop to kth-hop, ˜ M denotes the weight of the multihop syntactic dependency graph. Through experiment, we found that the length of syntactic relations typically ranges between 1 and 8, which corresponds to k ∈ {1, 2, ..., 8}.

As shown in Fig. 4, we present the normalized graph of M 1 -M 8 for the example in Fig. 1 a⃝. In the syntactic dependency tree, the importance of different nodes can be measured by their connectivity in the graph. It is observed that syntactic relationships like "Champions-Champions", "CrownedChampions", "Sink-Champions" and "Liga-Champions" have larger values. This is because words like "Champions", "Crowned", "Sink" and "Liga" are involved in more syntactic relationships, resulting in stronger connectivity. In our method, by calculating the connectivity of different orders, we can more accurately capture these important syntactic relationships.

3.5 Multi-hop Connected Graph Attention

To effectively highlight the role of key syntactic relationships in the syntactic structure, particularly those that play a core role in aspect and sentiment expression, we propose a multihop connected graph attention mechanism based on the multihead cross-attention mechanism. This mechanism enhances the focus on local contexts while also considering the flow of information in a broader context, aiding in the capture of deep semantic associations.

We employ a multi-head cross-attention mechanism to obtain a multi-head attention matrix MA for text and image, capturing their interactive relationships. We first define the single-head cross-attention matrix function (CF ) in Equation (6). In Equations (7) and (8), we define the multi-head crossattention matrix function (MCF) with k heads, and obtain a multi-head attention matrix MA that matches the ˜ M.

where Q,K denotes the query and key vectors. WQ and WK are learnable parameters, and dk denotes the dimension of k.

After that, in Equation (9), we use the multi-hop syntactic dependency graph weights ˜ M in combination with the multihead attention matrix MA to obtain the multi-hop connected graph attention MC. In this way, the model enhances the refinement of the unimodal intra-features while also optimizing the interactions of the cross-modal features.

where ⊙ denotes element-by-element multiplication.

Finally, as shown in Equation (10), we multiply the multihop connected graph attention MC with the value vector to obtain the multi-modal fusion feature Hf with multi-hop syntactic dependency graph information.

Here, Wv is the learnable parameter.

3.6 Prediction

As shown in Equations (11) and (12), we use a two-layer MLP as the predictor and use relu as the activation function and cross-entropy loss as the objective function.

where W1,W2,b1,b2 are the learnable parameters, ̂ y is the predicted outcome, and y is the true label.

4 Experiment

In this section, we verify the validity and superiority of our proposed MCG through comparative experiments and ablation experiments on JMASA, MATE and MASC tasks.

4.1 Experiment Data and Setup

Dataset: we use two multimodal benchmark datasets, Twitter-2015 and Twitter-2017 [Yu and Jiang, 2019], for our experiments. These two Twitter datasets collect user posts published on Twitter during 2014-2015 and 2016-2017, respectively. Table 1 summarizes the datasets.

Experiment Setup: Our experiments are implemented in the PyTorch framework using NVIDIA 3090 GPUs, with the learning rate set to 2e-5, the hidden layer dimension to 768, and the dropout set to 0.1.

Evaluation Metrics: On the JMASA task and the MATE task, we evaluate our model using Micro-F1 (F1), Precision (P) and Recall (R). And on the MASC task, we use Accuracy (Acc) and Micro-F1 (F1) following previous studies [Zhu et al., 2024; Zhou et al., 2023].

4.2 Baselines

We compare MCG with four types of methods.

Methods for textual ABSA. 1) SPAN [Hu et al., 2019] identifies opinion targets with their sentiments. 2) D-GCN [Chen et al., 2020] utilizes GCN [Wu et al., 2020a] to model syntactic dependencies. 3) BART [Yan et al., 2021] addresses seven ABSA subtasks.

Methods for JMASA. 1) UMT-collapse [Yu et al., 2020], OSCGA-collapse [Wu et al., 2020b] and Rpbert-collapse [Sun et al., 2021] use the same visual input to collapse individual tokens. 2) UMT+TomBERT, OSCGA+TomBERT are two pipelines that combine UMT, OSCGA and TomBERT [Yu and Jiang, 2019]. 3) JML [Ju et al., 2021] introduces hierarchical text-image relation detection with auxiliary modules for joint MATE-MASC optimization. 4) VLP- MABSA [Ling et al., 2022] is a multimodal framework employing five dedicated pretraining tasks to model aspect features, opinion expressions and cross-modal alignment. 5) CMMT [Yang et al., 2022] is a cross-modal learning approach that regulates multimodal information interaction through gating mechanisms. 6) AoM [Zhou et al., 2023] is an aspect-oriented method that simultaneously detects aspect-relevant semantic information and sentiment features from multimodal inputs. 7) Atlantis [Xiao et al., 2024] is a framework that augments multimodal representations using visual aesthetic characteristics. 8) AESAL [Zhu et al., 2024] is a method featuring an aspect correlation learning module and a syntax-adaptive mechanism for multimodal input processing. 9) GLM-4VPlus [Hong et al., 2024], Llama-3.2-11B-Vision-Instruct (Llama-3.2) [AI@Meta, 2024], and llama3-llava-next-8b-hf (LLaVA-NeXT) [Li et al., 2024a], all advanced large language models(LLMs), were meticulously directed to perform specific tasks in our experiments through carefully crafted prompt information.

Methods for MATE. 1) RAN [Wu et al., 2020b] specifically emphasizes aligning text with object regions. 2) UMT [Yu et al., 2020] effectively uses text-based entity span detection as an auxiliary task. 3) OS-CGA [Wu et al., 2020b] primarily focus on aligning visual objects with entities.

Methods for MASC. 1) ESAFN [Yu et al., 2019] imploys LSTM networks to perform sentiment analysis at the entity level. 2) TomBERT [Yu and Jiang, 2019] is a targetoriented multimodal utilizes BERT architecture to generate aspect-aware text representations. 3) CapTrBERT [Khan and Fu, 2021] translates images into text and constructs an auxiliary sentence for fusion, enhancing cross-modal interaction through generated captions.

4.3 Experiment Results

In this section, we show the excellent performance of MCG.

Performance on JMASA. As shown in Table 2, first, MCG significantly surpasses all text-based models, indicating that detecting richer visual and textual information is effective. Second, MCG also outperforms all multimodal methods across various metrics. In particular, compared to the secondbest AESAL, the P value increased by 2.6% on Twitter-2015 and by 7.8% on Twitter-2017. These results demonstrate that the identification of syntactic relationships based on syntactic connectivity is more effective than AESAL that rely solely on distance. Last, compared to LLMs, which requires aspect identification before sentiment determination, MCG can more accurately recognize both aspect and sentiment simultaneously, significantly reducing the task burden and demonstrating stronger task coherence.

Performance on MATE. As shown in Table 3, MCG achieved satisfactory results on the MATE task. First, MCG achieved optimal results across all metrics. These results demonstrate that MCG more accurately identifies core syntactic nodes through connectivity, facilitating the capture of syntactic structures that contain aspect terms and thereby enhancing aspect extraction. Second, compared to LLMs, our multi-hop syntactic dependency graph better leverages contextual information to identify correct aspects.

Performance on MASC. Table 4 highlights MCG's top performance on the MASC. First, on Twitter-2015, it boosts the F1 score by 3.4% over the runner-up AoM model, and by 3.1% compared to AESAL on Twitter-2017. This improvement is due to sentiment-enhanced pre-training, enhancing the model's sentiment differentiation. Second, MCG surpasses LLMs in identifying fine-grained sentiments.

4.4 Ablation Study

In this section, to demonstrate the effectiveness of each module of MCG, we compare the variants of MCG in terms of image (Img), sentiment-enhanced pre-training (SEP), multihop connected graph attention (MCGA).

W/o Img is a variant of MCG that removes image information but utilizes textual information. W/o SEP is a variant of MCG that does not use sentiment-enhanced pre-training. W/o MCGA is a variant of MCG that removes multi-hop connected graph attention and only uses cross attention.

As shown in Table 5, we found that: 1) Visual information enhances the accuracy of aspect word and sentiment recognition when interacting with textual data. MCG outperforms W/o Img across all three tasks, highlighting the effectiveness of visual inputs. 2) SEP can improve the capacity of multiple sentiment discrimination. Taking the MASC as an example, on the two datasets, MCG improves the Acc and F1 values by 2%, 4.7%, 1.3%, and 6%, respectively, compared to W/o SEP, demonstrating its effectiveness. 3) MCGA captures the importance of different syntactic relationships, enhancing focus on both local context and global information. On MATE, metrics for W/o MCGA decrease on both datasets, proving MCGA's effectiveness in highlighting key syntactic relations for text semantic understanding.

4.5 Case Study

Fig. 5 illustrates two examples of predictions using JML, CMMT, and MCG to further substantiate the efficacy of MCG. In Example (a), JML failed to fully recognize "Fifth Harmony" and misjudged the sentiment for the "BBMAs". CMMT incorrectly predicted the sentiment for the "BBMAs". However, MCG successfully determine this sentiment, which is attributed to our sentiment-enhanced pre-training task. In Example (b), both JML and CMMT provided incorrect sentiment judgments for "Darrelle Revis", with JML failing to identify the aspect term "cowboys" and its sentiment, and CMMT erroneously identifying "CBs" as "rookie CBs". Ours MCG, however, identified all aspect terms in both cases and provided correct sentiment predictions. This demonstrates the effectiveness of utilizing multi-hop connected graph attention to focus on key syntactic relationships for MABSA.

5 Conclusion

This paper introduces a sentiment-enhanced multi-hop connected graph attention network for MABSA. Firstly, we construct positive and negative samples to perform comparative sentiment pre-training, thereby enhancing the diversity of the training samples and improving the multiple sentiment discrimination capability. Secondly, we build a multi-hop connected syntactic dependency graph to capture diverse syntactic relationships, and utilize the multi-hop connection graph attention mechanism to emphasize significant syntactic relationships within the syntactic structure. Lastly, we conduct three tasks on two widely used datasets, and the experimental results validate the effectiveness of our method. Additionally, we examine the model's capability to address nuanced sentiment expressions through extensive empirical evaluations.

Acknowledgments

This work was supported in part by the National Science Foundation of China under Grant 62472348, in part by the Aviation Science Foundation 2023M071070002 and 2024M071070001, in part by the Key Research and Development Program of Shaanxi under Grant 2023-YBGY-230 and 2024GX-YBXM-533, in part by the Innovation Capability Support Plan of Shaanxi under Grant 2022PT-33, in part by the Xi'an Science and Technology plan Key industrial chain technology research project under Grant 23ZDCYJSGG0007, and in part by the Xi'an Science and Technology plan Key industrial chain, key core technology research project under Grant 23LLRH0022, and Qinchuangyuan Construction of Two Chain Integration Important Project 23LLRHZDZX0006, and Sichuan Science and Technology Program 2025ZNSFSC0468. Any opinions, findings, and conclusions expressed here are those of the authors and do not necessarily reflect the views of the funding agencies.