Inheriting the Wisdom of Predecessors: A Multiplex Cascade Framework for Unified Aspect-based Sentiment Analysis

1 Introduction

As one of the core directions of sentiment analysis, ABSA has received extensive research attentions within past decade [Wang et al., 2017; Li et al., 2018; Fei et al., 2021b]. ABSA has derived a number of subtasks, which all revolve around predicting three major elements of ABSA, i.e., aspect, opinion and sentiment polarity, or their combinations. For example, aspect term extraction (AE) and opinion term extraction (OE) seek to extract aspect and opinion terms respectively. Aspect-level sentiment classification (ALSC) aims at predicting the sentiment polarity given an aspect term, while triplet extraction (TE) targets as extracting all the correlated <aspect, opinion, polarity> triplets in a sentence. We illustrate the definitions of all the subtasks in Fig. 1(a) with specific examples. Most prior works solve one certain subtask in isolation [Wang et al., 2017;

∗Corresponding author

(b) Subtask Correlations.

Fan et al., 2018], while some efforts have been made to construct unified methods for modeling as more subtasks as possible, so as to narrow the gaps between the modeling divergences of them [Yan et al., 2021].

On the other hand, since different subtasks share much interactive correlation in essence, some works consider explicitly incorporating such underlying shared signals [Li et al., 2019; Hu et al., 2019]. Representatively, Chen and Qian [2020] and Yu et al. [2021] propose to collaboratively model the potential mutual interactions between ABSA subtasks based on the multi-task learning framework. Unfortunately, the interactions in these studies are limited to a small subset of ABSA subtasks (e.g., AE, OE and AESC), being inadequate enough to explore the rich shared information among all total seven subtasks. As depicted in Fig. 1(b), AE summarizes sentiment aspects, while OE emphasizes opinion words and meanwhile entails sentiment polarities. All the other subtasks are associated with each other based on dyadic (e.g., aspect-opinion) or triadic relations (e.g., aspectopinion-polarity). Therefore, how to effectively and sufficiently capture the inline interactions among all these subtasks for further enhancements still remains unexplored.

After carefully investigating the characteristics of the ABSA subtasks, we gain the following major observations:

• Lower-level subtasks (e.g., AE and OE) actually lay the foundations of ABSA, explicitly offering the core sentiment layouts and clues for building up high-level subtasks.
• Higher-level subtasks (e.g., TE) contain more complete and richer information, which can in return benefit low-level tasks implicitly.
• All the subtasks can be unified by sharing the same 'aspect-opinion-polarity' ternary backbone, with slight divergences on inputs or outputs.

Consequently, we consider modeling all these subtasks as a hierarchical dependency (HD) as detailed in Fig. 2, including four task groups ranging from Level A to Level D according to the task properties, i.e., from easy to hard.

On this basis, we then present a multiplex cascade framework for the unified ABSA. We formulate these subtasks with a new cascade grid-tagging scheme. As shown in Fig. 3, the system is built based on multi-task architecture, with all subtasks sharing one common encoder. Each decoder makes prediction for one certain subtask, all of which are cascaded following the HD structure, practicing the philosophy: (1) rendering the ABSA keynotes with aspect and opinion (also entailing polarity) information produced at level A; (2) multiplexing the information of low-level tasks to the higher-level tasks one by one. With such explicit multiplex design of reusing the knowledge from low-level subtasks, the predictions of upper subtasks will be gradually enhanced and the subtask interactions are tightly captured.

Further, we notice that the key bottlenecks of our multiplex framework lie in 1) the detection of the underlying aspect and opinion terms, i.e., OE and AE at level A, and 2) the pairing between aspect and opinion terms from level A to B. For further enhancement, we leverage the external syntax knowledge that has been extensively validated effective [Huang and Carley, 2019; Fei et al., 2020]. First, the linguistic part-ofspeech (POS) tags entail rich boundary information between span neighbors, which can essentially promote the recognition of aspect and opinion terms. Besides, the syntactic dependency features can provide additional clues for supporting more accurate reasoning of aspect-opinion pairing. We leverage a unified syntax graph convolution network [Wu et al., 2021] to simultaneously model POS tags and dependency features at the encoding side. In the decoding side, we further introduce a syntax-guided pairing method by reharnessing the syntax weights yielded at the encoder.

Our framework takes the current existing ABSA datasets, where however not all training sentences has the annotations simultaneously covering all seven subtasks. We thus train seven decoders jointly on the shared training sets, while updating those low-level decoders with partial annotations. Evaluation results on 4 benchmark datasets suggest that the proposed system significantly outperforms all the current state-of-the-art ABSA models on total seven subtasks with big margins, demonstrating the effectiveness of our method for the unified ABSA. Our contributions include:

• ⋆ We build a unified model for total seven ABSA subtasks. We introduce a novel multiplex decoding based upon a

cascade grid tagging method, fully reusing the shared information and interactions among different subtasks. Our model ensures high-efficient interflows, and meanwhile avoids duplicated or trivial training in previous single-task paradigm.

• ⋆ We propose to incorporate rich external syntactic knowledge, including POS tags and dependency trees, into both the encoder and decoders for enhancing aspect-opinion term extraction and pairing, which are the bottlenecks of all ABSA systems.
• ⋆ Our syntax-aware multiplex model (namely, SyMux) pushes the state-of-the-art results for all seven ABSA subtasks against best-performing baselines. The model can work without annotating any additional shared data, since the existing training data of high-level subtasks (e.g., TE) covers all the annotations needed by low-level subtasks. 1

2 Methodology

Task formulation. Based on the definition in Fig. 1(a), we now give the specific formulations of each subtask. The overall framework follows the multi-task learning scheme. The shared encoder takes an input sentence S={w1, · · · , wn}, and then seven separate decoders of different subtasks will yield their own outputs. For (1) AE, (2) OE, (3) AOE and (4) AOPE, the tag set is {O,I}, where O means a trivial word and I means a word belonging to a term or forming an aspectopinion pair. For (5) ALSC, (6) AESC and (7) TE, the tag set will be {O,P,N,M}, where O means a trivial word, and P/N/M imply that a word belongs to a term or an aspectopinion pair meanwhile whose sentiment polarity is positive/negative/neutral, respectively. Specifically, for the base tasks OE and AE at level A, we take the 1-D sequential tagging scheme, while the high-level subtasks take the 2-D grid tagging [Wei et al., 2020; Wu et al., 2020].

1 Resources at https://github.com/scofield7419/UABSA-SyMux

Considering the example "Great steak but costly beer" in Figure 2, each decoder produces the following outputs Y :

Note that underlined tags mean that gold-standard aspect terms are given, while those tags with vertical dots at left mean that opinions are not the required prediction targets.

2.1 Encoder

Contextual encoder. Pre-trained language models (PLM), e.g., BERT [Devlin et al., 2019] have been shown prominent on retrieving the contextualized features, becoming the defacto encoder in a wide range of NLP tasks [Eberts and Ulges, 2020; Zhao et al., 2020]. In this work we take the BERTvariant PLM, RoBERTa, as the context encoder, in which the calculations are summarized as:

where hn is an output representation for word wn.

Unified syntax GCN encoder. Essentially, the recognition of aspect and opinion terms as well as their pairing are two key foundations of our system. Therefore, we integrate external syntactic knowledge into our system for enhancement, including the word-level linguistic POS tags for potential span boundaries of aspect and opinion terms [Wu et al., 2021], and the syntactic dependency trees for aspect-opinion pairing [Fei et al., 2021a; Fei et al., 2021c]. Following Wu et al. [2021], we leverage a unified syntax GCN (USGCN) encoder to fuse these two sources simultaneously.

Based on the input S we first have a list of POS tags {wp i }n for each word. We maintain the POS embedding vector xpi via a look-up table. Then we have the corresponding dependency tree of S. We form a graph G = (V,E), where V is a set of words, and E is a set of dependency edges between each pair of words. We define an adjacency matrix {ai,j}n×n in E, in which ai,j=1 if there is an edge between wi and wj, and ai,j=0 vice versa. We additionally enable the self-loop of each word and bidirectional edges between valid word pair for information enhancement. Suppose eli is the hidden representation of wi at l-th USGCN layer:

where αli,j is the syntactic-aware neighboring weight obtained from:

where ⊗ is an element-wise multiplication. The weight matrix αi,j entails rich syntactic relationship between tokens.

We then apply a gate c to flexibly coordinate the contribution between POS information and USGCN representations, since term recognition may rely directly on the span boundary signals provided by POS tags. We will empirically validate the usefulness of such operation in the experiments.

where vs,l i is the final output of the l-thUSGCN.We take total L layers of USGCN for full syntax knowledge propagation. Finally, we explicitly concatenate the outputs of USGCN and RoBERTa encoders as the final contextualized representation

(a) Syntactic dependency structure.

for each word: vi = [vs,L

2.2 Decoder

AE and OE decoding. As shown in Fig. 3, the AE decoder first obtains its representation R1={r1 i }n based on the context representation vi via a non-linear transformation, i.e., r1 i =FFNs(vi). Also the representation R1 will be duplicated as the OE decoder representation R2={r2 i }n via another nonlinear transformation. AE and OE decoders then produce their own predictions (i.e., Y 1 , Y 2 ) based on their 1-D representations with two separate softmax classifiers.

Syntax-guided pairing. The next step is to construct a 2-D representation RM ={rM i,j }n×n based on R1 and R2 for the aspect-opinion pairs. We reach this via Cartesian product:

where the vertical direction in RM denotes aspect terms while the horizontal direction denotes opinion terms. To aid the pairing, we further utilize the syntactic dependency signals. Intuitively, the syntax structure reveals how aspect and opinion terms relate to each others, as depicted in Fig. 4(a). Although we encode the dependency features previously via USGCN, the information can be quite diluted at the decoding stage. We thus reharness the syntax weights calculated in Eq. (3) for the pairing, i.e., syntax-guided pairing, as in Fig. 4.

Multiplex decoding. As we formulate the unified ABSA subtasks into HD (cf. Fig. 2), all the remaining subtasks can share one same sentiment layout in our multiplex framework. Naturally, almost all the pivotal information needed for detecting the downstream subtasks is entailed in the initial 2-D representation RM . Thus we perform representation multiplexing based on RM along the order of [1/2→(3→4)→(5→6)→7] within the HD structure, as can be seen in Fig. 3.

Technically, we obtain the decoder representation Rk

(3≤k≤7) of the forward k-th subtask via the chain operation Rk←Rk-1 :

where Rk,P ={rk,P i,j }n×n is a private trainable 2-D parameter of k-th subtask for maintaining the task-private features, W k,S 4 ∈ Rn×n is a trainable transition matrix, and gk is a gate controlling the strength between shared and private information. Note that R2 is initialized with RM .

For AOPE, AESC and TE, the private representation Rk,P is randomly initialized. For AOE and ALSC where aspect annotations are available, we particularly inject the gold aspect signals into Rk,P by concatenating an embedding xai which is a binary aspect indicator, where 1 denotes an aspect term and vice versa for 0.

where '←' represents an affine transformation.

Predicting. Based on the representations Rk , each subtask yields its own prediction Y k via a softmax classifier on each elements of the 2-d matrix. For ALSC and AESC, the opinion terms in such 2-D prediction are not required, and only the aspects or polarities will be output. It is also worth noticing that in prior work some subtasks including AOE, ALSC and AESC are generally modeled as 1-D formulation (i.e., sequential labeling or sentence classification) since their inputs or outputs relate mainly to one certain element (i.e., either aspects or opinions/polarities). But in our multiplex framework, both two elements are fully considered as complementary sentiment clues for further enhancements.

2.3 Training

We perform multi-task training of our framework, where the input sentences should come with annotations of whole total seven subtasks. However, not all the training sentences has been labeled simultaneously with all subtasks. Therefore, we on the one hand perform error back-propagation for all subtask decoders on those shared training sets; on the other hand we partially train those decoders with input sentences that have target annotations and meanwhile keep the rest decoders fixed. The objective is to jointly optimize all seven subtasks:

where λk (∑7k=1 λk = 1) is a co-efficiency of k-th task loss:

here N is the number of training set for task k, and ˆ Y k is the ground-truth labels.

3 Experiments

3.1 Setups

Following Yan et al. (2021), we use the Semeval benchmark [Pontiki et al., 2014; Pontiki et al., 2015; Pontiki et al., 2016], including Res14, Lap14, Res15 and Res16, which have only the annotations for AE, ALSC and AESC. Followup works further partially contribute labels for different subtasks based on the Semeval sentences. Representatively, Wang et al. [2017] annotate the unpaired opinion terms (denoted as D17), while Fan et al. [2019] pair the aspects with opinion terms (D19), and Peng et al. [2020] provide the labels for triple extraction (D20). To enable multi-task training, we re-ensemble them so that most of the sentences' annotations cover all seven subtasks. Table 1 shows the data statistics.

We use the base version of RoBERTa PLM. L=3 in USGCN. Most of the representations have the shape of 300-d. The loss co-efficiency λk is fine-tuned for different tasks.

We make comparisons against some top-performing ABSA baselines, following Yan et al. (2021), including the models for ABSA subtasks at level A and C, such as SPAN [Hu et al., 2019], IMN [He et al., 2019], RACL [Chen and Qian, 2020] and MIN [Yu et al., 2021], and the systems capable of addressing all subtasks: CMLA [Wang et al., 2017], RINANTE [Dai and Song, 2019], Li-unified [Li et al., 2019], GTS [Wu et al., 2020], Dual-MRC [Mao et al., 2021], GEN [Yan et al., 2021]. All the comparing models install PLM (e.g., BERT, BART) for fair comparisons. For each subtask, we adopt the same evaluation metrics as the corresponding previous works do. We report the F1 scores for all tasks.

3.2 Results and Analyses

Main results. Table 2 shows the overall results on four datasets. We see that the unified systems (such as Dual-MRC and GEN) can bring very strong performances, while those models (e.g., RACL and MIN) considering rich interactions among subtasks can even achieve higher results on several certain subtasks. Yet our SyMux system significantly outperforms all these best-performing baselines universally on all subtasks over all datasets with big margins. This demonstrates the effectiveness of our method. More interestingly, we can observe that the higher-level tasks can benefit more from our system, comparing to those tasks at bottom levels, e.g., AE and oE at level A. Especially, ALSC, AESC and TE tasks always gain over 2 points of F1 over four datasets, compared with the best baseline. This substantially implies the necessity to take the multiplex idea for unified ABSA, fully reusing and recycling of the information learned at the lower level for facilitating the upper level tasks. Also it is worth explicitly noticing that such improvements of all subtasks by our system are obtained via the multiplex mechanism, without using any additional ABSA annotations.

Ablation. Since we take the external syntax information for enhancement, here we investigate the impacts. By removing the POS tag features away from USGCN, as in Table 3, there can be notable performance drops on AE and OE. This proves the importance of the linguistic POS knowledge for helping the term boundary detection. Likewise, if uninstalling the POS gate c (cf. Eq. 4) the overall results are also hurt, especially for the level-A tasks. This mostly verifies our assumption that it should have more sufficient and direct access to boundary information (i.e., POS tags) to aid the aspect or opinion extraction. Further stripping off the dependency knowledge from decoding, i.e., without syntax-guided pairing (SGP), we witness big performance reductions on all the higher-level 2-D subtasks. On the contrary, this influences very little on the AE and OE tasks. This further verifies that the POS and dependency features contribute to unified ABSA from different angles. And making use of both two features is complementarily favorable for the tasks.

Separating or sharing? sharing by multiplex! Recent related research tries to model the interactions between ABSA subtasks via sharing-aware learning, which has not been considered by those separating-learning methods. In Table 4, we show the comparisons between 'Separating' and 'Partial Sharing', and we learn that those baselines considering sharing mechanisms (e.g., RACL, MIN) can actually be beaten by those recent separating-learning models that employ certain powerful strategies, e.g., taking stronger pre-trained language model of BART in GEN, or remodeling the tasks as reading comprehension in DUAL-MRC. However, our sharing-based framework can surpass all strong separating-learning models. Via this, we prove that it is still important to perform sharingaware learning for unified ABSA. Notedly, with the multiplex mechanism, our system with partial sharing can still substantially outperform the separating-learning methods. For example, by merely sharing the four tasks at level A and B, ours greatly outperforms GEN and DUAL-MRC on all subtasks. In the meantime, this implies that our multiplex system can be of great usefulness in real-world applications, i.e., applied to some more strict scenarios where not all the data for total of seven subtasks are available.

As we revealed earlier, those existing sharing-aware baselines trying to promote the interactions among subtasks are unfortunately limited to a very small subset of tasks, i.e., partial sharing. Here we can observe that the full-scale information sharing of our model (in 'Full Sharing+Multiplex') enables to produce more accurate predictions, providing most sufficient interactions, comparing to any partial sharing scheme. Further looking into the SyMux itself, different schemes of partial sharing lead to varied performances, and more sharing generally gives better results. Furthermore, even with the same partial sharing as in baselines (i.e., shared on level A and C), ours still performs much better. We thus can conclude that the key to a successful sharing-aware learning among ABSA subtasks is the multiplex method.

Multiplexing order in hierarchical dependency. We suggest a hierarchical dependency of subtasks with the order of [1/2→(3→4)→(5→6)→7], which ensures a reasonable information interflow from low to high (level A to level D). Here we explore the impacts when following other different multiplexing orders. In Fig. 5 we plot the performance drop based on Res14. Overall, using other decoding orders can hurt all tasks' performances. Especially, the shuffled order and the parallel order, which are essentially the equivalents to getting rid of multi-task learning based multiplex strategy, lead to the biggest decreases. Even exchanging the subtasks at level B and level C can result in slight drops.

Model robustness against data scarcity. Owing to the mul-

0

Figure 6: ABSA results on data-rich (100%) and data-scarce (20%) setting where we only reduce the annotations for the target subtask, while the labels for training other tasks are available in full scale.

tiplex decoding design, the follow-up tasks in the HD structure can greatly benefit from their predecessor tasks. Here we further study the performances of each subtask when the annotations for the corresponding task are scarce, i.e., few-shot learning. Fig. 6 shows the results of three representative highlevel tasks (AOPE, AESC and TE) based on Res14 data. On the data-rich setting, i.e., with sufficient training signals, each system performs normally (ours performs the best). However, when only feeding 20% training annotations for training each of the three tasks respectively, the performances drop substantially. By comparison, our system even without using the full annotations for training one certain task, obtains much better results than baselines. This directly confirms that the multiplex cascade framework can fully recycle the signals of prior subtasks, e.g., sentiment layouts or clues, which semisupervisedly supports the reasoning of the topper tasks.

4 RelatedWork

The studies of ABSA mainly center on three key factors, aspect, opinion and sentiment polarity, which together depict the complete sentiment picture, i.e., aspect indicates what the sentiment targets are, sentiment polarity describes how the sentiment strengths are, and opinion explains why have such polarities. Within the scope of the ABSA, so far there can be a set of specific subtasks, each of them focuses on one specific element or the combinations of them, including 1) aspect term extraction (AE) [Li and Lam, 2017], 2) opinion term extraction (OE) [Wang et al., 2017], 3) aspect-oriented opinion extraction (AOE) [Fan et al., 2019], 4) aspect-opinion pair extraction (AOPE) [Zhao et al., 2020], 5) aspect-level sentiment classification (ALSC) [Tang et al., 2016], 6) aspect extraction and sentiment classification (AESC) [Hu et al., 2019] and 7) triplet extraction (TE) [Peng et al., 2020].

At the initial stage, separate solutions are extensively adopted for those standalone subtasks (e.g., AE, OE, ALSC) [Li and Lam, 2017; Fan et al., 2018]. Later, hybrid ABSA subtasks (e.g., AOE, AESC, TE) have been put on and solved with joint models [Li et al., 2019; Peng et al., 2020]. And then, some unified methods have been proposed to model as many subtasks as possible, so as to narrow the gaps between the modeling divergences of them [Chen et al., 2020; Mao et al., 2021]. Very recently, Yan et al. [2021] unify total seven subtasks with a pointer-network based sequence-tosequence framework. Unlike them, in this work, we construct a unified ABSA model to facilitate the information exchange. On the other hand, several studies attempt to promote the interactions between some ABSA subtasks. For example, Chen and Qian [2020] and Yu et al. [2021] take the multi-task strategy to enhance the collaborative learning between the tasks of AE, OE and AESC. Yet these works only consider a very small subset of the subtasks. In this work, we reveal that actually the underlying information of all subtasks can largely be reused, based on which we propose a novel syntax-aware multiplex framework for better unification.

5 Conclusions

We investigate a syntax-aware multiplex method for unified ABSA. We model total seven ABSA subtasks as a hierarchical dependency, based on which we present a multiplex cascade decoding framework, fully reusing the shared sentiment information and layout among different tasks. On the other hand, we incorporate both syntactic POS tag and dependency tree features for enhancing aspect-opinion term extraction and pairing. Our framework enables high-efficient interflows of ABSA subtasks, and meanwhile avoids duplicated and trivial training in previous separate-learning methods. Experimental results show that our method outperforms many state-of-the-art baselines for ABSA. In-depth analyses reveal the system's strengths, e.g., data robustness. We also find that our model performs well when the annotated data of some subtasks are not available or the data size is quite small, which suggests that the method has a wider potential application scenarios of ABSA. Future directions can revolve around the data-driven attempts for solving unbalanced sentiment polarities and inconsistent annotations of subtasks.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No.61772378, No. 62176187), the National Key Research and Development Program of China (No. 2017YFC1200500), the Research Foundation of Ministry of Education of China (No.18JZD015), the Youth Fund for Humanities and Social Science Research of Ministry of Education of China (No. 22YJCZH064), the General Project of Natural Science Foundation of Hubei Province (No.2021CFB385). This work is also supported by the Fundamental Research Funds for the Central Universities.