T-T: Table Transformer for Tagging-based Aspect Sentiment Triplet Extraction

1 Introduction

Aspect sentiment triplet extraction (ASTE) remains a crucial research direction in the era of large language models (LLMs) [Wang et al., 2023; Zhang et al., 2024], widely used for fine-grained opinion mining from user reviews, social media news, and other types of text [Zhang et al., 2022a]. ASTE task aims to extract corresponding aspect terms, opinion terms, and sentiment polarities from a given sentence. The sentiment polarity here is classified into three categories: {Positive, Neutral, Negative}. For example, as shown above in Figure 1, given the sentence "The screen of the phone is smaller, but overall it's good.", it has two sentiment triplets: (screen, smaller, Negative) and (phone, good, Positive), where screen and phone are the aspect terms, smaller and good are the corresponding opinion terms, Negative and Positive are the corresponding sentiment polarities.

Recently, a variety of techniques have been proposed for addressing the ASTE task, including span-based methods [Xu et al., 2021; Chen et al., 2021; Li et al., 2023b; Zhao et al., 2024], generative methods [Gao et al., 2022; Gou et al., 2023; Xianlong et al., 2023; Mukherjee et al., 2023; Zou et al., 2024], and tagging methods, among which the tagging method is particularly attractive. Sequence tagging methods [Peng et al., 2020; Xu et al., 2020; Xianlong et al., 2023] use the BIESO scheme 1 to tag aspect and opinion terms in sentences, but these methods fail to fully capture the relations between individual words. Another more competitive tagging approach is the table tagging methods [Wu et al., 2020; Chen et al., 2022; Zhang et al., 2022b; Liang et al., 2023b; Sun et al., 2024]. Given a sentence, it begins with encoding using a transformer-based pre-trained language model. Then, the obtained representations are encoded into a relation table. In this relation table, the horizontal axis represents aspect terms, the vertical axis represents opinion terms, and each cell (e.g. cellij) in the table denotes the relation between the i-th and j-th words in the sentence. With this approach, sentiment triplets can be easily annotated within the table, and the relations between any two words can be fully captured.

1 BIESOmeans "begin, inside, end, single, other", respectively.

One of the keys to the table tagging methods lies in designing different downstream relation encoding modules to help the model learn stronger relations representations. Previous methods [Wu et al., 2020; Jing et al., 2021; Chen et al., 2022; Zhang et al., 2022b; Liang et al., 2023a; Peng et al., 2024b; Peng et al., 2024a] have designed various downstream relation encoding modules, such as using Multi-Layer Perception (MLP) [Jing et al., 2021], Graph Convolutional Network (GCN) [Chen et al., 2022], and Convolutional Neural Network (CNN) [Zhang et al., 2022b]. These works demonstrate that a stronger relation encoding module can yield greater overall benefits for the model. This motivates us to seek even more powerful relation encoding modules.

In this work, we attempt to directly utilize transformer layers [Vaswani et al., 2017] as our downstream relation encoding module. Ideally, the use of self-attention mechanisms in transformers enables the model to capture dependencies across the entire table sequences. This capability holds the promise of significantly enhancing the model's performance. However, in reality, two formidable challenges impede the realization of this technique. Challenge 1: Overly long table sequences. It is known that, if assuming the input sentence sequence length is n, the time and space complexity of the transformer layer in the multi-head self-attention module is O(n2 ). However, when dealing with table sequences, where the length of the table itself is n2 , the computational complexity of the attention mechanism in the transformer layer escalates to O(n4 ). This is unacceptable with limited computational resources. Challenge 2: Unfair local attention interaction. Long sequences encourage us to use local attention mechanisms, but in reality, the correlation between tokens is not confined to local areas. When each token only focuses on its immediate periphery, capturing information from interactions with distant tokens becomes challenging.

To address the aforementioned challenges, we propose a novel Table-Transformer (T-T) for ASTE's table tagging method. As shown in Figure 1, when directly utilizing transformer-based sequence labeling models to process sentences, although cost-effective, their performance suffers due to their inability to fully capture word relations. However, when encoding sentences into table sequences first and then using transformer layers as the relation encoder can yield good results, but the costs are intolerable due to the challenge of overly long sequences. Our approach, tailored for table sequence inputs, introduces unique enhancements to the original transformer layers, enabling the model to achieve strong performance while maintaining relatively low computational costs. Specifically, to address the overly long table sequences challenge, we propose a stripe attention mechanism. It enhances the original global self-attention mechanism by restricting it to operate only within a fixed-size window range (assumed to be a constant k). This modification reduces the original time-space complexity from O(n4 ) to O(k2n2 ). For the unfair local attention interaction challenge, we devise a novel loop-shift strategy that effectively facilitates information interaction between different windows.

Our contributions can be summarized as follows:

1. We observe the two-dimensional nature of table sequences and based on this insight, we have designed an enhanced transformer layer called T-T. It can effectively capture local information within table sequences without relying on external knowledge or task-specific designs.
2. We introduce two novel techniques aimed at effectively addressing the two challenges encountered by the original transformer when handling table sequences.
3. Experimental results show that our approach achieves state-of-the-art performance within acceptable costs.

2 RelatedWork

2.1 Aspect Sentiment Triplet Extraction

To meet the demand for a more detailed exploration of the opinion contained within the text, [Peng et al., 2020] first proposed and addressed the ASTE task in a pipeline manner. Subsequently, more diversified techniques have been proposed. MRC methods [Mao et al., 2021; Zhai et al., 2022; Zou et al., 2024] treated the ASTE task as a form of machine reading comprehension task. Generative methods [Zhang et al., 2021; Gao et al., 2022; Gou et al., 2023; Mukherjee et al., 2023] treated the ASTE task as an index generation task. Span-based methods [Xu et al., 2021; Li et al., 2023b; Zhao et al., 2024] extracted all possible spans and considered the interplay of information at the span level.

Table tagging, as another vibrant research direction, was initially proposed by GTS [Wu et al., 2020] to annotate sentiment triplets in a 2D table. Subsequent extensive work, despite variations in the tagging schema, primarily focused on how to capture sufficient relation information for the table. EMC-GCN [Chen et al., 2022] utilized GCN to incorporate rich syntactic information. BDTF [Zhang et al., 2022b] employed CNN to fully grasp word boundaries. STAGE [Liang et al., 2023a] and SimSTAR [Li et al., 2023a] integrated the learning of span-level information into the relation encoder. MiniConGTS [Sun et al., 2024] proposed a new table tagging scheme based on contrastive learning. In conclusion, the design of a more powerful downstream relation encoder stands as a pivotal focus in the table tagging methods.

2.2 Efficient Transformer

The transformer model architecture [Vaswani et al., 2017] has become an indispensable tool in modern deep learning research due to its effectiveness in the field of NLP [Tay et al., 2022]. However, the self-attention mechanism and stacked design logic of the transformer layers result in high computational resource requirements. A large amount of previous work has focused on addressing this issue. Some efforts attempted to reduce computational costs by modifying the scope of self-attention calculations. These strategies include sliding windows [Beltagy et al., 2020; Zaheer et al., 2020]

or different attention mechanisms at various layers [Zhang et al., 2023]. Other efforts have aimed at optimizing the computational efficiency of the self-attention matrix, such as applying a low-rank matrix [Wang et al., 2020], a kernel function [Katharopoulos et al., 2020], or a focused attention module [Han et al., 2023].

3 Preliminary

3.1 Task Definition

Given a sentence X = {x1, x2, ..., xn} of length n, the aim of ASTE is to extract all sentiment triplets T = {(a1, o1, s1), (a2, o2, s2), ..., (a|T|, o|T|, s|T|)} from X. Here, a and o respectively represent the aspect term and opinion term, which are spans within X, while s denotes the corresponding sentiment polarity in {Positive, Neutral, Negative}.

3.2 Tagging Scheme

We choose the boundary-based tagging scheme [Zhang et al., 2022b; Ning et al., 2023] in our work. For sentence X, it is mapped onto a two-dimensional table of size n×n, where the vertical and horizontal axes represent aspect terms and opinion terms, respectively. For an sentiment triplet (ai, oi, si) in X, assuming that ai and oi are located at positions [x, y] and [m, n] (x ≤ y,m ≤ n). They can form a rectangular region in the table, with coordinates (x,m) as the top-left corner and (y, n) as the bottom-right corner. We label the cells on the boundary of this region as TL (top-left vertex) and BR (bottom-right vertex), while all other cells are tagged as None. When the sentiment elements are single words, the TL and BR positions coincide. Subsequently, we label the sentiment polarity of this rectangular region according to si.

3.3 Generalised AttentionMechanism

For an input sequence C = {c1, c2, ..., cn} ∈ Rn×d , the generalised attention mechanism of Transformer [Vaswani et al., 2017] treats the attention calculation over the entire sequence as a directed graph G = {C,A}, where the adjacency matrix A ∈ Rn×n is used to describe directed edges. For any aij ∈ A (aij = 1 or 0), if aij = 1, it indicates that the query ci attends to the key cj. Let N(ci) denote all neighbors of ci (including itself), the attention calculation for ci is defined as:

where Q(·),K(·) : Rd → Rm and V (·) : Rd → Rd are the query, key, and value functions, respectively. σ is a softmax function.

4 Method

Our framework is shown in Figure 2, with the relation encoding module configured as our T-T 2 .

4.1 Table Encoding

As illustrated in Figure 2 a), sentence X is first inputted into a pre-trained language model (PLM) for encoding, yielding the last hidden layer's representation H = {h1, h2, ..., hn}. Subsequently, we extract the features of aspects and opinions from H ∈ Rn×d using two separate linear layers:

2 Codes are available at https://github.com/KunPunCN/T-T

where hai ∈ Rd and hoj ∈ Rd denote the aspect and opinion representation, respectively. To ensure that the table captures sufficient relation information, which is beneficial for downstream tasks, we followed the previous works [Chen et al., 2022], employing Biaffine Attention [Dozat and Manning, 2017] to capture the relations distribution between words. This process is formalized as:

where ⊕ denotes the concatenation operation. The first two terms represent the biaffine attention process, where W1 ∈ Rd×2d and W2 ∈ Rd×√d×d are learnable parameters. The last term Pooling(h[i,j]) represents the maxpooling operation for the sentence from token hi to hj, which aims to extract the span-level information. rij ∈ R2d+√d is the table representation. Finally, a dense linear layer Lineard: R2d+√d → Rd′ is used to compress rij's dimension to d′ : r′ij = Lineard(rij). Therefore, the final representation of the relation table is denoted as R ∈ Rn×n×d′ .

4.2 Relation Encoding with T-Tmodule

The initialized relation table needs additional refinement of the relations distribution through the proposed T-T module. As depicted in Figure 2 b), we first use a simple convolutional layer with a kernel size of 3 to capture spatial correlations, and then feed the obtained representationR0 ∈ Rn×n×d′ into the T-T module. The T-T improves upon the original transformer architecture by replacing the full attention mechanism with an enhanced stripe attention mechanism and adding a novel loop-shift strategy at the end of each layer.

Stripe Attention Mechanism

Here we propose a novel stripe attention mechanism to address the overly long table sequence challenge. As described in the Preliminary section, the generalised attention mechanism treats the attention calculation as a directed graph G = {C, A}. In the original transformer layer, a full attention mechanism is used, which means A is the full ones matrix and N(ci) = C in Formula (1). When the input is a sequence C ∈ Rn×d′ , each token needs to interact with all other tokens and store all computation results, leading to a spatiotemporal complexity of O(n2 ). The derivation reveals that the spatiotemporal complexity of the full attention layer scales quadratically with the length of the input sequence. When our input is the flattened table sequence R0 ∈ Rn×n×d′ , the overall spatiotemporal complexity becomes O(n4 ). This is intolerable for limited computational resources.

To maintain good performance while reducing the spatiotemporal complexity, the proposed stripe attention mechanism partitions the table sequence R0 into multiple smaller blocks, treating them as the smallest computational units. Assuming each block has a width of b and a shape of Rb×b×d′

(where n is padded beforehand to a multiple of b), this results in a total of l2 (we define l = nb ) blocks: O = {o1, o2, ..., ol2} ∈ Rl2×d′ . It is worth noting that we apply a cyclic shift operation on this sequence, which means that for a block oi in this sequence, its left and right neighbors in the attention map are oi±1 = o(i±1+l2 )%l2 , and its upper and lower neighbors are oi±l = o(i±l+l2 )%l2, where % is used for the remainder calculation. Similar to Formula 1, for each block oi, it's attention calculation is defined as:

Per Formula 4, the query matrix and the key matrix require a dot product operation to generate the attention map.

As shown in the left half of Figure 2 c), the query and key matrices are also partitioned into multiple blocks. According to Formula 4, a dot product operation is needed for the query matrix and the key matrix to generate the attention map. In the full attention mechanism, each query block needs to undergo a dot product operation with all key blocks, resulting in a complexity of O(n4 ). To enhance attention performance while maintaining lower complexity, in our stripe attention mechanism, each query block only selectively attends to its neighboring blocks in the key matrix. As indicated by the highlighted red boundary box in Figure 2 c), for a query block, we define its neighbors in the 2D table space as all key blocks within a square area centered around it. Assuming a window width of w(w ≤ l) centered around each block for attention, then the neighbors of block oi can be defined as:

where w is odd. The number of N(xi) is |N(xi)| = w2 , which means that for each block (total of n2 tokens), it attends to all blocks within a square area of w2 , encompassing a total of w2 ×b2 tokens. The overall complexity is O(n2 ×w2 b2 ) = O(w2 b2n2 ). In addition, when w = l, N(xi) will include all blocks, and stripe attention will degenerate into full attention.

An example of the stripe attention map is shown in the right half of Figure 2 (c), where n = 4b and w = 3. Compared to local attention, stripe attention allows tokens to attend to a broader range of relevant positions in the 2D table space, thereby enhancing learning capability, while only marginally increasing complexity by a constant factor.

Loop-shift Strategy

Here we propose an innovative loop-shift strategy to address the unfair local attention interaction challenge. While the stripe attention mechanism achieves a balance between attention scope and computational load, partitioning the table sequence into fixed block widths prohibits interaction between different blocks. This presents an unfair scenario for tokens situated at block edges, as their crucial neighbors may reside in different blocks. Therefore, inspired by [Liu et al., 2021], we introduce a novel loop-shift operation to the table sequence between different T-T layers. For the output of any given layer, which has a shape of n × n, before feeding it into the next layer, we first transform the positional distribution of the table tokens. As illustrated in Figure 2 d), we perform a two-step shift operation on the table with a width of b2 . Firstly, we chop the tokens from the top of the table, forming a shape of b2 × n, and move them to the bottom of the table. Secondly, we chop the tokens on the left side of the table, forming a shape of n × b2 , and move them to the right side of the table. This operation effectively cyclically shifts the entire table to the upper-left direction by a distance of b2 . Similarly, for the next layer's output, we need to cyclically shift the entire table in the opposite direction, by a distance of b2 towards the bottom-right direction, to restore the table's distribution. Therefore, the shift operation occurs in pairs, implying that the number of T-T layers in our model is even.

After passing through N layers of the T-T module, we obtain the last layer's representation RN ∈ Rn×n×d′ . Then a weighted residual connection is used to derive the final representation: R = W3RN + (1 - W3)R0 .

4.3 Triplet Decoding

During the decoding stage, we employ two separate full linear layers to predict the coordinates of all TL and BR, respectively. The predicted results for each position are pTL ij and pBR ij , respectively. Given the labels yTL ij and yBR ij , the corresponding loss is:

where LCE is the cross entropy loss function. Subsequently, for any pair of TL and BR, they form a candidate rectangle when the TL is positioned to the upper left or coincides with the BR. This rectangle is represented as:

where rab and rcd respectively represent the token at TL and BR, rab, ..., rcd denote all tokens within this rectangle. Then, a sentiment classifier Linears: R3d → R4 is utilized to predict the sentiment polarity of this region as one of { Pos, Neu, Neg, Invalid}: psk = Linears(rabcd). Since each TL/BR may form candidate rectangles with multiple BR/TL, the label Invalid is additionally used to assess the validity of the candidate region. When the prediction psk is Invalid, the region should be dropped. When the prediction psk falls within { Pos, Neu, Neg}, the corresponding sentiment triplet can be extracted based on the TL, BR coordinates, and sentiment polarity of the region. The loss of this process is:

where m represents the number of candidate rectangles, psk and ysk are the prediction and label, respectively, for the k-th candidate rectangles. The overall loss is: L = L1 +L2.

5 Experiments

5.1 Experimental Settings

Datasets and Baselines

We conduct our experiments on four benchmark datasets [Peng et al., 2020; Xu et al., 2020], which are originally derived from the SemEval challenge [Pontiki et al., 2014; Pontiki et al., 2015; Pontiki et al., 2016]. The detailed statistics are provided in Table 1. We categorize the comparison baselines into five types: seq tagging, MRC-based, generative, span-based, and table tagging.

Implementation

The proposed model contains a table encoder and a relation encoder, with hidden state dimensions of d = 768 and d′ = 1024, respectively. We initialize the table encoder with the BERT-base-uncased [Devlin et al., 2019] version. As a relation encoder, our T-T module consists of two transformer layers and utilizes the parameters from the last two layers of the BERT-large version for initialization. The block width b is 7 for 14res, 15res, 16res and 5 for 14lap. The window width w is 3. During training, we use the AdamW optimizer with an initial learning rate of 3e-5 for all layers. The model is trained for 15 epochs on RTX 3090 GPUs, with a batch size of 4. In each epoch, we evaluate the training model on the development set and save the best one. We use the sentence-level F1 score as the evaluation metric, which means that a sentence is considered a true positive only when all triples within it are correctly extracted. All the reported results are the average of five runs with different random seeds.

5.2 Main Results

As shown in Table 2, we have the following observations: Our model surpassed the best baseline, D2E2S, with improvements of 0.66%, 0.02%, 2.75%, and 0.1% F1 scores on 14res, 14lap, 15res, and 16res, respectively, resulting in an overall average improvement of 0.89% F1 score. While D2E2S introduced additional tools to provide syntactic dependency information and developed task-specific unique modules, our model solely utilized a streamlined architecture to effectively capture local information within 2D table sequences.

Our model significantly outperformed all table tagging methods. Compared to the best table-based model, MiniConGTS, our model surpassed it by 0.2%, 0.07%, 3.46%, and 0.01% F1 scores on 14res, 14lap, 15res, and 16res, respectively, under the F1 scores, with an overall average improvement of 0.93%. This demonstrates that our T-T module exhibits stronger relation encoding capabilities compared to previous relation encoding modules.

Compared to initializing the T-T parameters using the last two layers of BERT, "Random init." indicates random initialization of the T-T, which resulted in an average decrease of 0.70% F1 score in overall model performance. We attribute this to the insufficient training data size to optimize the parameters of the two transformer layers, resulting in model underfitting. On the other hand, initializing with the last two layers of pre-trained BERT incorporates certain semantic knowledge, thereby improving the model's effectiveness.

5.3 Ablation Study and Computational Cost

We conduct an ablation experiment to validate the proposed module's improvements in model performance and costs. As shown in Table 3, since loop-shift and Stripe Attention (SA) are parameter-independent mechanisms, removing them will not alter the model's parameters.

When the loop-shift strategy is removed, the training and inference costs remain largely unchanged. However, the performance decreased by 2.16%, 1.32%, 2.23%, and 2.11% F1 scores on the four datasets, demonstrating the crucial importance of information interaction across different blocks for enhancing the overall model performance. When SA is removed, the model degrades to full attention while retaining the Loop-shift strategy, resulting in performance similar to that of directly using normal layers. After removing SA, we observe a slight improvement in model performance, but there is a significant deterioration in both training and inference costs, which are 13.21/ms and 2.96/ms, respectively. This demonstrates that the Stripe Attention mechanism not only greatly improves attention computation efficiency but also effectively attends to relevant tokens. When the T-T relation encoder module is removed, the model's performance significantly deteriorates, with respective drops of 4.37%, 3.39%, 3.94%, and 4.33%. This demonstrates the critical importance of designing a stronger relation encoding module.

5.4 Auxiliary Experiment on Subtask

To further investigate the effectiveness of T-T, we conduct an auxiliary experiment on the Aspect Opinion Pair Extraction

(c) layers

Figure 3: The sensitivity of different hyperparameters.

(AOPE) task, which aims to extract all aspect-opinion pairs from a sentence. By modifying the category width of the sentiment classifier Linears from 4 ({Pos, Neu, Neg, invalid}) to 2 ({Valid, invalid}), our model can directly address these tasks without any additional modifications. As depicted in Table 4, we chose the table tagging methods for comparison to demonstrate the specific improvements of our model. The results demonstrate that our T-T achieves comprehensive improvements across four datasets, with an average F1 score increase of 0.96% compared to the best baseline, MiniConGTS. This further validates the important role of T-T in capturing and matching token relationships within the table sequences.

5.5 Hyperparameter Analysis

To investigate the impact of different hyperparameter settings in the T-T module on model performance, we conduct an additional experiment. The results, depicted in Figure 3, demonstrate the effects when varying (a) the block width b, (b) the window width w, and (c) the T-T layer count while keeping other settings consistent with the main experiment.

In Figure 3 (a), as the block width b increases, the training time cost exhibits a nearly quadratic growth, consistent with the O(w2 b2n2 ) complexity of the stripe attention mechanism as stated earlier. Additionally, when the block width exceeds 7, there is minimal improvement in the model's performance. We attribute this to the observation range (b × w = 3 × 7) approaching saturation, effectively attending to all important tokens. The findings drawn from Figure 3 (b) are consistent with Figure 3 (a) as the training time cost exhibits a quadratic increase with the window width w. Additionally, when the window width exceeds 3, there is minimal performance improvement. In Figure 3 (c), as the number of T-T layers increases, the computational cost shows linear growth. When the number of layers exceeds 2, there is a slight decline in model performance, which we attribute to potential underfitting caused by an excessive number of parameters.

5.6 Further Analysis on Loop-shift Strategy

The loop-shift strategy is employed to address the challenge of unfair local attention interaction. We conducted an additional experiment to validate this strategy further. As shown in Figure 4, removing the loop-shift strategy results in a significant performance drop of the model on long-distance pairs. This demonstrates the effectiveness of the loop-shift strategy, as removing it prevents tokens at the boundaries of attention windows from fully capturing valuable information, resulting in significant performance degradation.

5.7 Performance of Different Word Spans

We also compare the performance of T-T with other table tagging methods across different word spans, including singleword (Single.), multi-word aspect (Mutil. A.), and multi-word opinion (Mutil. O.). The results are shown in Figure 5. Our model outperforms previous methods in all settings, with the improvement being more pronounced in the multi-word setting. This highlights the ability of T-T to effectively capture word boundary information.

6 Conclusion

In this paper, we propose a novel Table-Transformer (T-T) approach, which uses enhanced transformer layers as the relation encoding module for the table-based ASTE task. To address the challenges posed by overly long table sequences and hard local attention interaction, we respectively propose our stripe attention mechanism and the loop-shift strategy. The former reduces attention computation costs by focusing on local tokens within the 2D table space, while the latter facilitates interaction between attention windows through loop-shift operations. Extensive experiments on four datasets demonstrate the effectiveness of our T-T over the best baselines. In future work, we aim to adapt this method to a broader range of information extraction tasks, such as relation extraction and event extraction.

Acknowledgments

This research is supported by the National Key R&D Program of China (No. 2023YFC3303800), NSFC through grants 62322202, 62441612, and 62402054, "Pioneer and Leading Goose R&D Program of Zhejiang" through grant 2025C02044, National Key Laboratory under grant 241-HFD07-01, and Hebei Natural Science Foundation through grant F2024210008.