FastRE: Towards Fast Relation Extraction with Convolutional Encoder and Improved Cascade Binary Tagging Framework

1 Introduction

Relation extraction (RE) aims to identify the relationships between entities in texts, and plays an important role in natural language processing applications, knowledge graph construction, and question answering. Recent RE studies have made great progress in pursuing excellent performance [Wei et al., 2020; Ren et al., 2021]. However, in realistic scenarios, RE models are often expected to keep high performance and fast speed simultaneously. For example, financial investors require extracting relational triples effectively from massive and streaming real-time news, financial and political data to build knowledge graphs [Dong et al., 2014] to help make decisions. Unfortunately, most existing methods lack considering time consumed in model training and inference and cannot efficiently extract relations from texts. Early RE research applied approximate frequency counting and dimension reduction to speed up similarity computation in unsupervised RE [Takase et al., 2015], and adopted neural metric learning methods to accelerate RE [Tran and Kavuluru, 2019]. Various CNN based approaches [Zeng et al., 2014; dos Santos et al., 2015] had also shown effective in tackling this problem. However, these efforts fail to address the challenges of RE on both performance and efficiency.

∗Corresponding author

Recent advances on RE performance [Wei et al., 2020; Wang et al., 2020; Sui et al., 2021; Zheng et al., 2021; Ren et al., 2021; Huguet Cabot and Navigli, 2021] mainly due to adopting the encoders with the Transformer [Vaswani , 2017 ] based pre-trained language models (PLMs) such as BERT [Devlin et al., 2019] and BART [Lewis et al., 2020], which have powerful abilities to capture long-distance dependencies and context semantic features. However, the tokenpair based attention operation in Transformer requires much time consumption and memory consumption within GPUs. Moreover, the memory consumption of PLMs restricts the batch size during model training and inference, which means that models are restricted to set a relatively small batch size within limited computing resources. Although it is not a serious problem in training, it limits the parallel processing ability in the inference.

Instead of using the Transformer, we design a novel convolutional structure to tackle the computational efficiency issue in encoder. It significantly accelerates the training and inference speed with dilated convolution [Yu and Koltun, 2016], gated unit [Dauphin et al., 2017] and residual connection [He et al., 2016]. First, the dilated convolution increases the receptive region of network outputs exponentially with respect to the network depth, which results in obtaining drastically shortened computation paths and capturing arbitrarily longdistance dependencies. In other words, using dilated convolution can achieve the high efficiency of vanilla convolution with fewer layers. Second, gated unit is used to control what information should be propagated through the hierarchy of layers. Then residual connection is used for avoiding gradient vanishing to enable deep convolutional networks. Our convolutional encoder not only greatly reduces the model training and inference consumed time, but also ensures the competitive performance in RE.

On the other side, recent work also shows the effectiveness of the cascade binary tagging framework [Wei et al., 2020] for solving the overlapping RE. However, this framework suffers from two shortcomings: relation redundancy and poor generalization [Zheng et al., 2021]. Relation redundancy, namely, extracting tail entity extraction to all relations, would cause plenty of meaningless computations. Besides, due to that the cascade binary tagging framework reduces the multi-label problem to a binary classification problem, it needs heuristic threshold tuning and introduces boundary decision errors.

To solve above issues, we introduce an entity type to predefined relation (type-relation) mapping mechanism and a position-dependent adaptive thresholding strategy to improve the cascade binary tagging framework. Specifically, the mapping between head entity type and predefined relation is maintained, when the head entity type is determined, its potential corresponding relations are also determined. Under this mechanism, it avoids traversing for all the relations when predicting overlapping relations. In addition, incorporating entity type information into RE can improve performance [Zhong and Chen, 2021]. The positiondependent adaptive thresholding replaces the global threshold with learnable thresholds for different positions in a sentence when performing binary tagging. The thresholds are learned with a rank-based loss function [Zhou et al., 2021], that pushes positive classes scores above the thresholds and pulls negative classes scores below in training. And the tagger sets a position to 1 that has the higher score than its position-dependent threshold or set to 0 that has the lower score. This simple strategy avoids threshold tuning, and makes the threshold adjustable to different tagging positions, which leads to better generalization.

In this paper, we propose a simple and fast relation extraction model, FastRE, to significantly reduce training and inference time but retains competitive performance. To the best of our knowledge, FastRE is the first effort to address the balance between efficiency and performance in RE. In summary, the main contributions of this paper are three-fold:

• We propose a novel convolutional encoder combined with dilated convolution, gated unit and residual connection for RE, which significantly reduces the computation cost but maintaining the satisfactory performance.
• We improve the cascade binary tagging framework with type-relation mapping mechanism and positiondependent adaptive thresholding, which solves the relation redundancy and poor generalization issues.
• Experimental results on public datasets show that FastRE achieves 3-10× training speed, 7-15× inference speed faster, and 1/100 parameters compared to the

state-of-the-art models, while the performance is still competitive, namely, it is even slightly improved on NYT10 and NYT11, and only reduced by less than 4% on NYT24.

2 Methodology

2.1 ProblemFormulation andModel Overview

Given the predefined relation set R = {r1, r2, ..., rN}, for any sentence S containing the entity set E = {e1, e2, ..., eM} with the entity type set T = {t1, t2, ..., tK}, the relation extraction task aims to extract the relational triples in Z = {(ei, rk, ej)}, where N denotes the number of relations, M represents the number of entities, K is the number of entity types, and ei, ej ∈ E, rk ∈ R.

The principle of FastRE is shown in Figure 1. FastRE consists of a convolutional encoder and a improved cascade binary tagging framework. The convolutional encoder contains L stacked Block(·) which consists of two dilated convolutions, a gated unit and a residual connection. Compared with the original one, the improved cascade binary tagging framework not only adds type-relation mapping mechanism to determine potential relations based on entity types, but also adopts position-dependent adaptive thresholding strategy for different positions. First, the convolutional encoder efficiently converts the input sentence X composed of word embedding Xg and position embedding Xp into sentence representation H. Subsequently, two separate auxiliary features Hh and Ht are generated based on H via two different multihead self-attention layers. Then FastRE utilizes a feed forward network (FFN) to obtain all the head entities and their types. Concretely, sentence representation H and head entity auxiliary features Hh are concatenated and feed to the FNN. Then, for each entity type, FastRE calculates the score of every token as the start and end position of a head entity. With adaptive thresholding, the position that has higher score than its position-dependent threshold is set to 1. Through the mapping mechanism, FastRE determines the potential relations corresponding to the current head entity type. Finally, similar to head entity tagging, with the sentence representation H, tail entity auxiliary features Ht and head entity features Fh, FastRE utilizes another FFN to obtain all the tail entities and forms all the relational triples in Z.

2.2 Convolutional Encoder

Theoretically, we can achieve the aim of modeling arbitrarily long-distance dependencies similar to Transformer [Vaswani et al., 2017] by stacking enough vanilla convolutional layers. But too many parameters will bring too much computation. Therefore, we choose the dilated convolution [Yu and Koltun, 2016] instead because increasing the receptive region in convolution can achieve the high efficiency of vanilla convolution with fewer layers. However, the pre-determined receptive region prevents subsequent layers from examining previous information in detail. We alleviate this issue by using the gated unit [Dauphin et al., 2017] in convolutions to select important features of low-level tokens and phrases. To enable deep networks, we adopt the residual connection [He et al., 2016] to avoid gradient vanishing.

Convolutional Encoder Improved Cascade Binary Tagging Framework

Figure 1: The overall structure of FastRE.

Let X = [x1, x2, ..., xn] represent the input sentence, where xi ∈ Rd is the i-th token embedding with dimension d, composed of GloVe embedding [Pennington et al., 2014] Xg and trainable position embedding Xp. The encoder contains L stacked Block(·). The sentence representation H = [w1, w2, ..., wn] is obtained by:

where H ∈ Rn×d and wi ∈ Rd is the encoded contextual representation of i-th token. The i-th Block(·) in the encoder contains two dilated convolutions with dilation rate di, a gated unit, and a residual connection. It contains d convolution kernels with size ks. We denote dilated convolution as DilatedConv(·) to map the input X ∈ Rn×d to the output Ya,b ∈ Rn×d :

For all the Block(·), we ensure the output dimension matches the input dimension by padding. Similar to [Gehring et al., 2017], we implement a gated unit over Ya,Yb and add a residual connection from the Block(·) input to its output:

where ⊗ denotes the element-wise multiplication, Yi is the output of i-th block and the input of (i + 1)-th block. Obviously, the sentence representation H is equal to YL, which is the output of the last Block(·).

2.3 Improved Cascade Binary Tagger

FastRE first tags all the head entities, then tags their corresponding relations and tail entities, which is a typical cascade binary tagging framework [Wei et al., 2020]. Furthermore, FastRE introduces the type-relation mapping mechanism and position-dependent adaptive thresholding to overcome the drawbacks (i.e., relation redundancy and poor generalization [Zheng et al., 2021]) of cascade binary tagging framework.

Because the convolutional encoder shares the most parameters, it treats every token in the sentence equally and pays less attention to the most important parts of the sentence. Therefore, FastRE generates two separate auxiliary features via two multi-head self-attention [Vaswani et al., 2017] layers for head entity tagging and tail entity tagging, respectively. Take the head entity auxiliary features Hh as the example:

where dk = d is the dimension of the attention key. The Wq,Wk,Wv ∈ Rd×d , bq, bk, bv ∈ Rd are the weights and biases for querys, keys and values to obtain representations Q,K,V ∈ Rn×d , respectively.

For head entity tagging, we concatenate the sentence representation H and head entity auxiliary features Hh as [[w1, wh 1 ], ..., [wn,whn ]]. Then we calculate the score of the i-th token as the start and end position of a head entity with type tj ∈ T , respectively:

where ohs ij represents the score that the i-th token is the start of the head entity with type tj .

Intuitively, different positions should have different thresholds, e.g., the threshold at entity boundary could be different from the boundary of other words. And tuning a global threshold leads to suboptimal results because of the dependent on validation set data distribution, which leads to unsatisfactory test results and poor generalization. To improve the tagging accuracy and model generalization, we use the adaptive thresholding strategy to learn thresholds automatically. The tagger sets a position to 1 only when it has a higher score than its position-dependent threshold. For all start positions psij of head entities with their types tj ∈ T at position i ∈ [1, n], we denote the positive classes (i.e. the positions expected to be tag to 1) as Pi, negative classes as Ni. And we set a AT class to store all the start positions with respect to AT type, where the position i is denoted as pi;AT . During training, the start tagging loss is as follows:

where the oi;AT represents the threshold at position i. Note that the start and end tagging requires different thresholds and follow the same loss function. The head entity tagging loss is the sum of start tagging loss Lsh and end tagging loss Leh :

Finally, we need to identify the relations and tail entities. We set an entity type embedding layer Vt ∈ RK×dt and a relative position embedding layer Vp ∈ Rn×d . First, we obtain the head entity start and end token features wa, wb fromH, derive the head entity type feature wt fromVt and the relative position features wpa , wp b from Vp. Then the (wa + wpa ), (wb + wp b ) and wt are concatenated to form the head entity features wh . Through the type-relation mapping mechanism, we determine the potential relation set R′ based on current entity type, aiming to help identify the tail entity more accurately by incorporating entity type information. We concatenate the sentence representation H, tail entity auxiliary features Ht and head entity features wh as [[w1,wt1 ,wh ], ..., [wn,wtn ,wh ]]. Consequently, we calculate the score of the i-th token as the start and end position of a tail entity with potential relation rj ∈ R′ ⊂ R, respectively:

where ots ij represents the score that the i-th token is the start of the tail entity with relation rj. Similar to head entity tagging loss, we directly calculate the tail entity tagging loss Lt according to Equation (12).

Let D denote all the sentences, and Zi denote all the relational triples in sentence Si. The loss function L is composed of two parts as follow:

3 Experiments

3.1 Datasets, Baselines, and Settings

We evaluate FastRE on three widely used public relation extraction datasets: NYT10 [Takanobu et al., 2019], NYT11 [Takanobu et al., 2019] and NYT24 [Zeng et al., 2018]. The statistics of these datasets are shown in Table 1. We use [PER], [LOC], [ORG] and [OTH] to indicate the four entity types and determine the mapping mechanism between entity types and predefined relations based on the semantics of each relation.

We compare FastRE with the strong state-of-the-art relation extraction models: (1) Tagging based models have been widely explored, such as NovelTagging [Zheng et al., 2017], CasRel [Wei et al., 2020], TPLinker [Wang et al., 2020] and PRGC [Zheng et al., 2021]. (2) Generating based models convert the relation extraction task into a generation task, such as CopyRE [Zeng et al., 2018], WDec [Nayak and Ng, 2020], and SPN [Sui et al., 2021]. (3) Other strong baselines cast the relation extraction task in a reinforcement learning perspective such as HRL [Takanobu et al., 2019], or table filling framework like GRTE [Ren et al., 2021].

The standard micro precision, recall, and F1 score are used to evaluate the results. To fairly compare with existing models, we follow previous work [Takanobu et al., 2019; Wei et al., 2020] and use Partial Match on all three datasets.

To evaluate the robustness and stability of the models, we use the same hyper-parameter setting on all datasets. Specifically, FastRE uses the 300-demensional GloVe embeddings [Pennington et al., 2014] as the initial word vectors with d=128 hidden dimension. The kernel size ks and type embedding dimension dt is 3 and 64, respectively. FastRE applies 6 stacked Block(·) and the dilation rate in each layer are 1, 2, 4, 1, 1, 1. Our model is optimized with AdamW [Loshchilov and Hutter, 2019] using learning rate 1e-3 with a linear warm up [Goyal et al., 2017] for the first 6% steps followed by a linear decay to 0. We apply dropout [Srivastava et al., 2014] between convolutional layers with a rate 0.1. All the hyperparameters reported in this work are based on the results on the validation sets. Other parameters are randomly initialized. Following CasRel [Wei et al., 2020], TPLinker [Wang et al., 2020] and GRTE [Ren et al., 2021], the max length of input sentences is set to 100. We train FastRE for 60 epochs with batch size 32, and all experiments are conducted on a NVIDIA RTX 2080Ti GPU.

3.2 Main Results

According to the main results shown in Table 2, FastRE is very effective compared with the state-of-the-art models based on BiLSTM and BERT, and achieves excellent performance on NYT10 and NYT11. Especially, FastRE outperforms the existing models in terms of training and inference speed on all datasets, which demonstrates the advantages of our convolutional encoder in efficiency. For example, the inference speed of FastRE is more than 7 × as PRGC, 15 × as CasRel and more than 10 × as other BERT based models. And the training speed of FastRE is 3 × as PRGC and more than 10 × as TPLinker. Note that except PRGC, the training time of other BERT based models varies greatly on different datasets. The training time required by FastRE on all datasets is close, although these datasets have different relation numbers. This is mainly because of introducing the mapping mechanism. From the results in PRGC which predicts relations first and extracts entity later, alleviating relation redundancy typically leads a relatively stable training time. In addition, FastRE achieves significantly better performance on NYT10 and NYT11 compared to CasRel, indicating the effectiveness of the improved cascade binary tagging framework.

We observe that the performance on NYT24 of FastRE is not good as the models based on the BERT. We believe that there exists two potential reasons for this phenomenon: (1) insufficient expression ability of FastRE, and (2) irrationality of the NYT24. First, FastRE employs a simple and efficient structure to approximate large Transformer based structures, so as to accelerate model training and inference speed, which limits its expression ability. Nonetheless, FastRE still outperforms BiLSTM based models such as WDec and closes to CasRel, means that the convolution based structure achieves comparable expression ability with pre-trained model BERT on RE task. In addition, the irrationality of the dataset would greatly impact the model performance. As [Zhang et al., 2020] pointed out, over 90% triplets in the NYT24 test set reoccurred in its training set, which easily overfitted by existing models and significantly limits the ability to verify the model generalization. We believe that these BERT based models gain a high score in NYT24 via memorizing the frequently reoccurred training set triplets because of the strong expression and fitting ability of BERT, which causes the overfitting instead of better generalization.

3.3 Inference Efficiency

According to the results shown in Table 3, the total parameters in FastRE are almost 1/100 of other baselines, that ensures the faster forward calculation speed and back propagation update speed. Consequently, very few parameters can help FastRE achieve fast inference. Moreover, FastRE has lower theoretical decoding complexity compared to most models. When there are fewer entity types (m < k) and fewer average number of corresponding relations, FastRE has lower decoding complexity than CasRel. Although SPN has the lowest decoding complexity, the BERT based encoder and the non-autoregressive decoder become the most consuming parts and limit its inference efficiency.

We set the batch size of all BERT based models to 8 for evaluating parallel processing ability. Consistent with the results obtained by [Ren et al., 2021], TPLinker, SPN and GRTE have similar inference speed. FastRE is significantly faster than other models during inference. The results also indicate that the single-thread speed of FastRE is still much faster than other models. Similarly, although CasRel takes the longest time in whole inference stage, it is significantly faster than other BERT based models when processing a single instance, which shows the efficiency of cascade binary tagging framework. When the batch size of FastRE is 8, the speed of FastRE is about 5.5× as PRGC. And we can increase the batch size to 128 because of the simple model structure and the small parameter scale to obtain faster inference speed, which shows the superior parallel processing ability.

3.4 Convergence Efficiency

To evaluate the model convergence efficiency, we analyze the iteration time and total iterations of different models on NYT11 as shown in Figure 2. As different models require different batch sizes for training, we use the official implementation with default batch size configuration. Note that some models such as CasRel and TPLinker set the total epochs to 100 but could perform early stopping when the model performance no longer improving. For example, CasRel typically converges within 30 epochs. Therefore, we only count the time when a model converge, not the total number of epochs, which is quite different with another analysis [Ren et al., 2021]. We believe that our evaluation can more accurately reflect the real convergence efficiency.

The results show that FastRE outperforms other models in both aspects. FastRE consumes the least iteration time because it requires the least computation with least parameters. Note that the iteration time in SPN is less than other BERT based models, but it requires much more iterations to converge as it casts the RE task to a complex bipartite matching problem and contains the most parameters compared to other models. Similar drawbacks occur in GRTE, which performs fussy table-filling with an iterative way and requires the most iterations among all the models. Intuitively, FastRE should have few iterations for convergence with far less parameters than the BERT based model. However, the results show that FastRE has less advantage in the number of total iterations compared with PRGC. This phenomenon motivates us to adopt more effective training strategies in the future.

3.5 Structure Search

We conduct structure search experiment as shown in Table 4. Setting the stacked block numbers to 3 greatly accelerates the model speed, but brings serious performance degradation. By changing the dilation rate of different blocks, we observe that the appropriate dilation rate plays an important role in improving the final performance. For example, when the dilated convolution is not used (the dilation rate is set to 1) or the dilation rate is set too large, the performance suffers grave decline. This is because dilated convolution is hard to capture long-distance dependencies by simply increasing dilation rate exponentially. The few parameters of CNN fail to capture so much information and blindly increasing the receptive region will lead to worse results. Therefore, after a certain number of dilated convolutional layers, we replace the rest layers with vanilla convolutional layers to narrow the receptive region for fine tuning. In addition, continue increasing the number of stacked blocks only brings little performance improvement, but greatly slows down the model convergence speed and inference speed. Therefore, FastRE adopts 6 stacked blocks and sets the dilation rate to 1, 2, 4, 1, 1, 1, which is a structure that balances speed and performance.

3.6 Ablation Study

We conduct ablation experiments to demonstrate the effectiveness of each component in FastRE with F1 scores (%) reported in Table 5. It is important to note that the impact of each component on efficiency is not obvious (except adaptive thresholding), so we only discuss the ablation results on performance.

Dilated Convolution. We set all dilation rates to 1. The dilated convolution can capture more local context information with less parameters and more speed. Without dilated convolutions, the ability to capture long-distance dependencies decreases significantly, resulting in a decline in the overall performance.

Gated Unit. The gated unit part (⊗sigmoid(Yb)) in Equation (4) is removed. The purpose of using gated unit is controlling information flow and selecting the most important information dynamically. Removing the gated units has a greater performance impact than removing dilated convolution and residual connection.

Residual Connection. We remove the residual connection part (+X) in Equation (4). Generally, the residual connection can prevent gradient vanishing, but it still has a positive impact on performance. The residual connection transmits information in multiple channels in multi-layer neural networks, resulting in simpler learning process and better generalization.

Mapping Mechanism. Without the mapping mechanism, namely, we set all entity types to [ANY], and [ANY] can map to all the relations, that causes a small drop in performance. However, one drawback of the mapping mechanism is that it can only deal with the case where the mapping from relation to entity type is uniquely determined. More general methods are worth exploring in the future.

Adaptive Thresholding. We perform sigmoid operation on all the positions and tuning a global threshold. We find that using the global threshold improves the F1 scores on validation sets. However, the F1 scores on test sets with the global threshold are nearly 1% lower than the results under the adaptive thresholding, which indicates that using adaptive thresholding makes the model have better generalization. We also observe that the convergence speed using adaptive thresholding loss is twice that using cross entropy loss. Sigmoid operation is simple but suffers from imbalanced classes issue as it produces plenty of 0 positions which affects the convergence speed. Rank-based adaptive thresholding loss balances the contribution of positive and negative classes in loss, making the training easier and faster.

3.7 Discussion

Other baselines and benchmarks. Some latest work such as REBEL [Huguet Cabot and Navigli, 2021] and RERE [Xie et al., 2021] are not addressed in baseline experiments. Both models have comparable performance but lower efficiency compared to FastRE. REBEL (BART-large as backbone) cannot run in our experimental settings as it requires more computing resources. The parameters in REBEL are 400 × as FastRE and REBEL mainly focuses on performance rather than efficiency, which are different from the goal of our work. FastRE achieves the comparable performance as RERE on NYT10 (73.8 vs 73.8) and NYT11 (56.3 vs 56.2), while the inference speed is 20 × faster than RERE. The reason is that RERE adopts pipeline extraction and uses two separate BERT models in relation classification and entity extraction.

Furthermore, we have conducted systematic experiments on other benchmarks such as ADE and WebNLG, and the results consistently support the findings of our work. Due to space limitation, we cannot report all results.

Trade off between performance and efficiency. The goal of FastRE is to accelerate RE efficiency while maintaining good performance. It employs a simple architecture to achieve a balance between performance and efficiency, because increasing the model complexity will decrease the efficiency. We examined the NYT24 and found that 90.49% triplets overlapping in test set and training set, while it is only 72.64% and 41.62% in NYT10 and NYT11. This would leads BERT based models and Seq2seq based models to gain high scores because of rote memorization and exposure bias. Therefore, the results on NYT24 is not suitable to test the model generalization. Finally, we directly replaced the BERT encoder in CasRel with our convolutional encoder and the results show that the performance drops 1.8%, 0.7%, 3.3% on three benchmarks, which shows that the representation ability of convolutional encoder is not as good as BERT on RE task. To sum up, FastRE has somewhat limited representation ability as its simplicity, but it achieves better generalization and is more suitable and applicable for unbiased datasets like NYT11 rather than NYT24.

Explainability. FastRE improves training and inference speed while keeps promising performance, and it could be somewhat explained by the gradient techniques. Firstly, we visualized the distribution of attention scores on two separate self-attention layers, and found that the score of entity boundary is significantly higher than that of other words in both layers, which highlights the boundary features for subsequent entity prediction. Secondly, we collected the gradient of the loss w.r.t the input and computed the norm of the gradient for each word vector in the input. After normalization on each word gradient norm, we found that the important predicates in sentences (which indicate the relationships between entities) always receive higher (relative) gradient norms, while trivial conjunctions and prepositions have gradient norms close to 0.

4 Conclusion

This paper proposes a simple and fast relation extraction model, FastRE, to significantly reduce training and inference time but retain competitive performance. We propose a novel convolutional encoder combined with dilated convolution, gated unit and residual connection for RE. We also improve the cascade binary tagging framework with typerelation mapping mechanism and position-dependent adaptive thresholding. Experimental results show that FastRE achieves faster training and inference speed compared to the state-of-the-art models, while the performance is competitive.

Acknowledgements

We thank the anonymous reviewers for their insightful comments. The work was supported in part by the 13th Five-Year All-Army Common Information System Equipment PreResearch Project (Grant Nos. 31514020501, 31514020503). All opinions are of the authors and do not reflect the view of sponsors.