Position-aware Joint Entity and Relation Extraction with Attention Mechanism

1 Introduction

Named entity recognition and relation extraction are two core subtasks of information extraction. Early research methods can be divided into two main categories: pipeline methods and joint methods. The pipeline methods usually need to train two models, one is used to identify named entities, and the other is used to extract the relation between entitiess [Zhong and Chen, 2020; Ye et al., 2021]. The joint approach is to model the two tasks of named entity recognition and relation extraction jointly [Luan et al., 2018; Eberts and Ulges, 2019; Ji et al., 2020; Lin et al., 2020; Wang and Lu, 2020], either by projecting them into a structured prediction framework or by performing multi-task learning with a shared representation.

∗Corresponding author

In this work, we propose a joint entity and relation extraction model with an attention mechanism and positionattentive markers, and our encoder for the joint model is built on top of a deeply pre-trained language model [Devlin et al., 2018; Beltagy et al., 2019; Lan et al., 2019]. Unlike previous work based on BIO/BILOU labels [Bekoulis et al., 2018b; Li et al., 2019; Nguyen and Verspoor, 2019], the potential entities of our model are represented by span, which can identify overlapping entities. Although span-based representation models can identify overlapping entities, they also bring a large number of negative samples, and such a large number of negative samples of candidate entities will bring noise to the model and hurt the performance of the model. To address this issue, we consider to filter out some negative samples of candidate entities by designing a specific attention mechanism. On the other hand, the position information of the words in the input sentences helps to improve the performance of the model. We finally desire to know whether combining the above two into a unified model can further improve the overall performance. Our model is called PERA (Position-aware Joint Entity and Relation Extraction with Attention Mechanism).

Using the same pre-trained encoders, our model outperforms all previous joint models on three standard benchmarks: ACE05, SciERC and CoNLL04, advancing the previous state-of-the-art 4.7%-17.8% in relation F1. The experimental results show that, (1) introducing the attention mechanism to reduce the number of negative samples of candidate span can improve the performance of the model in named entity recognition; (2) adding position-attentive markers to the model can help improve the overall performance of the model in named entity recognition and relation extraction; (3) attention mechanism and position-attentive markers can complement each other, and combining them can further improve the model's performance.

In summary, we summarize our contributions as follows:

• We propose a new method for joint entity and relation extraction that fuses attention mechanisms and positionattentive markers, where entity recognition and relation extraction share the same encoder. Our model PERA is built on three standard benchmarks and outperforms all previous joint models.

• We conduct a series of experiments to understand why our method performs so effectively. We find that a large number of negative samples of candidate entities hurts the performance of the model, and removing some of the negative samples before feeding candidate spans to the span classifier can improve the performance of the model.
• We conduct a careful analysis to examine how different factors affect the final performance of our model. The experimental results show that attention mechanism and position-attentive markers can complement each other, and combining them together can further improve the performance of the model.

2 RelatedWork

In the past, named entity recognition and relation extraction in the text were generally regarded as two independent tasks. However, in recent years, joint entity and relation extraction have attracted attention and there has been a surge of research interest in joint entity and relation extraction models. At the same time, the span representation method has received extensive attention from the academic community and is widely used to extract entities and relations from plain texts.

Span-level representation Since the BIO/BILOU-based model can only give each token a tag, and a token cannot be part of multiple entities at the same time, this does not cover the case of overlapping entities. Our method is based on a span representation approach, which performs an exhaustive search for all spans and effectively solves the problem of entity overlap. This uses span representation in both coreference resolution [Lee et al., 2017; Lee et al., 2018] and semantic role labeling [Ouchi et al., 2018].

Recently, some joint entity and relation extraction models [Luan et al., 2018; Dixit and Al-Onaizan, 2019] based on span representation have been proposed, using the span representation of BiLSTM. [Dixit and Al-Onaizan, 2019] study a relation extraction model based on span-level representation. [Luan et al., 2018] perform a beam search on the hypothesis space to estimate which spans are involved in entity classes, relations and coreferences. DyGIE++ [Wadden et al., 2019] has replaced the BiLSTM encoder with BERT and is a Transformer-based span method for joint entity and relation extraction.

Joint model In recent years, joint entity and relation extraction models have been intensively studied and explored. [Li and Ji, 2014] propose an incremental joint framework to extract mentioned entities and relations simultaneously using structure perceptrons and effective cluster search. [Miwa and Sasaki, 2014] adopt a history-based structured learning approach to jointly extract entities and relations in sentences. [Zheng et al., 2017] use an annotation model containing information about entities and relations to transform the joint extraction into an annotation problem. [Zhang et al., 2017] build a globally optimized neural network model to extract end-to-end relations and propose LSTM features to learn contextual representations. [Fu et al., 2019] present an end-to-end relation extraction model GraphRel, which uses GCN to jointly learn named entities and relations. [Li et al., 2019] propose an approach of multi-turn question answering to solve entity and relation extraction. [Wang and Lu, 2020] treat the joint extraction task as a form-filling problem. These methods have a common feature, and it is necessary to find a good global optimization in the two tasks of named entity recognition and relation extraction.

[Miwa and Bansal, 2016] present a model for entity detection and relation extraction separately, while the two parts form an overall single model by stacking and sharing their respective parameters so that named entity recognition and relation extraction interact with each other. [Bekoulis et al., 2018a] model the relation extraction task as a multi-label head selection problem. [He et al., 2018] adopt an end-to-end method to jointly predict all predicates, parameter ranges and their relation. The model makes independent decisions about the relation between each possible word span pair, and learns the span representation of the context, providing rich shared input features for each decision. [Lin et al., 2020] proposes an information extraction framework called ONEIE, which adds a global feature to make joint decisions between instances and subtasks. These methods are similar to pipeline methods. There is a sequential order between the two tasks, with named entity recognition performed first and then relation extraction performed based on the predicted entities.

The closest to our work is coreference resolution [Lee et al., 2017] and DYGIE [Luan et al., 2019]. In their work, the key idea of DYGIE [Luan et al., 2019] is to use a dynamic graph propagation layer to achieve a shared span representation between two tasks. [Lee et al., 2017] use the attention mechanism combined with pruning to train to obtain the most likely reference span in the common reference cluster. We will show our method in detail in Section 3.

3 Method

In this section, we first formally define the named entity recognition and relation extraction problem in Section 3.1, and then describe our approach in detail in Sections 3.2 and Section 3.3. Finally, we present the details of the training and inference in Section 3.4.

3.1 ProblemDefinition

The input of the problem is a sentence X consisting of n tokens, X = {x1, x2, ..., xn}. Let S = {s1, s2, .., sm} be all the possible spans in X of up to length L, where si consists of tokens (xa(i), ..., xb(i)) and a(i) and b(i) denote start and end indices of si. The problem can be decomposed into two sub-tasks: named entity recognition and relation extraction.

Named entity recognition Let E to be a pre-defined set of entity categories. The named entity recognition task is, for each span si ∈ S, to predict an entity type ye(si) ∈ E ∪ {none}. none represents spans that do not constitute entities. The output of the task is Ye = {(si, e) : si ∈ S, e ∈ E}.

Relation extraction Let R denote a set of predefined relation classes. The task is, for every pair of spans si ∈ S, sj ∈ S to predict a relation type yr(si, sj) ∈ R ∪ {none}. {none} represents there is no relation between si and sj . The output of the task is Yr = {(si, sj, r) : si, sj ∈ S, r ∈ R}.

3.2 Named Entity Recognition

As shown in Figure 1, our method consists of named entity recognition and relation extraction. First, a raw sentence is input, tokenized to obtain the tokens corresponding to the sentence, and then these tokens are fed into the pre-trained language model to obtain a contextual representation of each token. Note that in order to clearly show our approach, Figure 1 does not show tokenization of the original input sentences. As in previous work [Lee et al., 2017; Luan et al., 2019; Wadden et al., 2019], our approach is a standard span-based model. For each input token xt, we first use a pre-trained language model (e.g., BERT) to obtain contextualized representations ht. Given a span si = (xa(i), ..., xb(i)) ∈ S, the span representation Ee(si) contains He(si) and Me(si).

He(si) is defined as:

where ha(i) and hb(i) respectively denotes the contextual representation of xa(i) and xb(i), w(si) ∈ RdF denotes the learned embeddings of span width features.

Me(si) is defined as:

where B(i) indicates the position ID of the word where span si starts, E(i) indicates the position ID of the word where span si ends. mB(i) and mE(i) respectively denotes the contextual representation of the position-attentive marker [BB(i)] and [EE(i)].

Therefore, Ee(si) can be defined as:

where [A; B; C; D; E] denotes the concatenation operation on the vector A, B, C, D, and E.

Inspired by [Lee et al., 2017], we incorporate an attention mechanism in the entity naming entity recognition phase. For all the possible spans S = {s1, s2, .., sm}, we first compute its attention score A = {α1, α2, .., αm}. S = {s1, s2, .., sm} is sorted according to their scores A = {α1, α2, .., αm}, and only those span si with high scores αi have the chance to be retained and fed into the span classifier. Before sorting, we set the attention score αi of the positive samples si ∈ Sgold ⊂ S to the maximum value. For example, if si ∈ Sgold ⊂ S, we let the attention score αi = MAX, where MAX = max(α1, α2, .., αm). At the same time, we set the attention score αj to the minimum for those padded negative samples sj due to batch computation. For example, if sj is a padded negative sample due to batch computation, we let the attention score αj = MIN, where MIN = min(α1, α2, .., αm). Finally, all possible spans S = {s1, s2, .., sm} are ranked from highest to lowest according to the attention score A = {α1, α2, .., αm}.

In our work, the number of reservations n = λm, where λ is the retention factor. By controlling the size of the retention factor λ, not only the number of candidate spans input to the span classifier can be controlled, but also a part of the padded negative samples can be removed.

When some of the negative samples are removed from S = {s1, s2, .., sm}, we feed span representation E′ = {Ee(s′1 ), Ee(s′2 ), ..., Ee(s′n )} of the retained candidate spans S′ = {s′1 , s′2 , .., s′n } to a feedforward network to predict the probability distribution of the entity type e ∈ E ∪ {none} : Pe(e | s′i ).

3.3 Relation Classification

A simple way to predict the relation between si and si is use the span representation Ee(si) and Ee(sj) of the named entity recognition phase [Luan et al., 2019; Wadden et al., 2019]. However, there are some differences with [Luan et al., 2019; Wadden et al., 2019], our span representation Ee(si) has the position-attentive information Me(si) = [mB(i);mE(i)], as shown in the white rectangle wireframe in Figure 1. Here, we do not include the width feature embedding of span, because the width feature embedding of span may capture the semantic information of entities, but may fail to capture the semantic information of the relation between two entities. As shown in Figure 1, our span-pair representation Rr(si, sj) can be defined as:

Finally, the representation Rr(si, sj) will be fed into a feedforward network to predict the probability distribution of the relation type r ∈ R∪ {none} : Pr(r | si, sj).

3.4 Training and Inference

Our training is supervised. Given sentences with entities (including their entity types) and relations (including entity pairs and types of relation), we define a joint loss function for named entity recognition and relation classification:

where Le is the cross-entropy loss of named entity recognition, Lr is the cross-entropy loss of relation classification. These two losses are averaged on each batch of samples.

In relation extraction tasks, the number of negative samples of relations is often much larger than the number of positive samples. Therefore, in our model, we reduce the number of relation negative examples before feeding candidate entity pairs into the feedforward neural network to predict the relation type.

Here, we call the entities in the entity pair whose relation type belongs to R as relational entities, and the set of relational entities is denoted as Srel = {srel1, srel2, ..., sreln}.

For example, if (si, sj) ∈ R, then si, sj ∈ Srel, and si, sj ∈ Sgold. Obviously, Srel ⊂ Sgold ⊂ S. For example, suppose there are 7 gold entities Sgold = {sgold1, sgold2, ..., sgold7} in a sentence, of which only 3 entities {sgold1, sgold2, sgold3} in the entity pair whose relation type belongs to R, such as (sgold1, sgold2) ∈ R and (sgold2, sgold3) ∈ R. Then we have Srel = {sgold1, sgold2, sgold3}. In the training relation classification phase, we only consider the relational entities Srel = {srel1, srel2, ..., sreln} and use the relational entities labels in the training set. During inference, we first predict the entities type by performing ye(si) = argmaxe∈E∪{none}Pe(e | si). When all the prediction results Spred = {si : ye(si) ̸= none} are obtained, we enumerate all the spans si, sj ∈ Spred and use si, sj to construct the input Rr(si, sj) for the relation classification Pr(r | si, sj).

4 Experiments

4.1 Setup

The following content shows the setup details of the experiments.

Datasets We use three popular relation extraction datasets: ACE05, CoNLL04 and SciERC. Table 2 shows the statistical information of each dataset. The ACE05 dataset consists of English, Arabic, and Chinese data collected from various domains, such as newswire and online forums. The ACE05 dataset can be used for entity, relation, and event extraction. We use the English part of the ACE05 dataset data. The CoNLL04 dataset contains named entities with annotations and sentences of relations extracted from news articles. SciERC is derived from abstracts of 500 artificial intelligence papers. We follow previous work and use the same preprocessing procedure and splits for the ACE05 and SciERC datasets. For the CoNLL04 dataset, we adopt the training set (1,153 sentences) and test set (288 sentences) split by [Gupta et al., 2016]. To tune the hyperparameters, 20% of the training set is used as the development set.

Evaluation metrics We use the micro F1 measure as the evaluation metric and follow the standard evaluation protocol. (a) For named entity recognition, if a predicted entity span boundaries and its entity type are both correct, we considered this predicted entity as a correct prediction. (b) For relation extraction, considering different situations, we use two evaluation metrics: 1) boundary evaluation (Rel): if the boundary of the two spans is correct and the predicted relation type is correct, then the predicted relation is considered to be a correct prediction. 2) strict evaluation (Rel+): in addition to the requirements in boundary evaluation, the entity's prediction type must also be correct.

Implementation details For fair comparison with previous work, we adopt bert-base-uncased [Devlin et al., 2018] and albert-xxlarge-v2 [Lan et al., 2019] as the base encoders for ACE05 and CoNLL04. Because the pre-trained model scibert-scivocab-uncased [Beltagy et al., 2019] is shown to be more effective than BERT [Wadden et al., 2019], we used it as the base encoder for SciERC. We adopt a cross-sentence contextual text. We expand each sentence according to the context and make sure that the original sentence is located in the middle of the expanded sentence as much as possible. For datasets ACE05, CoNLL04 and SciERC, we set the maximum length of the extended sentences to 256. We consider spans up to L = 8 words. On the data sets ACE05, CoNLL04 and SciERC, we take the retention factor λ = 0.05.

4.2 Main Results

In Table 1, we compare our model PERA with all previous joint model results and pipeline model results. Our model uses cross-sentence contextual text, so our F1 scores are cross-sentence F1 scores. As shown in Table 1, our model achieves better performance compared to previous work. Compared to all previous work, our BERT-base model and SciBERT model achieve similar or better results, and our experimental results are further improved by using a larger encoder ALBERT.

For named entity recognition, our method outperforms the previous state-of-the-art, respectively achieving +1.1%, +6.6%, +4.3% absolute F1 improvements on ACE05, SciERC, and CoNLL04. For relation extraction, our method respectively outperforms the previous state-of-the-art with absolute F1 values of +4.7%, +5.2%, and +17.8% on ACE05, SciERC and CoNLL04.

Compared to [Wang and Lu, 2020], our model obtained 2.5% higher F1 for named entity recognition and 6.5% higher F1 for relation extraction using a similar ALBERT pre-trained model on ACE05. Our approach is much more effective than previous best approaches using global features [Lin et al., 2020] or complex neural models such as MT-RNNs [Wang and Lu, 2020], and achieves large improvements on all datasets. This improvement demonstrates the effectiveness of incorporating attention mechanisms and position-attentive markers in a joint entity and relation extraction model. We also note that compared to the previous state-of-the-art model PURE [Zhong and Chen, 2020] based on a similar ALBERT, our model shows a significant improvement not only in entity F1 (90.9 vs 92.0) but also in relation F1 (69.4 vs 74.1). This clearly demonstrates the superiority of our model.

5 Analysis

We effectively combine attention mechanisms and positionattentive markers into a joint entity and relation extraction model, and experimental results show that our approach outperforms all previous joint models. In this section, our goal is to go deeper and understand what factors determine its final performance.

5.1 Effect of Negative Sample Size

The model based on the span representation can exhaust all possible entities and therefore can identify overlapping entities. However, the span representation method generates a large number of negative samples of candidate entities, and such a large number of negative samples of candidate entities will bring noise to the model and hurt the performance of the model. For example, if a sentence with n tokens is input, then the number of all its possible candidate spans is O(n2 ), but in general the number of gold entities in this sentence is only a few. Our main observation is that it is crucial to reduce the number of input negative samples during the named entity recognition phase. As shown in Figure 2, the Rel F1 basically does not change with the change of retention factor, while the Ent F1 and Rel+ F1 increase drastically with the decrease of retention factor. Therefore, an intuitive conclusion is that a large number of negative samples of candidate entities hurts the performance of the model, and removing some of the negative samples before feeding candidate spans to the span classifier can improve the performance of the model.

5.2 Importance of Position Information

In our model, we append position-attentive markers at the end of the original input sentences with the aim of allowing the model to learn the position information of the original input words to help improve the performance of the model. The experimental results prove that our approach is effective. As shown in Table 3, adding position-attentive markers to the model improved Ent F1 by 0.5% and Rel F1 by

0.2% on dataset ACE05, and Rel F1 improved by 0.9% on dataset CONLL04. Therefore, a reasonable explanation is that adding position-attentive markers to the model enables the model to learn the positional information of the input words. The position information of words is helpful for both named entity recognition and relation extraction.

5.3 Combined Effect

Table 3 shows that although the methods of adding only attention mechanisms or only position-attentive markers to the base model improved the performance of the model compared to the base model, they did not achieve the best performance of our model. The experimental results show that combining attention mechanisms and position-attentive markers in the model can bring out the best performance of the model.

Both adding the attention mechanism to reduce the number of negative samples and adding the position-attentive markers can improve the performance of the model, but they improve the model in different directions. To summarize, (1) introducing the attention mechanism to reduce the number of negative samples of candidate span can improve the performance of the model on named entity recognition; (2) adding position-attentive markers allows the model to learn the position information of the words in the input sentence, which is helpful for both named entity recognition and relation extraction; (3) the results of the ablation experiments show that attention mechanism and position-attentive markers can complement each other, and combining them together can further improve the performance of the model.

6 Conclusion

In this paper, we present a joint entity and relation extraction model that fuses attention mechanism and position-attentive markers. Our model PERA shares the same encoder for named entity recognition and relation extraction, and our experimental results show that it significantly outperforms previous state-of-the-art on three standard benchmarks.

Acknowledgments

This work was supported by National Natural Science Foundation of China (No.62072112, 62176185), Scientific and Technological Innovation Action Plan of Shanghai Science and Technology Committee (No.20511103102), Fudan Double First-class Construction Fund (No. XM03211178).