A Relation-Specific Attention Network for Joint Entity and Relation Extraction

1 Introduction

Jointly extracting entities and relations is to capture structural knowledge in the form of (head, relation, tail) from unstructured texts. The process can promote many graph-based tasks in data mining and NLP fields, such as knowledge graph construction [Luan et al., 2018] and graphical dialogue system [Liu et al., 2018]. Traditional pipelined extraction systems [Zelenko et al., 2003; Chan and Roth, 2011] treat entity and relation extractions as two separate tasks, which perform relation classification after the recognition of all the entities in the text. Such models suffer from error propagation and ignore the relevance between the two subtasks [Li and Ji, 2014]. Thus, many researchers focus on building joint models to simultaneously extract entities and relations.

Prior joint learning methods [Kate and Mooney, 2010; Miwa and Sasaki, 2014] depend heavily on complex feature engineering and other off-the-shelf NLP tools. The later studies concentrate more on learning neural networkbased models, and some of them apply parameter sharing strategy for jointly training [Miwa and Bansal, 2016;

∗Corresponding Author

Katiyar and Cardie, 2017]. Although these neural methods perform better than the former, they still make predictions separately on extracting entities and relations, and the connections between the two subtasks are not fully utilized. Recently, a NovelTagging model [Zheng et al., 2017] combines the two tasks as a single sequence labeling problem. However, a word cannot be assigned with more than one tag, so that the model fails to extract the triplets with overlapped entities (see the examples in Figure 1).

To address the overlapping issue, many entity-guided joint learning methods, such as PA-LSTM [Dai et al., 2019] and ETL-Span [Yu et al., 2020] are proposed. They perform head entities recognition as the first step, and develop some joint decoding strategies for extracting the corresponding tail entities and relations. On the contrary, CopyRE [Zeng et al., 2018] and HRL [Takanobu et al., 2019] present a relationguided joint extraction process, which takes relation classification as the first step of their models. It is because relations are usually triggered by the context of sentences rather than target entities. For example, descriptions like 'was born in' in the sentence will directly lead to the place of birth relation. Thus, the relation information can be first introduced as prior knowledge and reduce the model's focus on semantically unrelated entities, which avoids the redundant extraction operations on them. However, CopyRE and HRL simply utilize the results of relation classification as the guidance of entity extraction, ignoring the fine-grained semantic connections between relations and the words in the sentence. We argue that the words should have different contributions to the underlying semantic expression of the sentence under different relations. Based on this assumption, we use attention mechanism for assigning higher weights to the relationrelated words in the sentence.

In this paper, we propose a relation-specific attention network (RSAN) for joint entity and relation extraction. We use relation-based attention to construct the specific sentence representation under each relation, and then perform sequence labeling to extract its corresponding entities. Our model can not only capture fine-grained semantic features from the words, but also effectively solve the overlapping problem by decomposing the extraction task into separate entity tagging processes for different relations. Moreover, we employ a relational gate to reduce the noise brought by the unrelated relations in entity recognition. During training, we further use a relation-level negative sampling strategy to avoid most of the redundant decoding processes. In summary, the main contribution of this paper are as follows:

• We present a joint entity and relation extraction model named RSAN, which incorporates the relation finegrained semantic information to guide the entity recognition process. Our RSAN is suitable for extracting the overlapping triplets as it performs entity extraction for different relations separately.
• We apply relation-based attention mechanism to construct different sentence representations under different relations, and propose a relational gated mechanism to adaptively control the relation information provided for entity decoding.
• Training with a relational negative sampling strategy, our model achieves state-of-the-art results on two public datasets, which proves its effectiveness.

2 RelatedWork

Researchers have made great efforts in relational facts extraction, which can be directly used for knowledge graph construction, or supporting downstream text mining tasks. Early methods [Zelenko et al., 2003; Chan and Roth, 2011] regard entity and relation extractions as two separate subtasks, which apply pipelined approach to perform relation classification after extracting all the entities. To construct the bridge between the two subtasks, building joint model that extracts entities together with relations simultaneously has attracted much attention. The prior feature-based models [Kate and Mooney, 2010; Miwa and Sasaki, 2014] rely on other NLP tools to do feature engineering, and suffer from the error propagation. The later works are mainly based on neural architectures, which can be roughly divided into Table-filling, Tagging and Seq2Seq methods.

Table-filling methods [Miwa and Sasaki, 2014; Gupta et al., 2016] construct a table for each sentence and specify an order of the table cells, incrementally filling the table with entity or relation tags. The recent work GraphRel [Fu et al., 2019] can also be seen as table-filling methods, which applies 2-phrase Graph Convolutional Network (GCN) to predict word entities and relations for each word pair. Due to Text: United States President Donald Trump was born in New York City

(Donald Trump, President_of, United States) (Donald Trump, Born_in, New York City)

Figure 2: An example for our relation-specific tagging scheme. For different given relations, we will generate a specific tag sequence for each of them.

the sparsity of the tables, these methods suffer from plenty of redundant predictions. Tagging methods usually use welldesigned tagging strategies to construct connections between entities and relations. Among these methods, NovelTagging [Zheng et al., 2017] first treats entity types and relation roles as different parts of the tag, and models the joint extraction task as a single sequence labeling problem. However, it cannot handle the overlapping cases. As the improvement, [Takanobu et al., 2019; Dai et al., 2019; Yu et al., 2020] perform the tagging process for multiple turns. The Seq2Seq methods try to directly generate all triplets sequentially. CopyRE [Zeng et al., 2018] generates the relation followed by its two corresponding entities with copy mechanism, but only the last word of the entity can be generated. Therefore, CopyMTL [Zeng et al., 2019a] applies a multi-task learning framework to extract multi-token entities. WDec [Nayak and Ng, 2020] designs a new representation scheme for the triplets, and then generates triplets as word sequences. OrderRL [Zeng et al., 2019b] applies reinforcement learning to optimize the extraction order of triplets.

In this paper, we propose a new tagging method named RSAN to decode the specific entities for each relation, which is helpful for dealing with overlapping problem. Different from the above mentioned models, we use attention mechanism to incorporate fine-grained relation information as the guidance of the entity tagging process.

3 ProblemFormulation

We describe the relational triplets as {π = (h, r, t) | h, t ∈ E, r ∈ R} and a sentence as S = {w1, w2, ..., wn}, where E and R are the entity and relation sets respectively, a triplet π indicates entity pair (h, t) and relation r between them, and wi is the i-th word in the sentence. In this paper, given a sentence S and a predefined relation set R, the purpose of the joint entity and relation extraction task is to identify all existing triplets π from S.

Note that the extracted triplets may share the same entities or relations, i.e. the overlapping problem. Thus designing a joint extraction model to overcome such issue is a big challenge in this task.

4 Methodology

In this section, we will first introduce our tagging scheme which transforms overlapping triplets extraction task to several sequence labeling problems. Then we elaborate the details of our relation-specific attention network based on a certain relation.

4.1 Tagging Scheme

We incorporate head and tail roles {H,T} in the triplets into the typical BIES signs (Begin, Inside, End, Single) as our entity tags. For a sentence with multiple triplets, we will generate separate tag sequences according to different relations. In the tag sequence of a certain relation, only its corresponding head and tail entities will be annotated, while the rest of words are assigned with label O. Figure 2 shows an example of our extracting method. There are two triplets in the sentence: (Donald Trump, President of, United States) and (Donald Trump, Born in, New York City), we will perform sequence labeling for the relation President of and Born in separately. As we can see, the two triplets have the overlapped entity Donald Trump, and they can be extracted without conflict based on the separate labeling operations.

Besides, when multiple triplets share the same relation, i.e., the relation overlapping cases, we follow [Zheng et al., 2017] and apply the heuristic nearest principle to combine the entity pairs. Concretely, the nearest head and tail entities will be combined into a triplet.

4.2 Relation-Specific Attention Network

Figure 3 gives an overview of RSAN under a certain relation rk. Note that the extracted entities will be directly combined with the current relation rk, thus there is no extra relation classification operations in our model. We first encode the input sentence with a bi-directional Long Short Term Memory (BiLSTM) network [Hochreiter and Schmidhuber, 1997], and then apply attention mechanism to construct the specific sentence representation of rk. After filtering by relational gate, the final representation of the sentence will be used for the sequence labeling process to extract the corresponding entities.

BiLSTM Layer

Given a sentence S = {w1,w2, ...,wn} of length n, we denote xi = [ww i ;wp i ;wc i ] as the representation of the ith word, where ww i ∈ Rdw is randomly initialized word embedding, wp i ∈ Rdpos is the part-of-speech (POS) embedding, and wc i ∈ Rdc is character-based word features. The character-level word features are extracted by a convolution neural network (CNN) running on the character sequence of wi. Then we choose BiLSTM to capture the dependencies of the words. The sequence of word representations {x1, x2, ..., xn} are taken as the input of BiLSTM network. We concatenate the forward and backward LSTM hidden states of xi as the contextual word representation:

where hi ∈ R2×dhe , and dhe indicates the dimension of the BiLSTM hidden state. Then we use Sc = {h1, h2, ..., hn} to represent the context-level sentence features.

Relation-Based Attention Mechanism

Based on our assumption, the words in the sentence play different roles under different relations. To this end, we propose a relation-based attention mechanism for assigning different weights to the context words under each relation. The attention score is obtained as follows:

where rk ∈ Rdr is the trainable embedding of the k-th relation, and v ∈ Rdatt , Wr ∈ Rdatt×dr , Wg, Wh ∈ Rdatt×2dhe are trainable parameters. Here sg indicates the global representation of the sentence. In this way, the attention score can not only measure the importance of each word to the relational expression, but also its contribution to the entire sentence. The specific sentence representation sk under relation type rk is then generated by weighted sum of the sentence words,

Relational Gated Mechanism

So far we have obtained sentence representations fused with relation information. As we argued before, the relationoriented representations make sense to the followed entity extraction only when the relation is positive to the sentence, while that of the unrelated relations will only confuse the subsequent decoding process. In order to adaptively control the relation information provided by the previous attention layer, we propose a gated mechanism as the bridge. Still taking the k-th relation as an example, the gated operations are defined

where W1, W2, W3 ∈ Rdg×2dhe , b1, b2, b3 ∈ Rdg are parameters, ⊕ is concatenating operation, and ⊙ is dot product. σ indicates the element-wise sigmoid activation function, which returns values from 0 to 1, therefore the results can be viewed as percentage of information to keep. The purpose of Eq. 5 is to measure whether the inherent sentence representation sg or the relation-based representation sk is more useful for the entity extraction. uk is the reserved relational features. We concatenate hi and uk to obtain the final representation of the i-th word.

here hki ∈ R2dhe+dg . Sentence S is thus represented as Sk = {hk1 , hk2 , ..., hkn }, and will be used for the entity extraction process.

Relation-Specific Entity Decoder

We perform a relation-specific sequence labeling process as the entity decoder. Here we run another BiLSTM network on the word sequence Sk , and map each of the word to the tag space:

where i ∈ [1, n]. Wo ∈ R2dhd×nl , bo ∈ Rdnl are parameters, and dhd is the dimension of hidden state of BiLSTM, nl is the total number of tags. P(yk i ) indicates the probability of i-th word's predicted tag under relation rk.

Training

Notice that the number of relations present in a sentence is much smaller compared to the size of R. If we perform entity decoding for all given relations during training, there will be a large amount of negative samples, which makes it difficult for convergency. Therefore, we apply a relational negative sampling strategy, i.e., randomly select nneg relations from the negative set of the current sentence. Here nneg is a hyperparameter. All of the words will be labeled with tag O since there are no triplets based on those negative relations. Then for a sentence S with nsp positive relations, our model will totally generate ns = nsp +nneg tag sequences while decoding. We use negative log-likelihood (NLL) loss function to train our model. We denote the ground truth labels under relation rk as {ˆ yk 1 , ˆ yk 2 , ..., ˆ ykn }, then the NLL loss can be defined as:

5 Experiments

5.1 DataSets

Following [Zeng et al., 2018], we evaluate our model on two widely used datasets: New York Times (NYT) and WebNLG. NYT is first constructed using distant supervised method, which automatically aligns knowledge base and plain texts to generate large-scale training data. WebNLG is created by [Gardent et al., 2017] for Natural Language Generation (NLG) task, and all of the standard sentences are written by annotators. To be consistent with all other baselines, we only select the first standard sentence in each instance to reconstruct the corpus. Statistics of the two datasets are shown in Table 1.

5.2 Implementation Details

We set the dimension of word embedding dw = 100, POS embedding dpos = 15, character embedding dc = 50, and relation embedding dr = 300. All of those embeddings are randomly initialized. The window size of CNN for character-based word feature vector is set to 3, the maximum length of words is set to 10, and the number of filters is 50 (df = 50). Hidden State of the encoder BiLSTM (dhe), attention (datt), gate (dg) and the decoder BiLSTM (dhd) are all set to 300 dimensions. The sentence-level relational negative sampled number nneg is set to 4. The model is trained using Adam [Kingma and Ba, 2014] with learning rate of 0.001 and batch size of 16. We apply dropout mechanism to the embedding layer with a rate of 0.5 to avoid overfitting.

5.3 Baselines and EvaluationMetrics

We compare our model with the following baselines:

• NovelTagging [Zheng et al., 2017] applies a novel tagging strategy that incorporates both entity types and relation roles, and converts the joint extraction task to a sequence labeling problem. This model cannot extract triplets with overlapping entities.

• CopyRE [Zeng et al., 2018] first explores Seq2Seq model for the joint entity and relation extraction task, and generates the triplets in the sentence sequentially using copy mechanism. This model can only copy the last word of an entity.
• GraphRel [Fu et al., 2019] constructs a complete word graph for each sentence, and employs GCN to predict relations between all word pairs.
• CopyMTL [Zeng et al., 2019a] improves the copy strategy of CopyRE, and applies a multi-task learning framework to solve the problem of generating multi-token entities.
• OrderRL [Zeng et al., 2019b] incorporates reinforcement learning into Seq2Seq model to learn the extraction order of triplets.
• HRL [Takanobu et al., 2019] applies a hierarchical paradigm which performs relation detection first as a highlevel reinforcement learning process, then identifies entities as a low-level one.
• ETL-Span [Yu et al., 2020] applies a novel decomposition strategy, which first distinguishes all head entities, and then identifies corresponding tail entities and relations.
• WDec [Nayak and Ng, 2020] proposes a novel triplets representation scheme and employs Seq2Seq to generate the word sequences.

We use standard Precision (Prec), Recall (Rec) and F1 score as our evaluation metrics. A triplet is considered to be correctly extracted if and only if its relation type and two entities are exactly matched.

5.4 Results

Table 2 shows all of the comparison results. Overall, our RSAN outperforms all other baselines. We attribute the gains of RSAN to its two advantages: (1) RSAN focuses more on the relation-related entities, which excludes the error caused by predictions on the redundant entity pairs; (2) The relationattentive entity tagging process has the ability to capture the dependencies between the extraction of entities and relations.

In addition, our RSAN also achieves higher performance among the relation-guided methods, like CopyRE [Zeng et al., 2018], OrderRL [Zeng et al., 2019b] and HRL [Takanobu et al., 2019]. We consider that is because our attention mechanism incorporates fine-grained relation information, which enables more explicit guidance on the entity extraction process.

6 Analysis

6.1 Ablation Study

We conduct ablation experiments to demonstrate the effectiveness of POS embedding, character-level word embedding, relation-based attention mechanism and the relational gate in our model. We remove one component at a time to observe its impact on the experimental results, which is summarized in Table 3. (1) POS embeddings in the input layer effectively provide additional syntactic information to the sentence. (2) The character-level embeddings are helpful to provide prior knowledge for OOV words. (3) In order to verify the usage of the relation-based attention mechanism, we no longer construct the relation-attentive sentence representation sk (Eq. 4), and replace the sk in Eq. 5 and 6 with relation embedding rk. That is to say, we try to directly use the relation embedding as the guidance of entity extraction. As shown in the results, the model's precision drops significantly. We consider that using relation embedding simply learns the shallow co-occurrence of the triplets, resulting in more triplet predictions but lower precision of the model. On the contrary, our attention mechanism can capture fine-grained semantic relation features in the sentence, which lead to a more significant distinction between positive and negative relations. (4) For the relational gate component, we omit the operations of Eq. 5 and 6. As an alternative to uk in Eq. 7, we explicitly use the sentence representation sk, ignoring the possible impact of negative relations. We found a decrease in the result, which indicates that our relational gated mechanism has contributed to reducing the noise brought by unrelated relations.

6.2 Parameter Analysis

In each training iteration, we randomly sample nneg negative relations for the sentence, which aims to balance the model's convergence speed and generalization performance. It is obvious that constructing more negative samples can improve the robustness of the model, but we don't have to rely on

higher value of nneg to achieve this. Actually, with the appropriate setting of random sampling count, almost all negative relations of the sentence will be covered as the increasing iterations. Thus, there should be an upper bound for hyperparameter nneg, leading to no longer improvement on the model's performance when it is larger than the bound value.

There are 24 relation types in the NYT dataset, with an average of 1.44 positive ones per sentence. Therefore, we try to select nneg among {1,2,4,6}, which is an appropriate range based on the number of positive relations in average. Figure 4 shows the curves of F1 score on validation set varying with training epochs under different values of nneg. It can be observed that when nneg = 4 or 6, there is almost no difference in convergence and prediction performance. Thus for NYT dataset, we consider nneg = 4 as the upper bound, which can ensure the effectiveness of the model and speed up training process at the same time.

6.3 Analysis on Overlapping Cases

To verify the capability of our RSAN in extracting multiple triplets, we follow [Zeng et al., 2018], and conduct further experiments on NYT dataset. The test sentences are divided into three categories based on different overlapping cases, i.e., Normal, SingleEntityOverlap (SEO), and EntityPairOverlap (EPO) (See the examples in Figure 1). We then verify several latest models' performance on each of the category. The results are shown in Figure 5. As we can see, RSAN outperforms all other methods in the overlapping situations, especially for the EPO class. We attribute the improvements to the fact that the entity pair overlapped triplets only have different relations, hence our separate prediction on each of the relation can effectively handle such cases. Another observation is that ETL-Span achieves the best performance in Normal class. It is because its decomposition strategy is designed more suitable for the Normal cases, while our RSAN performs much better in the overlapping classes.

Figure 5: F1 score of CopyRE, GraphRel, OrderRL and our RSAN

in different overlapping categories on the NYT test set.

We also compare the models' ability of extracting multiple triplets in a sentence. We divide the sentences of NYT test set into 5 categories, which respectively indicate its number of triplets is 1,2,3,4 and ≥ 5. The results are shown in Figure 6. It can be observed that our RSAN gains great improvements compared with other models in extracting multiple triplets. Besides, RSAN shows more stable performance with the increasing of triplets numbers in the sentence. These two additional experiments fully demonstrate the advantages of our proposed model in dealing with complex extracting situation.

7 Conclusion

In this paper, we propose a relation-attentive sequence labeling framework named RSAN for joint entity and relation extraction task. It decomposes the overlapping triplets extraction problem into several relation-specific entity tagging processes, and applies attention mechanism to incorporate finegrained relational information as the guidance of entity extraction. Experiments on the NYT and WebNLG corpus show that our proposed model RSAN has achieved significant improvement. The extended experiments demonstrate the effectiveness of RSAN in handling overlapping and multiple triplets extraction scenarios.

Acknowledgments

This work is supported by National Key R&D Program 2016 (No.2016YFB0801300), and the National Natural Science Foundation of China (No.61202226). We thank all anonymous reviewers for their constructive comments.