PasCore: A Chinese Overlapping Relation Extraction Model Based on Global Pointer Annotation Strategy

1 Introduction

Relation extraction aims to obtain semantic relations between entities from text, which plays an important role in constructing knowledge graphs [Hogan et al., 2020], question answering [Mohammed et al., 2018] and recommender systems [Wang et al., 2019]. Recent studies have made great progress in simple relation extraction [Miwa and Bansal, 2016; Lin et al., 2016], but these methods perform poorly on overlapping relation extraction because BIO (short for Beginning, Inside, and Outside) tags cannot mark overlapping entities simultaneously. However, entities are often shared between triples in real world, which can be summarized into four patterns as shown in Figure 1. SEP (SingleEntityPair)

∗Corresponding author

represents only one entity pair exists in a sentence. When multiple triples exist, following patterns could happen: NEO (NoEntityOverlap), indicating no entity overlaps; SEO (SingleEntityOverlap), indicating only one entity overlaps; EPO (EntityPairOverlap), indicating both head entity and tail entity have overlapped. Intersection represents the triples hold multiple overlapping types in one sentence.

The frequency of overlapping entities varies for different languages, and shared entities are especially common and complicated in Chinese. To verify this phenomenon, this paper compares two Chinese real-world datasets, DuRED and ICRED, with English datasets NYT [Riedel et al., 2010]. First, we find that Chinese datasets have a larger proportion of samples with overlapping entities, which conforms to a long-tailed distribution. Second, compared with English, the Chinese dataset is featured by (1) generally longer sentences, (2) more complex types of entities and relations, and (3) a larger number of entities and relations. For example, in a sentence: Wang Qing, chief analyst of Oriental Jincheng, it contains three overlapping entities, namely, Wang Qing, Oriental Jincheng and chief analyst of Oriental Jincheng. At the same time, it holds both SEO and EPO overlapping patterns, which demonstrates that Chinese is challenging for overlapping relation extraction.

Traditional relation extraction methods often use pipeline to first extract entities and then classify candidate entity pairs [Hoffmann et al., 2011]. However, these methods ignore the association between them, and easily cause cumulative errors [Li and Ji, 2014]. Moreover, joint relation model integrates the associations between entities and relations, which has achieved better performance in recent years [Tan et al., 2019]. However, overlapping entities cannot be labeled well under these methods via the BIO tagging strategy.

To extract overlapping triples, CopyRE utilizes sequenceto-sequence model to avoid tagging conflict [Zeng et al., 2018]. However, it only copies the last token of the entity and cannot resolve the situation where entity contains multiple tokens. GraphRel uses graph convolutional networks (GCNs) to model text as relational graphs [Fu et al., 2019]. ETLSpan applies a span-based marking strategy to model the semantic relevance [Yu et al., 2020]. Although these studies have made a good improvement, they all treat relations as discrete labels, making relation classification difficult: (1) The matching between entities is prone to a large number of negative cases; (2) When the same entity appears in multiple triples, the classifier is prone to confuse. We noticed that CasRel utilizes a two-stage cascading binary annotation framework to first label the head entity, and then label the relation and the tail entity simultaneously [Wei et al., 2020]. It alleviates the problem of too many entity pair combinations, but ignores the interaction between features. In other words, the entity type features are not considered well in the subsequent element prediction. At the same time, CasRel combines multiple relations with the same head and tail entities into a string, and essentially performs single relation extraction. Therefore, in fact, CasRel does not perform well on Chinese overlapping relation extraction, and fails to achieve the claimed performance.

To address these issues, we propose PasCore, a simple and effective relation extraction model which uses global pointer annotation strategy [Su et al., 2022] for Chinese overlapping relation extraction. It transforms the relation extraction into identifying relations, head entities and tail entities. First, the sentence vector obtained by pre-trained language model (PLM) encoders [Devlin et al., 2018; Sun et al., 2019] is used to predict the relation. Subsequently, PasCore utilizes the previous output as precondition to label the head entity and tail entity. The global pointer in entity annotation stages marks the start and end positions of the entities uniformly. The advantages of this strategy lie in: (1) It uses the idea of global normalization to label entities, and can effectively identify all kinds of entities with high accuracy; (2) Overlapping entities can be extracted via the pairs of different distances based on the global tags. However, BIO tagging strategy cannot deal with the shared entities because they would conflict with each other; (3) The order of relation prediction first and then entity annotation can make full use of features of relation and the entity type. Experiments show that PasCore outperforms state-of-the-art methods on Chinese real-world datasets and is competitive on English datasets, especially achieving excellent distinguished performance on Chinese overlapping relation extraction.

2 RelatedWork

Recent studies of relation extraction are mainly based on deep learning methods. Recurrent neural network (RNN) and convolutional neural network (CNN) were early utilized to solve the relation extraction problem [Zeng et al., 2014]. Subsequently, to enlarge the distinction between relations, new loss function and attention mechanism were applied [Wang et al., 2016]. Furthermore, Miwa and Bans adopted bidirectional LSTM and tree LSTM to model entities and sentences [Miwa and Bansal, 2016]. Lin et al. applied a sentence-level attention mechanism to assign weights to sentences [Lin et al., 2016] . To reduce the noise in datasets, Su et al. tried embedding textual relations with global relation statistics [Su et al., 2017]. Besides, reinforcement learning and token-level remote supervision were adopted on large-scale datasets [Liu et al., 2018]. Afterwards, Wang et al. designed a graph model to transform joint extraction into a directed graph [Wang et al., 2018]. Tan et al. proposed TME to adaptively discover triples via the ranking of the transfer mechanism [Tan et al., 2019] .

Traditional relation extraction uses pipeline methods to first extract entities and then classify candidate entity pairs [Hoffmann et al., 2011]. However, it completely separates the two subtasks, ignoring the interaction between them and causing errors to accumulate [Li and Ji, 2014]. On the other hand, joint extraction model has achieved good results. Bekoulis et al. designed a joint neural model to extract entities and relations at the same time [Bekoulis et al., 2018]. However, these methods cannot extract overlapping entities well due to the BIO tags conflict.

To deal with the problem of entity overlapping, CopyRE utilizes copy mechanism to avoid overlapping BIO tags [Zeng et al., 2018]. Besides, GraphRel uses a stacked BiLSTM sentence encoder and GCN dependency parsing encoder to extract hidden features of tokens, and decomposes entity pairs into token pairs for prediction [Zeng et al., 2019]. In addition, such method further applied reinforcement learning to improve the effect of CopyRE. Yu et al. proposed an endto-end sequence tagging framework ETLSpan, which applies a span-based tagging strategy to jointly extract entities and relations [Yu et al., 2020]. However, these methods all treat relations as discrete labels, which causes a large number of negative entity pairs and increases the burden of the relation classifier.

To further effectively extract overlapping triples, Wei et al. proposed CasRel [Wei et al., 2020], a two-stage cascading binary labeling framework, which greatly improves the performance of model on overlapping relation extraction. Besides, Wang et al. proposed a single-stage joint extraction model TPLinker and a novel handshake marking scheme [Wang et al., 2020]. Recently, Ren et al. used a global feature-oriented relational triple extraction model while they cannot get strong performance on more complex extraction [Ren et al., 2021] .

3 Methodology

3.1 Problem Statement

Overlapping relation extraction refers to extracting multiple triples from text where the same entities are shared in one sentence. Given a sentence S, an entity set E = {e1, e2, . . . , eM} and a predefined relation set R = {r1, r2, . . . , rN}, all overlapping triples T =

{(e1, r1, e2), . . . , (ei, rk, ej)} need extracting, where M and N represent the number of entities in sentence and all relations, respectively. ei, ej ∈ E, rk ∈ R. This paper mainly focuses on Chinese overlapping relation extraction, namely, given a sentence S in Chinese, a model can extract all triples contained with overlapping entities simultaneously.

3.2 Framework

The framework of PasCore is depicted in Figure 2. PasCore mainly consists of three stages, namely, relation prediction stage, head entity annotation stage and tail entity annotation stage.

First, for the input sentence, PasCore uses PLMs to encode and obtains the sentence vector and the word vectors. Then, the relation classifier uses such encoders for relation prediction. After predicting the relations that exist in the sentence, PasCore uses the word vectors with relations to mark head entities via global pointer annotation strategy. Subsequently, PasCore fuses the head entity vector with word vectors via conditional layer normalization. Finally, the integrated feature will be cascaded with the entity type feature and the relation feature, to assist in tagging tail entities.

3.3 Encoder

In our work, two pre-training models BERT [Devlin et al., 2018] and ERNIE [Sun et al., 2019] are used to evaluate the effectiveness of the proposed framework.

BERT is a multi-layer bidirectional Transformer based language representation model. It is designed to learn deep representations by jointly conditioning on both left and right context of each word , and it has recently been proven surprisingly effective in many downstream tasks [Zhong et al., 2019].

ERNIE is a pre-trained semantic representation model based on transformer. Compared with BERT, it can capture more semantic information on the Chinese corpus, because it introduces multi-source Chinese knowledge and models the combined semantics of words, improving the versatility and scalability. Let X = [x1, x2, . . . , xn] represent the input, where xi represents the vector representation of each token, composed of token embedding, sentence embedding, position embedding and task embedding. ERNIE contains F transformer encoders, denoted as Trans(x) as follows.

Suppose that the sentence contains n tokens, the vector representation of n + 1 tokens including [CLS] will eventually be obtained. [CLS] represents the embedding of the entire sentence embS = hF [0], while the embedding of the i-th token is represented as vh i = hF [i].

3.4 Relation Prediction

The relation prediction layer extracts possible relations in the sentence according to the sentence vector obtained by the PLM encoder. For the sentence vector embS, the score of the relation or is calculated by Equation 2.

where W(·) represents the trainable weight, and b(·) represents the bias. Let ori represent the score of the i-th relation. In order to obtain the probability of each relation, PasCore performs sigmod function on all relations.

PasCore sets a threshold δr, and only keeps the relation whose probability exceeds the threshold.

3.5 Head Entity Annotation

PasCore concatenates vh with the relation vector vr to obtain the input for head entity annotation stage, which is denoted as Xe1 = [[vh 1 , vr ], [vh 2 , vr ], . . . , [vh n , vr ]]. Then, it uses one classifier to identify the start and end positions of the head entity, which has a global view.

where pes1 ,ee1 i represents the probability that the i-th token is the spatial location of a head entity. Besides, PasCore sets a threshold δe1 . When the probability exceeds it, we set the tag to 1. σ is the sigmoid function. During training, the likelihood function is as follows.

where n represents the length of sentence. If z is true, I(z) = 1, otherwise, I(z) = 0 . When the i-th token is the global position of the head entity, yes,e 1 i = 1.

3.6 Feature Fusion

For the head entity, we obtain its corresponding embedding via the encoder. Specifically, let vh es,e 1 represent the embeddings at the spatial position of the head entity, respectively. We use reduced dimension vh es,e 1 to represent the head entity vector ve1 .

Subsequently, PasCore uses conditional layer normalization to fuse ve1 with vh to obtain vf . Let the input of the l-th layer be zl , then the mean µl and variance σ2 l are calculated as follows.

where nl is the number of neurons. Then, output ˜ zl is obtained via conditional layer normalization.

where fλ and fβ are two conversion functions using full connection, representing zoom factors and translation factors, respectively. ϵ is a small constant used to smooth.

3.7 Tail Entity Annotation

Consequently, the head entity type vector veid , vr and vf are concatenated to form Xe2 = [[vf 1 , vr , veid ], [vf 2 , vr , veid ], . . . , [vfn , vr , veid ]] as the input. Similarly, PasCore identifies the start and end positions of the tail entity as follows.

where pes,e 2 i represents the probability that the i-th token is the spatial location of a tail entity. At the same time, threshold δe2

is set, and only when the probability is greater than it, sets the binary tag to 1. During training, the likelihood function is as follows.

where n is the length of the sentence. When the i-th token is the global position of the tail entity, yes,e 2 i = 1.

3.8 Loss Function

PasCore processes the training data into a set of multiple sentences with their triples. Let D denote the number of sentences, and Ti denote the triples of the i-th sentence. The loss function L is expressed as Equation 11.

4 Experiments

To evaluate the proposed method, we conduct experiments on two Chinese datasets 1 and one general English dataset. Then we analyze the results comprehensively as well.

4.1 Datasets and Settings

Due to lack of Chinese overlapping relation extraction datasets, this paper constructs and releases two new datasets DuRED and ICRED for evaluation. DuRED and ICRED are based on the open datasets of relation extraction tasks in Baidu 2020 Language and Intelligent Technology Competition and Intelligent Computing Platform Competition 2 . Based on these raw real-world datasets, we first remove inapplicable multiple relations and wrong triples via manually fine-grained inspection. Secondly, we mine entity types through datasets and introduce external prior knowledge to complete the triples. Finally, we perform data augmentation and obtain the final datasets. In addition, there are many overlapping triples in other languages (such as English), and we choose the popular English dataset NYT24 for evaluating the generality of PasCore.

1 https://github.com/seukgcode/pasCore

2 http://lic2020.cipsc.org.cn

The details of datasets are shown in Table 1. The peak of text length distribution in DuRED lies between 50 and 100, and the maximum length does not exceed 300. Such peak in ICRED is around 100, and the maximum length is less than 500. The long-tail interval in ICRED is longer, which makes relation extraction more difficult. In both two datasets, the distribution of entity types and relations is extremely unbalanced. Furthermore, Table 2 reports the details of the division, overlap relation, and triple distribution of the two Chinese datasets and English dataset NYT24.

PasCore is implemented by PaddlePaddle 3 , and uses AdamW [Loshchilov and Hutter, 2018] to optimize weights. In experiments, we use BERT-base for all datasets. Two versions of ERNIE are utilized, i.e., ERNIE-base which are experted in Chinese, another namely ERNIE-en in English.

4.2 Baselines

For comparison, we reproduce the following baselines: NovelTagging [Zheng et al., 2017], which uses annotation strategy to combine entities and relations; CopyRE [Zeng et al., 2018], an end-to-end model which uses copy mechanism to jointly extract relations; GraphRel [Fu et al., 2019], a joint relation extraction model based on GCN network; ETLSpan [Yu et al., 2020], which hierarchically decodes triples to model the semantic relevance; CasRel [Wei et al., 2020], a two-stage cascading binary labeling framework which extracts head entities, tail entities and relations step by step; GRTE [Ren et al., 2021], a global feature-oriented relational triple extraction model based on table filling.

3 https://www.paddlepaddle.org.cn

4.3 Main Experimental Results

Table 3 reports the main experimental results. First, on DuRED dataset, PasCore outperforms all baselines on precision (P), recall (R) and F1 scores. When ERNIE-base is used as the encoder, PasCore achieves the best result, which is 0.9% higher than the F1 score of CasRel. Second, on ICRED dataset, the best performance is still obtained by PasCore. When using the BERT-base as encoder, PasCore also surpasses all baselines and is 5% higher than the F1 score of GRTE. When using the ERNIE-base as the encoder, the F1 score is 0.6% higher than that of BERT-base, reflecting the excellent and stable relation extraction performance of PasCore. Third, on English dataset NYT24, the performance of PasCore is still very competitive and is closed to GRTE. Note that, GRTE performs better on English dataset, probably because it employs a global feature-oriented table filling method, which is more suitable for English. However, for PasCore, its performance gap on different PLMs is very small, which reflects the fact that PasCore is less sensitive to PLMs and more versatile or generic. Moreover, CasRel performs better on DuRED and relatively poorly on ICRED. The reason is that CasRel combines multiple relations with the same head and tail entities into a string, and treats it as a single relation extraction, resulting in a poor performance on ICRED, which contains a lot of overlapping entities.

For different overlapping patterns, the left part of Table 4 reports F1 scores of all models on four overlapping types. It can be seen that, on DuRED and ICRED, PasCore achieves the best performances on all overlapping types. For ICRED, most models have the highest F1 scores on SEP, while for DuRED, the type is SEO. The reason is that a model should not only fit simple cases, but also needs to coordinate all overlapping types to maintain consistency. Besides, the distribution of each type could affect the performance. For English dataset NYT24, the F1 scores of SEO and EPO appear the highest alternately. Meanwhile, PasCore is very close to GRTE in English overlapping relation extraction, which demonstrates the effectiveness of PasCore in English.

In terms of different numbers of triples, the right part of Table 4 reports the F1 scores of PasCore and baselines according to N (the number of triples) in datasets. We can observe that PasCore (ERNIE-base) achieves the best performances in almost all cases, but is slightly less effective than GRTE (on DuRED and N = 4) and CasRel (on ICRED and N = 3). Especially, when N ≥ 5, PasCore (ERNIE-base) has more significant performance advantages over other baselines. In addition, we can see that in DuRED, the performances of PasCore fluctuate less that of ICRED. This is mainly because the ICRED is only one-tenth of the DuRED, which makes the model cannot fully fit the distribution of triples, and fluctuates larger. On the English dataset NYT24, the F1 score of PasCore on each number of triples is only lower than GRTE, the gap between PasCore and GRTE is marginal.

4.4 Ablation Experimental Results

We conduct ablation experiments with PasCore (ERNIEbase), and the results on ICRED dataset are shown in Table 5. -RelType represents removing vr in feature vector. -HEntText represents removing ve1 in feature vector. -HEntType denotes removing veid in feature vector.

It can be seen that after removing the relation vector, F1 scores drop dramatically. It means that the relation plays a great role in entity annotation. Another observation is that both the head entity and the head entity type can improve the performance of PasCore. After removing these two, F1 scores both drop. This is because PasCore uses the normalization method to integrate them into the feature vector, which further promotes the performance. After removing all the features mentioned above, F1 scores drop significantly, fully verifying the importance of these features.

4.5 Error Accumulation Analysis

Considering that PasCore is a pipeline method, we carry out experiments on error accumulation between modules. In Table 6, experiments 1. relation and 3. relation+head entity represent gold and predicted experimental results. Experiments 2. relation→entity and 4. relation+head entity→tail entity use arrows to show that the input is gold or the results predicted in the previous module and the experimental results in the next module.

Compared with experiment 1 and experiment 2, it can be seen that when the relation predicted by the model is used as input, the F1 score of the predicted head and tail entities is not significantly reduced. Similarly, in experiments 3 and 4, the use of the gold and predicted relation with the prediction of the head entity to the tail entity does not cause significant performance drops. This indicates that the error of the upstream module of PasCore will not cause disastrous influence to the subsequent modules.

4.6 Case Study

To further analyze the advantages of PasCore, we also study the cases in Figure 3, which contains three overlapping patterns: NEO, SEO and EPO.

It can be seen that the results of PasCore are consistent with the answer, while for CasRel, the triples related to Wang Qing and Zhang Xiaolong are confused. This is mainly because CasRel first extracts the head entity, and then extracts the relations and tail entities at the same time, making the matching prone to errors. Specifically, after identifying Wang Qing, the relation Work in and the entity Tencent are extracted simultaneously, resulting in a false match.

In contrast, PasCore firstly performs relation prediction. After Work in is predicted in success, Wang Qing and Zhang Xiaolong are marked via global pointer annotation strategy. Take the former as an example, PasCore then fuses the features of Work in, Wang Qing and its type, and finally marks the entityOriental Jincheng successfully. This is because PasCore extracts triples in strict accordance with the order of relation prediction and entity annotation, and the features obtained from the previous stage are also retained to the latter stages, which enriches the association between relations and entities and ensures the correct matching of the entity pair.

5 Conclusion and FutureWork

To extract overlapping relations especially in Chinese, this paper proposes PasCore, a relation extraction model based on global pointer annotation strategy. It transforms the relation extraction into relation predication, head entity annotation and tail entity annotation three stages in order. Besides, in order to enrich the feature representation, the output of each stage will be passed to the next stage as precondition. Experiments show that PasCore achieves excellent performances on both Chinese real-world datasets and English dataset.

In the future, we plan to explore a more effective method to integrate relations into entity annotation, and integrate the head entity into tail entity annotation. At the same time, we also plan to generalize PasCore to more languages.