Interactive Information Extraction by Semantic Information Graph

1 Introduction

Information extraction (IE) aims to extract structured information from unstructured text. According to the definition from the ACE2005 program [Walker and Consortium, 2005], there are three IE subtasks, i.e., entity extraction, relation extraction, and event extraction. In general, the three subtasks are regarded as an information network GI (V I ,EI ) construction [Li et al., 2014]. Figure 1 shows the reentrancy and isolation features of the information network. Reentrancy means that a node may play a multi-role in an information network. As the name suggests, isolation denotes that entities or events exist in isolation. The two characteristics indicate that these three subtasks of IE are independent of each other and have potential connections.

∗ Indicates equal contribution

†Corresponding Author

Existing studies rarely take into consideration the reentrancy feature. Early works solve the three subtasks separately [Li et al., 2014], which leads to the error propagation problem and disallows interactions among subtasks during decoding. Then, deep learning-based methods [Wadden et al., 2019; Luan et al., 2019] are used to extract information separately. Further, an end-to-end IE framework is proposed to incorporate global features during decoding to capture interactions between IE tasks [Lin et al., 2020]. Recently, there are studies using Graph Neural Network (GNN) to do the subtasks and achieving promising performance. The dependency graph consumed by GNN is proposed to build an information network by restricting subtasks jointly [Nguyen et al., 2021]. Despite the success of the above approaches, they are hard to utilize the correlation information of IE subtasks. We know that relation extraction is based on subject and object entities. Therefore, the recognition of subject and object entities is essential for relation extraction.

For argument extraction, an argument can be an entity, time expression, or value (e.g., MONEY) [Lin et al., 2020], and their extraction is the difficult point. Recently, Abstract Meaning of Representation (AMR) [Banarescu et al., 2013] is used to guide the encoding and decoding process [Zhang and Ji, 2021]. AMRIE is proposed to manually assign the relation clusters of AMR to the label of IE tasks before putting them into graph attention networks [Zhang and Ji, 2021]. However, AMR is not designed for information extraction, so it cannot be used directly and effectively. Furthermore, although the AMR graph helps enrich the representation vectors of information network nodes, it also introduces redundant semantic information interference during the encoding and decoding stage.

To overcome those issues, we propose an Interactive Information Extraction (InterIE) model based on a novel Semantic Information Graph (SIG). SIG is able to guide the model to tackle the three subtasks jointly. Meanwhile, SIG is capable of reducing the noise information of AMR. Compared with AMR, the edge of SIG is the label of IE tasks, including relation and argument without transformation. Given an input text, we use the proposed InterIE to detect nodes using local classifiers [Lin et al., 2020] firstly. Then, we map the candidate nodes to SIG and feed them to Graph Attention Networks (GATs) [Velickovic et al., 2018], which enhances node representation for IE. Lastly, we use the SIG structure to determine the beam order and search for the optimal graph with global features during decoding. Experiments conducted on ACE05 dataset show that our InterIE model reaches the new state-of-the-art on all IE subtasks.

The major contributions of this paper are summarized as follows:

• We propose an Interactive Information Extraction (InterIE) with a novel Semantic Information Graph (SIG) to guide the IE subtasks jointly.
• Compared with AMR, SIG does not contain noise. Benefiting from this, InterIE could enrich entity and event trigger representations, and capture the edge connection between the information types through neighbors.
• We conduct experiments on benchmark datasets compared with strong baselines. Experiment results show that InterIE is competitive. More importantly, it is not sensitive to decoding order.

2 Approach

Figure 2 illustrates the overview of InterIE. Given an input text (i.e., a sentence S), InterIE extracts the information network in three stages: i) SIG generation, generating the SIG for S based on sequence to graph transformer; ii) Graph aggregation, aggregating the candidate nodes and SIG through Graph Attention Networks (GATs); iii) Decoding processing, determining the decoding order and search the optimal graph with the global feature.

2.1 ProblemFormulation

We formulate the subtasks of the joint information extraction problem as follows.

Entity Extraction is used to identify the span of the entity and assign pre-defined entity types to them. For example shown in Figure 1, "French" is recognized as a GPE entity.

Relation Extraction is used to assign a relation type to an ordered pair of entities. For example, the relation between "US" and "firm" is assigned as GEN-AFF.

Event Extraction is used to recognize the event and corresponding arguments. In general, it is divided into event identification and argument extraction. Event identification is the detection of event triggers (the words or phrases expressing the occurrences of the event clearly). As the name says, argument extraction denotes the extraction of arguments involved in the event (the roles participated in the event). For example, the word "sell" triggers a Transaction event, and the word "firm" denotes the Buyer argument while the phrase "Vivendi Universal" denotes the Seller argument.

These three IE subtasks can be formulated as an information network. Given an input sentence, our goal is to construct an information network Ginfo (V info ,Einfo ), where each node in V info represents an entity or event trigger. Each edge in Einfo represents a relation or argument role in events.

2.2 SIGGeneration

As illustrated in Figure 1, the SIG G(V,E) is a rooted, directed and labeled graph. The node of SIG represents an entity or an event trigger. The edge of SIG denotes label (i.e., entity type, relation type and argument). Different from the information network Ginfo (V info , Einfo ), SIG equips with the virtual rooted structure which brings discrete IE subtasks together. We adopt the sequence to graph transformer to generate SIG graph incrementally. Next, we detail this generation process.

Graph Encoder. Given an input sentence S = (w1, . . . ,wn) with a special [START] token w0 at the beginning, we feed them into a transformer encoder to generate the corresponding embeddings (hw0 , hw1 , . . . , hwn ). Here, hw0 is the feature vector representing the whole sentence. Since the graph is sequentially generated, there is one new generated node vt at step t. To trigger the generation process, we set a dummy node, marked as v0. We feed these nodes (v0, v1, . . . , vt-1) into the transformer encoder with a masked attention mechanism to generate the updated graph node states (hv0,t , hv1,t , . . . , hvt-1,t ). Here, hv0,t is the representation of the graph at step t, and it is initialized as hw0 . We put it into decoder to predict the source of graph node vt and its relationships with previous nodes.

Node Solver. Given the graph state hv0,t at step t and input sentence tokens S = {w1, ..., wn}, the node solver generates the node of SIG by deciding its position in the sentence or a pre-defined vocabulary (the set of all entities and triggers in training data). Following [Cai and Lam, 2020] , we calculate the attention distribution over all input tokens:

where {W Q,W K } ∈ Rd×d are the weight matrices that project the graph state token and sentence tokens into key and query subspaces, respectively. For brevity, we simply mark the concatenation of (hw1 , . . . , hwn ) as hw1:n ∈ Rn×d . The attention weight αt ∈ Rn indicates the position of the node vt coming from the input sentence S.

In addition, the node can also be generated from a predefined vocabulary. To calculate its probability distribution on the vocabulary, we firstly attend sentence tokens with αt as the representation of the node α′t ∈ Rd . Then, we feed it into a fully-connected layer followed by a softmax classifier to produce the probability distribution Pvocab,t:

where W V ∈ Rd×d denotes the parameter matrix that projects sentence tokens into the value subspace, and W vocab denotes the weight matrix that transforms the dimension from d to the size of the vocabulary.

To control the contributions from the vocabulary and the input sentence, we utilize a soft switch layer to compute their weights:

where W switch is the weight matrix. We mark the contributions of vocabulary and input sentence as p0,t and p1,t, respectively. Therefore, the final distribution of the next node P v t can be written as:

Algorithm 1 SIG generation

Input: Sentence S = w1, ..., wn

Output: SIGG(V,E)

• 1: hw0 , hw1 , . . . , hwn = Transformer(S) // Initialized SIG
• 2: G0 = (V = {dummy}, E = {∅}) // Begin interation graph spanning
• 3: for t=0 to MaxInteration do
• 4: hv0,t , . . . , hvt-1,t = Transformer(Gt) // ot represents output layer of step t
• 5: onode t = Node Solver(hw0:n ,hv0,t:t-1,t )
• 6: oedge t = Edge Solver(hw0:n ,hv0,t:t-1,t )
• 7: if onode t is end then
• 8: break
• 9: end if
• 10: update Gt to Gt+1
• 11: end for
• 12: return Gt

where ⊙ denotes element-wise product, j is the index of tokens and α[j] is the j-th element of αt. S(vt) is the index set of tokens corresponding to the node of graph.

Edge Solver. After obtaining the newly generated node vt, the edge solver is used to decide the node's connection with previous nodes. To this end, the edge solver is designed to classify the edge types of nodes. We conduct multi-head attention between vt and previous node states to calculate the attention weight over all existing nodes:

where m represents the index of the head. Then, we take the maximum of attention weights over multiple heads as the final edge probability βt. Edges whose probability value exceeds the threshold are kept [He and Choi, 2021]. Then we employ the deep biaffine classifier [Dozat and Manning, 2017] to assign their labels. The current graph node states hv1,t:t-1,t are used to represent the dependencies in a biaffine decoder, which will generate the edge label score matrix oedge label t ∈ R(t-1)×k . Here k is the number of edge labels in the training data. The score matrix is used to predict whether the target node is the head of other nodes or not. Algorithm 1 details the generation processing of SIG.

2.3 Graph Aggregator

In this subsection, we describe our processing of detecting entities and triggers. To fully exploit the information of external knowledge between them, we propose a SIG aggregator to enrich the candidate nodes of entities and triggers to aggregate information from their neighbors based on the message passing mechanism. Previous study [Zhang and Ji, 2021] uses AMR graph to aggregate candidate nodes. However, the noise in AMR lowers the extraction ability. Our SIG aggregator overcomes this by rebuilding the graph with the corresponding information. For example, the node in SIG will be replaced by the candidate node when meeting the same span.

Entity and Event Trigger Detection. This component aims to detect spans of entities and event triggers that could be used as candidate nodes. First, we feed sentence S into RoBERTa [Liu et al., 2019] to obtain the contextual representation X = [x1, x2, .., xn]. Afterwards, X is fed into conditional random field layers to determine the tags sequences of the entities and event triggers [Chiu and Nichols, 2016]. Let Ld be the log-likelihood of gold tag sequences for entities and triggers span detection. We obtain the entity spans {xstart, xend} and trigger spans {ystart, yend} by maximizing the log-likelihood Ld .

Graph Enriched. Once we have the SIG G(V,E) and the candidate nodes set V c , we map those candidate nodes to the SIG through corresponding spans in the original sentence. For each candidate node vc i in V c , there are two situations in the map processing: (1) if vc i is already in G(V,E) with the same span, we take the representation vector of the vc i as the initialized feature vector; (2) if vc i does not match any node span in the G, we add vc i into G as well as all related edges. Finally, we get the enriched graph G+ = (V +, E+).

GATs for Message Passing. Similar to [Zhang and Ji, 2021], we use GATs to propagate messages in the enriched graph G+. Given the node v+ i ∈ V + and its connected neighbor set Ni, we compute the attention score αli,j in l-th layer for each v+ j ∈ Ni with the corresponding edge embedding ei,j by

where hli is the l-th layer embedding of node v+ i . W ln andW l e are trainable weight matrices for node and edge features. f l denotes a fully connected layer. σ is an activation function, and we use LeakyReLU here.

We compute the message for node i by summing its neighbor features with weights:

Finally, the updated node representation is calculated by

where γ controls the level of message passing between neighbors. W lm is the parameter. We select the last layer hl as the final representation for each entity/trigger.

2.4 Training and Decoding

In this subsection, we introduce how we jointly decode the output information network with the identified entity, trigger nodes and their aggregated features hli . We decide decoding order in two manners: a SIG-based top-down manner and a flat left to right manner.

Optimal Graph with Global Constraint. The total score S(Ginfo ) for the local information network is calculated by

where hli is the final representation of each entity or trigger node from the last layer of GATs. FFN is a feed-forward network. T is the set of IE subtasks, and N t is the number of instances for task t. We further utilize global features fg in [Lin et al., 2020] and combine it with local score S(Ginfo ) to compute the global score:

where fg is a set of global feature templates to capture the cross-subtask and cross-instance interactions defined in [Lin et al., 2020], and u is the weight vector of fg. Finally, we maximize the following joint objective function

where ˆGinfo is the ground-truth network and Ginfo is the predicted information network.

DecodingOrder Detection. Given all nodes and their pairwise edges, one way to output the information network Ginfo is to calculate the global score S′ (Ginfo ) for each candidate graph. Then, we select the one with the highest global score in the beam search-based way [Lin et al., 2020]. Besides, we also incorporate the SIG hierarchically to decide the topdown decoding manner [Zhang and Ji, 2021].

3 Experiments

3.1 Dataset

We conduct experiments on the Automatic Content Extraction (ACE) 2005 dataset, which provides the entity, relation and event annotations. Following the previous works of [Wadden et al., 2019; Lin et al., 2020], we use two different versions of ACE05 for the Joint IE task, i.e., ACE05-E and ACE05-E+. ACE05-E is the widely used version of ACE05 dataset for event extraction. We follow the preprocessing method in [Wadden et al., 2019]. ACE05-E+ is proposed in [Lin et al., 2020] by adding some important elements ignored by previous studies, which improves the quality and the data scale. Following these works, we keep 7 entity types, 6 relation types, 33 event types, and 22 event argument roles for ACE05-E and ACE05-E+. Table 1 shows the detailed statistics of ACE05-E and ACE05-E+.

3.2 Experimental Setup

We use the same evaluation metrics (i.e., F1) with the previous works. The criteria for each subtask are as follows.

Entity. Both the offsets and type of entities are correct.

Relation. The entities of subject and object and the relation type are correct.

Event Trigger. A trigger (Trg-I) is identified correctly if its offsets are correct. A trigger (Trg-C) is classified correctly if its event type is also correct.

Argument. An argument (Arg-I) is correctly identified if its event type and offsets are correct. An argument (Arg-C) is correctly classified if its role type is also correct.

We train our model with Adam [Kingma and Ba, 2015] on NVIDIA 3090 with a learning rate 2e-5 for RoBERTa parameters and 4e-4 for others. The number of epoch is 100. To get a fair comparison, we use the same settings of other parameters with [Lin et al., 2020; Zhang and Ji, 2021]. Our code is released to support research 1 .

3.3 Overall Performance

The two datasets have been used widely in IE. This allows us directly compare the results of our InterIE against baselines in the same experimental setting.

1 https://github.com/LucyFann/InterIE

BaselineMethods. We adopt the most recent joint IEmodels as our baselines. DyGIE++ [Wadden et al., 2019] is a joint IE framework using local-specific classifier with span representations. OneIE [Lin et al., 2020] is a graph-based joint IE framework decoding with global features. AMRIE [Zhang and Ji, 2021] uses AMR graph to guide encoding and decoding based on OneIE. FourIE [Nguyen et al., 2021] applies GNN to perform type predictions and regularizations.

Table 2 lists the F1 of baseline models on the two datasets reported in their papers. Our InterIE is listed in the last row of each dataset. We have the following observations from the experiment results:

• InterIE achieves improvements almost on all IE subtasks, including entity, relation and event extractions on both ACE05-E and ACE05-E+ datasets. The performance greatly outperforms on edge classification tasks such as relation extraction (↑ 2.9% and ↑ 1.6%) and event argument role labeling (↑ 1.6% and ↑ 0.8%). This indicates InterIE can better recognize relations between entities and event triggers with the incorporation of SIG.
• InterIE performs better on ACE05-E+ than ACE05-E. This is because ACE05-E+ adds back the ignored information, including the order of relation arguments, pronouns, and multi-token event triggers. It tells us that the InterIE model is good at handling multi-event and multirelationship on sentence level, which demonstrates the rooted reentrance structure of SIG captures the interdependency of different IE subtasks.
• To further show the influence of decoder order, two decoding orders are compared. In InterIE w/o Dec, we keep the flat left-to-right decoding order, while InterIE uses hierarchical order based on SIG. From the results, we can see that InterIE is not sensitive to the decoder order. It indicates that our model breakthroughs the limitations of AMRIE.

3.4 Analysis on Hyperparameters

We conduct hyperparameters analysis including γ defined in 8 and batch size. Figure 3(a) shows the effect of γ, which ranges from 10-4 to 1. The result shows that when γ continually increases higher than 10-2 , the performance of the subtasks will decrease rapidly. This draws a similar conclusion to [Zhang and Ji, 2021]. It makes sense because if the nodes focus too much on their neighborhood information, they will lose some of their inherent semantic features, which causes a significant performance decrease. Figure 3(b) shows the performance of our InterIE model improves significantly with a larger beam size. When the batch size reaches 8, the model performance reaches a smooth level. In addition, we find that mapping all IE edge types from SIG to GATs for the training process may lead to overfitting, so we only selected edge types that represent relation extraction and events.

3.5 Case Study

To give an intuitive understanding, we select two sentences from the testing datasets to illustrate how the SIG jointly helps the relation and event extractions. Figure 4 shows the complementation mechanism. From the first sentence in Figure 4, we find that the SIG and recognized entities (i.e., French) and triggers could be complemented to improve relation extraction. On the other hand, SIG keeps the relation of unrecognized entities and triggers, e.g., the "ORG-AFF" between official and government. In summary, SIG could enhance the recognized elements and keep the unrecognized elements to improve the extraction ability. From the second example in Figure 4, we observe that SIG detects the trigger (i.e., summit), which is not recognized by the entity and trigger detection. Meanwhile, entity and trigger detection recognize the "nations", which is the argument role of the event. From the two above examples, we find that the well-designed SIG and entity/trigger detection is complementary to achieve a synergistic effect.

4 RelatedWork

Early works solve the joint information extraction subtasks with feature engineering such as Lexical and WordNet features [Ahn, 2006], global constraints [Roth and Yih, 2004;

Li et al., 2014], Markov Logic Networks [Venugopal et al., 2014; Poon and Domingos, 2007]. However, these studies still fail to consider all of the three main IE subtasks together in a joint way. Since the IE subtasks can be considered as the problem of building an Information network Ginfo (V,E) [Li et al., 2014], graph-based methods have been proposed to do subtasks jointly. With the development of deep learning, [Wadden et al., 2019] designed a neural model using contextualized embeddings in local-specific classifiers for different IE subtasks. [Lin et al., 2020] exploits BERT and global features to constrain the decoding step. Meanwhile, some works focus on entity and relation extraction [Zheng et al., 2017; Zhong and Chen, 2020; Fu et al., 2019] and event extraction [Zhang et al., 2018; Liu et al., 2018; Zhang et al., 2021], respectively. These works can not scale to extracting a information network of all IE subtasks. With the help of graph neural networks(GNN), some studies take further steps to capture the interaction among IE subtasks. [Nguyen et al., 2021] applies GNN to perform type predictions and regularizations. [Zhang and Ji, 2021] introduce AMR graph to guide the encoding and decoding stages for IE subtasks.

5 Conclusion

In this study, we propose InterIE model based on SIG for jointly extracting information (i.e., entity, relation, and event). The key idea of InterIE is to design a SIG aggregator to capture the inter-dependency between different IE subtasks. The major difference with AMRIE is that AMR graph introduces redundant semantic noise, which leads to redundant interference during the encode and decode stages. To this end, we design the rebuilding mechanism for SIG to avoid the noise inherent in AMR. Experiment results show that the proposed InterIE achieves state-of-the-art performance on all subtasks of IE. More importantly, InterIE is not sensitive to decoding order, which shows the robustness of our InterIE model.

Acknowledgments

This work is supported by the National Key R&D Program of China (2020AAA0105200) and the National Science Foundation of China (NSFC No. 62106249, U2001212, 62032001, and 61932004).