Efficient Document-level Event Extraction via Pseudo-Trigger-aware Pruned Complete Graph

1 Introduction

Event Extraction (EE) aims at filling event tables with given texts. Different from sentence-level EE (SEE) which focuses on building trigger-centered trees [Chen et al., 2015; Nguyen et al., 2016; Liu et al., 2018; Wadden et al., 2019; Lin et al., 2020], document-level EE (DEE) is to decode argument combinations from abundant entities across multiple sentences and fill these combinations into event record tables as shown in Figure 1, where annotated triggers are often not available [Yang et al., 2018; Zheng et al., 2019; Xu et al., 2021].

One of the challenges for DEE is event argument combination without triggers. Triggers in DEE datasets are either absent or in a low annotation quality since the large-scale datasets are usually generated via distantly supervised (DS) alignment with existing knowledge bases (KB) [Mintz et al.,

∗Corresponding author

2009; Chen et al., 2017]. Consequently, the absence of triggers drives the development of trigger-free argument combination methods in recent studies. Yang et al. [2018] first identify a key sentence and then fill the event record table by finding arguments near the sentence, while partial global features and arguments are still missing. Zheng et al. [2019] and Xu et al. [2021] fully utilize the global context and make significant improvements by building an directed acyclic graph (DAG). However, such DAG-based methods require massive computing resources as they rely on an autoregressive fashion to decode argument combinations, which is inefficient in both training and inference for long documents. Meanwhile, building DAGs consumes lots of memories to store previous paths. As a result, it takes almost one week with a minimum of four 32GB GPUs to train such DAG-based DEE models, and the inference speed is also extremely slow.

Considering the above challenge in DEE and the speed & memory consuming problems in the DAG-based methods, in this paper, we aim to 1) propose a universal event argument combination strategy that works on both triggeraware and trigger-free DEE; and 2) provide a blazing fast and lightweight model for document-level event extraction.

We propose a novel non-autoregressive approach named as Pseudo-Trigger-aware Pruned Complete Graph (PTPCG). Specifically, we formulate each argument combination as a pruned complete graph, where the important arguments are identified and treated as a group of pseudo triggers with bidirectional connections to each other and other ordinary arguments are linked from these pseudo triggers in a directed manner. Based on the pruned complete graph with the pseudo triggers, we design an efficient algorithm with the non-autoregressive decoding strategy for event argument combination extraction.

The experiments results show that our PTPCG can reach competitive results with only 19.8% parameters of DAGbased SOTA models, just taking 3.8% GPU hours to train and up to 8.5 times faster in terms of inference. Besides, our PTPCG is highly flexible and scalable that can be used as a general architecture for non-autoregressive trigger-based event extraction. If only one pseudo trigger is selected for each combination, the pruned complete graph becomes a triggercentered tree like SEE. Furthermore, pseudo triggers can be adopted as supplements to enhance the annotated-triggerbased methods.

In summary, our contributions include:

• We propose a novel non-autoregressive event argument combination paradigm based on pruned complete graph with pseudo triggers, which is compatible in documentlevel event extraction with (or without) triggers.
• Our model is fast and lightweight for end-to-end document-level event extraction and we conduct extensive experiments to show the efficiency and efficacy.
• To our best knowledge, our present approach is the first work that explores the effects of using some arguments as pseudo triggers in DEE, and we design a metric to help select a group of pseudo triggers automatically. Furthermore, such metric can also be used for measuring the quality of annotated triggers in DEE.

2 Methodology

As shown in Figure 2, our model can be divided into four components: 1) Event detection performs multi-label classification to identify all possible event types. 2) Entity extraction extracts all entities from the documents and encodes these entities into dense vectors. 3) Combination extraction builds pruned complete graphs and decodes argument combinations from such graphs. 4) Event record generation combines the results of event types and extracted argument combinations to generate the final event records.

2.1 Event Detection

For a document D, a bidirectional long short-term memory (BiLSTM) network [Hochreiter and Schmidhuber, 1997] is used to encode each sentence si into token-wise hidden states (h(1) i , h(2) i , . . . , h(|si|) i ), where h(j) i ∈ Rdh is the concatena---→

tion of two direction representations h(j) i ‖←-- h(j) i and |si| is the length of i-th sentence. The last hidden states in each direction are concatenated to get the sentence representation gi ∈ G as gi = ---→ h(|si |) i ‖←-- h(0) i .

Subsequently, we follow Doc2EDAG [Zheng et al., 2019] and use randomly initialized event queries with multi-head attention on G to make a binary classification for each event type. The loss function of event detection component Ldet is defined as a binary cross entropy loss.

2.2 Entity Extraction

The entity extraction task can be formulated as a sequence tagging task in BIO scheme [Zheng et al., 2019]. To enhance the entity recognition, we add more money, date, percentage ratio and shares entities into the dataset by simple regular expression matching. To deal with the document-level entity extraction, we first split the whole document into sentences and perform a sentence-level mention extraction. Then, we use BiLSTM with CRF [Lample et al., 2016] to extract entity mentions with the same BiLSTM used in event detection. The training objective of entity extraction is to minimize the negative log-likelihood loss Lent of CRF for each sentence.

For all tokens of a mention, a max-pooling operation is applied on token-level representations to get the mention representation ˜mj. As mention types have been proved effective for downstream sub-modules [Zheng et al., 2019; Xu et al., 2021], we convert the predicted discrete mention types into vectors by looking up an embedding table. After concatenating ˜mj and type embeddings lj, we get the final mention representation mj = ˜mj‖lj ∈ Rda , where da = dh + dl and dl denotes the dimension of lj. Finally, all the mentions for an entity are aggregated to get the entity representation ˆ ei via another max-pooling. For better modeling entity semantics and the latent connections for combiantion extraction, an additional BiLSTM layer is applied to get the set of entity representations E = {˜ ei}|E| i=1 .

2.3 Combination Extraction

In this section, we introduce the details of selecting pseudo triggers, building pruned complete graphs, and decoding combinations from the graphs.

Pseudo Trigger Selection

It is hard to annotate conventional triggers manually in documents due to the document length and dataset scale. We instead select a group of pseudo triggers for each event type in an automatic way.

Empirically, the triggers are keywords that play two roles: 1) triggers can be used to identify combinations; 2) triggers are fingerprints that can distinguish different combinations. Combinations are made of arguments and we can extract a specific combination by finding all the corresponding arguments. To this end, we design an importance metric that evaluates the possibility of whether a group of arguments can serve as the pseudo triggers. In general, we first select a group of argument roles as candidates for each event type according to the importance scores, and take the corresponding arguments as the pseudo triggers.

Formally, the importance score is obtained by the existence and the distinguishability. For a subset of predefined argument roles in type ti, R = {rj}|R| j=1 are selected as the pseudo trigger candidates. |R| is a hyper-parameter that denotes the number of selected pseudo triggers for each argument combination. The existence measures whether the arguments of R can identify combinations. N (R) e is the number of event records that at least one corresponding argument of R is not NULL, and N (i) is the number of total records of ti. The distinguishability is defined to satisfy that the triggers can distinguish different combinations, where N (R) u is the number of records that the arguments of R do not appear in other records in the same document. With the multiplication of existence and distinguishability, pseudo triggers are selected by picking the candidate with the highest importance score.

Pruned Complete Graph Construction

Based on the DEE task setting and data analysis, we propose an assumption that arguments in the same combination are close to each other in the semantic space. Following this assumption, we take the pseudo triggers as the core of argument combinations and formulate each combination as a pruned complete graph. As shown in the pruned complete graph of Figure 2, for any two arguments as pseudo triggers a(i) t and a(j) t in the same combination, they are bidirectionally connected, where the adjacent matrix y(i,j) A = y(j,i) A = 1. For a pseudo trigger a(i) t and an ordinary argument a(j) o in the same combination, they are connected with a directional link and y(i,j) A = 1. Besides, each argument a(i) has a self-loop connection where y(i,i) A = 1. Other entries in yA are zeros, where entities that do not participate in any combination are isolated nodes in the graph.

After obtaining entity representations, a dot scaled similarity function (Eqn 2) is applied to estimate their semantic distances:˜

where ˜ A denotes the similarity matrix, Ws,We ∈ Rda×da and bs, be ∈ Rda are trainable parameters for semantic space linear projection.

In training, we use binary cross entropy function to formulate the combination loss Lcomb.

To predict the binary adjacent matrix A of the pruned complete graph for further decoding, the threshold γ is used here (Eqn 3).

Non-Autoregressive Combination Decoding

Event argument combinations are extracted based on the predicted adjacent matrix A with a non-autoregressive decoding algorithm.

First, all the pseudo triggers are identified based on nodes' out-degree and each pseudo trigger group is recognized as a clique. If the out-degree of an entity is greater than 0 except for self-loop, then the entity is treated as a pseudo trigger. For |R| = 1, all the combinations are pseudo-trigger-centered trees, where each combination is made of a pseudo trigger with its neighbors. Otherwise, Bron-Kerbosch (BK) algorithm [Bron and Kerbosch, 1973] is applied first to find all possible cliques (step 1 in Figure 3). To apply BK algorithm, the links between arguments must be undirected, thus we first extract all bidirectional links as undirected input.

As shown in Figure 3, the next step is to find the ordinary (non-pseudo-trigger) arguments in the combination. We further exploit all the neighbors of each pseudo trigger in the clique. After that, an intersection operation is performed to find commonly shared ordinary arguments. The combination is consist of a pseudo trigger clique and their commonly shared ordinary arguments. For those extreme records that have only one argument in the combinations, all predicted entities are aggregated together as a default combination.

The document length does not affect the event record generation speed, and the time complexity of our event records generation is polynomial (O(Nt × Nc)), while DAG-based model is O(Nt × Ns × Nr), where Nt, Nc, Ns, Nr are the number of predicted types, combinations, predicted spans and the average number of roles per type. In most cases, Ns × Nr ≫ Nc, so our PTPCG consistently holds the speed advantage.

2.4 Event Records Generation

After the set of combinations C are obtained from the pruned complete graphs, the next step is to fill these combinations into event tables. First, all the combinations should match with event types. Since event detection is a multi-label classification task, there may be more than one type prediction. For all type predictions Tp = {tj}|Tp| j=1 and combinations C, we perform a Cartesian product and get all type-combination pairs {< tj , ck > |1 ⩽ j ⩽ |Tp|, 1 ⩽ k ⩽ |C|}.

For each pair < tj, ck >, we use an event-relevant feedforward network (FFN) as classifier to get possible role results for all arguments Ek in ck, and use sigmoid function to get the probabilities p(j) role (rj|ck). The loss Lrole is calculated by a binary cross entropy function.

In the role classification task, an entity can act as more than one role in a record, while a role in a table can only be filled with one entity. Following this setting, we take the entity e(k) i∗ as the argument of role r(q) j if and only if it satisfies the constraint in Eqn 4. If ∀j ∈ [1, |Tp|], p(j) role (rj|ck) < 0.5, then the pair is recognized as an invalid candidate and will be dropped.

2.5 Optimization

Our PTPCG is an end-to-end model with joint training and scheduled sampling [Bengio et al., 2015] strategy. The overall loss is a weighted sum of all losses as below:

where α1, α2, α3, α4 are hyper-parameters to reduce the unbalanced loss effect.

3 Experiments

3.1 Datasets

We use ChFinAnn [Zheng et al., 2019] and DuEE-fin [Li, 2021] datasets to make fair comparisons across all methods. (1) ChFinAnn is by far the largest DEE dataset constructed by distantly supervised alignment without trigger annotations, which is widely used in the previous studies. This dataset contains 32k financial announcements and 48k event records, in which 29% documents have more than one record and 98% of records have arguments scattered across different sentences. On average a document contains 20 sentences and the longest document contains 6.2k Chinese characters. (2) DuEE-fin is another dataset for DEE with trigger annotations. It contains 13 event types and 11.7k documents, where the test set is evaluated online. Each record in DuEE-fin has a labeled trigger word without specific positions, and 36% of records share the same trigger in a document.

3.2 Experiment Settings

We choose the hyper-parameters of our system according to the performance on the development set in ChFinAnn. In PTPCG, we use 2 layers of shared BiLSTM for event detection and entity extraction, and another 2 layers of BiLSTM for entity encoding. We use the same vocabulary as [Zheng et al., 2019] and randomly initialize all the embeddings where dh=768 and dl=32. Adam [Kingma and Ba, 2015] optimizer is used with a learning rate of 5e-4 and the mini-batch size is 64. The weights in Equation 5 are 0.05, 1.0, 1.0, and 1.0, and γ in Equation 3 is 0.5. Following the setting in Zheng et al. [2019], we train our models for 100 epochs and select the checkpoint with the best F1 score on the development set to evaluate on the test set.

3.3 Baselines andMetrics

Baselines. (1) DCFEE [Yang et al., 2018] has two variants here: DCFEE-O extracts only one record from one document while DCFEE-M extracts records as much as possible. (2) Doc2EDAG [Zheng et al., 2019] constructs records as argument chains (DAG) and uses an auto-regressive way to extract final results. (3) GreedyDec is a baseline from Zheng et al. [2019] which fills one event table greedily. (4) GIT [Xu et al., 2021] is another variant of Doc2EDAG and utilizes a graph neural network to further encode entities and add more features in DAG generation.

Metrics. We use the same evaluation setting as in the previous studies [Zheng et al., 2019; Xu et al., 2021]. For each predicted record, a golden record is selected without replacement by matching the record that has the same event type and the most shared arguments, and F1 scores are calculated by comparing arguments. DuEE-fin uses an online evaluation fashion and the platform only provides micro-averaged scores.

3.4 Main Results

As shown in Table 1, PTPCG achieves better or competitive results than the previous systems. On the ChFinAnn dataset, our PTPCG achieves the best scores evluated on ChFinAnn-Single. As for ChFinAnn-All, our model outperforms Doc2EDAG and is very close to GIT. On the DuEEfin dataset, PTPCG achieves the best F1 scores whether or not trigger words are provided, demonstrating good compatibility and universality. Our PTPCG outperforms GIT and improves the overall F1 with 7.2% w/o manually annotated triggers. For DuEE-fin w/ Tgg, we find our approach declines from 60.3% to 58.1%. After anaylzing the dataset, we find that the manually annotated triggers have less importance scores (62.9) than our pseudo triggers (83.8, |R|=1). Although adding annotated triggers when |R|=1 can boost the importance score to 93.7, it is a trade-off between the number of triggers (annotated & pseudo) and the adjacent matrix prediction. We will discuss more in Section 3.7.

3.5 Comparison onModel Parameters and Speed

Comparison on the Number of Parameters. As shown in Table 1, among all the models, PTPCG is the most lightweight one and is in the same scale with DCFEE, while PTPCG outperforms DCFEE-O with 16.2% absolute gain in F1 scores on ChFinAnn-All. Without considering 16M vocabulary embedding parameters, PTPCG takes only 19.8% parameters of GIT and reaches competitive results.

Comparison on Speed. Benefiting from the lightweight architecture design and non-autoregressive decoding style, our PTPCG is fast in both training and inference.

Comparing the training time as shown in Table 1, it takes 633.6 GPU hours to train GIT, while PTPCG is 26.4 times faster and takes only 24.0 GPU hours. These facts indicate that PTPCG is more efficient in training and requires much lower computation resource cost than other methods, and the whole training process has been reduced from almost a week with 4 NVIDIA V100 GPUs to just one day with 1 GPU.

According to the inference speed test in Figure 4, PTPCG is more scalable than other models. With the growth of batch size, PTPCG becomes faster and finally stabilized near 125 documents per second, while Doc2EDAG and GIT are barely growing with batch size, peaking at 19 and 15 docs/s respectively. PTPCG is up to 7.0 and 8.5 times faster compared to Doc2EDAG and GIT. As an enhanced version of Doc2EDAG, GIT is 21.2% slower than Doc2EDAG on average, and raises OOM error on a 32GB memory GPU when batch size is 128. To further check the effect of different encoders, we substitute all BiLSTM layers into transformer encoders and create the TransPTPCG model with the same scale to Doc2EDAG parameters. The results in Figure 4 show that TransPTPCG is up to 2.5 times faster than Doc2EDAG, validating the advantage of non-autoregressive combination decoding.

3.6 Pseudo Triggers as Supplements

PTPCG is capable of handling with or without manually annotated triggers, and the automatically selected pseudo triggers can be supplements to enhance the performance.

As shown in Table 2, we find the pseudo triggers could assist the trigger words to boost the importance score from 62.9 to 93.7, and the results show that this helps identify combinations and bring 0.7% improvement in offline dev evaluation and 0.8% improvement in the online test set.

To further validate the effectiveness of importance scores for pseudo trigger selection, we select groups of pseudo triggers with the middle and the lowest importance scores instead of the highest ones and analyze the results in Table 3. The results show that there is a positive correlation between the importance and overall scores. Nevertheless, the highest importance score (88.3%) is not equal to 100.0%, which may limit the upper bound of decoding. We will explain more about error analysis in Section 3.7.

3.7 Error Analysis

Although pruned complete graph structure is efficient for training and inference, it is not perfect in similarity calculation (adjacent matrix prediction) and combination decoding. Here we analyze the upper bounds of the combination decoding algorithm and discuss the future directions.

Results in Table 4 show that models with the lower number of pseudo triggers are better. However, models with more pseudo triggers have higher upper bounds for decoding and have higher importance scores. Why models with more pseudo triggers have greater importance but still result in lower performance? Results in Table 3 and 4 may answer this question. Models with the same |R| has a strong correlation that higher importance brings higher metric scores which validates the effectiveness of the pseudo trigger selection strategy based on importance scores. However, more pseudo triggers bring more links to connect, and the model may be not robust enough for predicting each connection correctly, leading to an adjacent accuracy decline.

Overall, it is a trade-off that more pseudo triggers improves the upper bounds and reduces combination error rates, but also brings new challenges to recover connections between entities. We believe it is the future direction to improve the similarity calculation and adjacent matrix prediction.

4 RelatedWork

Fully annotated datasets are usually small in scale [Ebner et al., 2020; Li et al., 2021]. DS-constructed datasets are large in scale, but they can hardly match triggers to records, so triggers are likely to be absent. Unlike trigger-based methods [Pouran Ben Veyseh et al., 2021; Du and Cardie, 2020], Liu et al. [2019] argue that event types can be detected without triggers. Yang et al. [2018] extract event records using entities in a window of key sentences. It is efficient but misses lots of information from other sentences. To fully utilize all entities in the whole document, Doc2EDAG [Zheng et al., 2019] formulates the argument combination to be a directed acyclic graph (DAG), making great progress in DEE. GIT [Xu et al., 2021] is a variant of Doc2EDAG where a graph neural network is added to help entity encoding and further exploit the global memory mechanism during decoding. DAG extracts combinations in an auto-regressive way, which is very time consuming and needs huge space in global memory module to store all previous paths. Besides, Doc2EDAG and GIT are both large models, where Doc2EDAG and GIT use 12 and 16 layers of transformer encoders. To train such models, a minimum of four 32GB GPUs would have to be run for almost a week. Huang and Jia [2021] utilize BERT [Devlin et al., 2019] as sentence representations and exploit relations between sentences via graph neural networks [Veliˇckovi´c et al., 2018] to help identify fixed number of combinations. To speed up DEE, we propose a novel non-autoregressive decoding strategy and boost the training & inference speed with competitive results compared with the previous works.

5 Conclusion

Pursuing fast and general document-level event extraction (DEE), we propose a non-autoregressive model named as PTPCG. For DEE without triggers, we first select a group of pseudo triggers to build pruned complete graphs, and then train a lightweight model to extract all possible combinations, which enables non-autoregressive decoding and is up to 8.5x faster compared to SOTA models. For DEE with annotated triggers, pseudo triggers also show the power to make improvement and are even better than annotated triggeronly method. In summary, our model costs less resources yet achieves better or comparable results, compared with the previous systems.

Although PTPCG is not perfect in adjacent matrix prediction, we believe it has the ability to combine different DEE tasks and needs more explorations.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant No. 61936010 and 61876115), the Priority Academic Program Development of Jiangsu Higher Education Institutions, and the joint research project of Huawei Cloud and Soochow University. We would also like to thank the anonymous reviewers for their valuable comments.