Knowledge Enhanced Event Causality Identification with Mention Masking Generalizations

1 Introduction

Event causality identification (ECI) aims to identify causal relation of events in texts. For example, in a sentence S1 (shown in Figure 1): "The earthquake generates a tsunami that rose up to 135 feet", an ECI system should identify that a causal relationship holds between the two mentioned events, i.e., earthquake cause ---→ tsunami. ECI supports a wide range of intelligent applications including why-question answering [Girju, 2003; Oh et al., 2016], future event/scenario forecasting [Hashimoto et al., 2014], machine reading comprehension [Berant et al., 2014], and others.

To date, various approaches have been proposed for ECI, ranging from the early feature based methods [Do et al., 2011; Hashimoto et al., 2014; Ning et al., 2018; Gao et al., 2019] to the recent representation based methods [Kadowaki et al., 2019]. While, existing methods typically train ECI

models on human annotated examples solely, and they generally lack the ability to leverage background knowledge for reasoning. Moreover, owing to the small size of training data (for example, the largest ECI corpus contains less than 300 documents [Caselli and Vossen, 2017]), existing ECI methods suffer from over-fitting issue and have difficulty in handling new, previously unseen cases.

To address the limitations of previous methods, we propose a new approach for ECI, featured by its ability to: 1) explicitly leverage external (commonsense) knowledge for reasoning, which can build more expressive representations for events; and 2) mine event-agnostic, context-specific patterns for reasoning, which results in a decent generalization ability of our model to tackle new, previously unseen examples.

Specifically, one key component of our model is a knowledge-aware causal reasoner, which can exploit background knowledge in external knowledge bases (KBs) to enhance the reasoning process. We prefer CONCEPTNET [Speer et al., 2017] as the external KB, which contains abundant semantic knowledge of concepts (represented as words or phrases). For example, in CONCEPTNET, the encoded knowledge associated with "earthquake" includes "earthquake" IsA --→ "natural disaster", "earthquake" Causes ---→ "a tsunami" and others (as shown in Figure 1 a)). Such knowledge can be used to enrich the representations of events for a more accurate event causality inference. For example, the knowledge-aware causal reasoner may directly predict earthquake cause ---→ tsunami in S1 based on the semantic knowledge "earthquake" Causes ---→ "a tsunami" encoded in CONCEPTNET.

This indicates that explicitly introducing external knowledge may benefit the ECI task.

Nevertheless, a potential issue of the above method is that a KB is never complete [Min et al., 2013]; Especially, a KB may lack definitions of newly emerging events. To mitigate this problem, we propose a complementary mention masking reasoner, aiming to exploit the event-agnostic clues for reasoning. We motivate our approach by noting that causal statements usually contain event-independent patterns, which are helpful for identifying causality of unseen events. For example, we can distill a causality pattern: "The [SLOT] generates [SLOT] ... " from S1, which can be used to identify traffic congestion cause ---→ environmental pollution in a new sentence "The traffic congestion generates environmental pollution and economic loss". To learn such context-specific patterns, we propose a learning mechanism called event mention masking generalization, which explicitly excludes event information for learning. Methodologically, it replaces event mentions with a placeholder symbol [MASK], and force our model to make predictions based on such mask-containing texts (as shown in Figure 1 b)). This can be seen as adding a hard attention on context information, and thus enhance the ability of our model in handling unseen cases.

Lastly, we build an attentive sentinel to allow a trade-off between the aforementioned two components. This trade-off is crucial because in some cases text context should override the background knowledge for the task, and in other cases the opposite is true (For example, despite the sentence "Both of earthquake and tsunami are natural disasters" contains an event pair of earthquake and tsunami, it does not express a causal relation according the context).

In experiments, we evaluate our model on three benchmark datasets. We first concern the standard evaluation and show that our model attains state-of-the-art performance. We then estimate the generalization ability of our model by performing i) cross-topic adaptation, ii) exploring unseen predicates, and iii) cross-task adaptation. Our model demonstrates definite advantages over previous methods.

To summarize, we make the following contributions:

• We propose a new approach for ECI, which can leverage external knowledge to enrich representations of events for accurate reasoning. To our best knowledge, this is the first work explicitly introducing external knowledge for this task.
• Moreover, we propose a mention masking generalization mechanism to learn event-agnostic, context-specific patterns. This grants our model a decent generalization ability to handle new, previously unseen data.
• We conduct extensive experiments and show that our model sets up a new state-of-the-art for ECI. Moreover, our approach shows definite advantages over previous ECI methods regarding generalization evaluation.

2 RelatedWork

2.1 Event Causality Identification

The task of ECI aims to identify causal relations of events in texts, which has attracted a lot of interests among researchers.

Earlier methods for ECI are predominantly feature-based, which adopt lexical and syntactic features [Hashimoto et al., 2014; Gao et al., 2019], causality cues (such as "because", "for") [Riaz and Girju, 2014], event co-occurance patterns [Beamer and Girju, 2009; Hu et al., 2017], temporal patterns [Mirza, 2014a; Ning et al., 2018], and others for the task. The very recent work [Kadowaki et al., 2019] employs BERT architecture [Devlin et al., 2019], which can learn context-dependent representations for the task and achieves superior performance. Regarding datasets construction, Do et al. [2011] annotated a corpus consisting of 25 documents for evaluation; Mirza [2014a] annotated event causal relations in the TempEval-3 corpus and release a corpus called CausalTimeBank; Caselli and Vossen [2017] had built a corpus called EventStoryLine, which contains 258 documents in total. Hashimoto [2019] exploited weakly supervised method to construct ECI datasets. However, as introduced in Introduction, previous methods typically train a model on the annotated examples only and disregard a lot of background knowledge. Moreover, they generally have difficulty in handling new, previously unseen data, owing the limited size of training data.

2.2 Knowledge Enhanced Text Understanding

The importance of background knowledge in text understanding has long been recognized [Minsky, 1974]. With the development of knowledge bases (KBs) - ranging from manually annotated networks like WordNet [Miller, 1995] to semi-automatically/automatically constructed knowledge graphs like DBPedia [Lehmann et al., 2014] and ConceptNet [Speer et al., 2017] - large amounts of knowledge become available. Many studies have investigated to leverage such knowledge to boost text understanding tasks. To name a few, Rahman and Ng [2011] studies knowledge-enhanced entity co-reference; Yang and Mitchell [2017] took advantage of external KBs to improve recurrent neural networks for entity recognition and event detection; Zhou et al. [2018] studied incorporating commonsense knowledge for conversation generation. But to the best of our knowledge, no work has studied introducing external knowledge for ECI.

3 Approach

Figure 2 schematically visualizes our approach. Specifically, we formulate ECI as a binary classification problem, following previous works [Mirza, 2014a; Ning et al., 2018; Gao et al., 2019] - for every pair of events in a sentence, we predict whether a causal relation holds. Our approach contains three major components:

• Knowledge-aware reasoner, which retrieves background knowledge from CONCEPTNET, and then integrate the knowledge with texts for reasoning (§ 3.1).
• Event masking reasoner, which masks event mentions in texts, aiming to learn event-agnostic, context-specific patterns for reasoning (§ 3.2).
• The attentive sentinel, which adopts an attention mechanism to balance the above two components for the final prediction (§ 3.3).

We illustrate each component in details in the followings.

3.1 Knowledge-Aware Reasoner

Given a pair of events (denoted as e1 and e2), the knowledge aware reasoner first retrieves the related knowledge in CONCEPTNET, and then encodes the knowledge into contexts for reasoning.

Knowledge Retrieving. CONCEPTNET structures knowledge as graph, where each node corresponds a concept, and each edge corresponds to a semantic relation. For e1 and e2, we search their definitions in CONCEPTNET but we only consider 18 semantic relations that are potentially useful for ECI: CapableOf, IsA, HasProperty, Causes, MannerOf, CausesDesire, UsedFor, HasSubevent, HasPrerequisite, NotDesires, PartOf, HasA, Entails, ReceivesAction, UsedFor, CreatedBy, MadeOf, and Desires. Part of the knowledge related to earthquake and tsunami in S1 is shown in Figure 2.

Knowledge Encoding. To encode the knowledge and enrich representations of e1 and e2, we first conduct knowledge linearization, to transfer the discrete knowledge into a structured sequence, motivated by [Fan et al., 2019]. As shown in Figure 2, for each semantic relation, one special marker (such as 〈IsA〉) is designed and following the marker is the related knowledge separated by a delimiter 〈s〉. Then, we adopt a BERT based encoder to encode the knowledge with the context texts jointly. Specifically, we first incorporate the linearized knowledge into the sentence; then we add additional event markers 〈E1〉, 〈/E1〉 and 〈E2〉 〈/E2〉 to indicate boundaries of events (Two special tokens [CLS] and [SEP] are added at the beginning/ending of the sentence following BERT). Finally, after using BERT encoder to compute representations of the entire sequence, we concatenate representations of [CLS], 〈E1〉, and 〈E2〉 as the final representation regarding to (e1, e2), namely

where ⊕ indicates the concatenation operator; h[CLS], h〈E1〉, and h〈E2〉 are representations of [CLS], 〈E1〉, and 〈E2〉 respectively. F (e1,e2) KG is the knowledge-aware representation that would be used for further computation.

3.2 MentionMasking Reasoner

The mention masking reasoner aims to explore eventagnostic, context-specific patterns for reasoning. Specifically, e1 and e2 are firstly replaced with a special token '[MASK]' to exclude event information. Then, another BERT encoder is adopted to encode the mask-containing sentence ([CLS] and [SEP] are also added). Similar as in the knowledge aware reasoner, we regard F (e1,e2) MASK as the masked representation of (e1, e2):

where he1 [MASK] and he2 [MASK] are BERT representations of e1 and e2, which have been replaced by [MASK]. We train the mention masking reasoner with two different objectives:

Discrimination Learning. Our model is forced to predict whether e1 and e2 forms a causal relation based on the masked representation F (e1,e2) MASK . As F (e1,e2) MASK does not contain any event-specific information, our model has to explore context-specific clues for reasoning, which would gain the ability to tackle unseen events.

Distributional Similarity Learning. In distributional similarity learning, we assume causal statements may share similar representations in some ways, by taking in pair of maskcontaining statements as inputs and encouraging their representations to be similar if both of them express causal relations. Assume A and B are two pairs of events; F AMASK and F BMASK are their masked representation. We optimize the following loss to achieve distributional similarity:

where δA,B is the Kronecker function that take the values of 1 when both of A and B express a causal relation and 0 otherwise. p(l = 1|A,B) = 1 1+exp(F AMASK T F BMASK ) defines the distributional similarity score. In practice, we alternately adopt discrimination learning and distributional similarity learning to train the mention masking reasoner.

3.3 The Attentive Sentinel

The attentive sentinel aims to learn a trade-off between the knowledge aware reasoner and the mention masking reasoner, by learning an attentive gate as their combination weights, namely:

where W and b are model parameters; ⊕ denotes the concatenation operator. Then it adopts an weighted summation to integrate F (e1,e2) KG and F (e1,e2) MASK as the final feature for (e1, e2), namely:

The attentive sentinel allows to balance the knowledge aware reasoner and the mention masking reasoner to make the final prediction.

3.4 Model Prediction and Training

To make the final prediction, we perform a binary classification by taking Fe1,e2 as input:

where oe1,e2 denotes the probability of e1 cause ---→ e2; Wo and bo are model parameters. For training, we adopt crossentropy as the loss function:

where Θ denotes the parameter set of our model; s ranges over each sentence in the training set; ei, and ej ranges over each events in s. We adopt the Adam [Kingma and Ba, 2015] algorithm to optimize model parameters.

4 Experiments

4.1 Experimental Setups

Datasets and Evaluations. Our experiments are conducted on three benchmark datasets, including: a) EventStroryLine [Caselli and Vossen, 2017], which contains 258 documents in 22 topics, 5,334 events in total, and 1,770 of 7,805 event pairs are causally related; b) Causal-TimeBank [Mirza et al., 2014], which contains 184 documents, 6,813 events, and 318 of 7,608 event pairs are causally related. c) EventCausality [Do et al., 2011; Ning et al., 2018], which contains 25 documents, 1,134 events, and 414 of 887 event pairs are causally related. For evaluation, we adopt Precision (P), Recall (R) and F1-score (F1) as evaluation metrics, same as previous methods to ensure comparability. Significant test is conducted using paired t-test at a significance level of 0.05.

Implementations. In our implementations 1 , both the knowledge aware reasoner and the mention masking reasoner are implemented as BERT-Large architecture, which has 24layer, 1024-hidden, and 16-heads. We use CONCEPTNET 5.0

1 https://github.com/jianliu-ml/EventCausalityIdentification

as the KB. Regarding hyper-parameters, the batch size is set as 10, and the learning rate is initialized as 5 × 10-5 with a linear decay. We also adopt a negative sampling rate of 0.5 for training, owing to the sparseness of positive examples.

Baseline Systems. We prefer different baseline systems to compare for different datasets. For EventStoryLine, we prefer OP [Caselli and Vossen, 2017], a dummy model assigns causal relation to every event pair; 2) LSTM [Cheng and Miyao, 2017], a dependency path based sequential model that models the context between events to identify causality; 3) Seq [Choubey and Huang, 2017], a sequence model explore complex human designed features for the task. 4) LR+, and 5) LIP [Gao et al., 2019], state-of-the art ECI system that adopts document structure for the task. For Causal-TimeBank, we prefer 1) RB, a rule-based system; 2) ML, a machine learning based model, and 3) HB, a hybrid method combine rules with features for comparison. These models are designed by [Mirza, 2014a; Mirza and Tonelli, 2016] for ECI. For EventCausality, we prefer PMI, ECD, and CEA [Do et al., 2011], which adopt different co-occurrence patterns for the task as baselines systems. For each dataset, we add a BERT based model as baseline.

In our approach, we use MKG to denote the knowledgeaware reasoner, which adopts F (e1,e2) KG for prediction; we denote MMMR as the mention masking reasoner, which adopts F (e1,e2) MASK for prediction. MFULL indicates our full model.

4.2 Experimental Results

Experimental results on the three benchmark datasets follow.

EventStoryLine. Table 1 shows the results on EventStoryLine, where we use the last two topics as development set, and conduct a 5-fold cross-validation on the rest 20 topics, as suggested by [Gao et al., 2019]. From the results, our full model MFULL outperforms all baseline methods and achieves the best performance (50.1% on F1 score), outperforming the state-of-the-art model LIP by a margin of 5.5%, which justifies its effectiveness. Comparing with MKG with BERT, we note adding external knowledge improves the performance by 3.6% in F1 score. Moreover, the mention masking reasoner (MMMR) is more effective than the knowledge aware reasoner (MKG) (43.9% v.s. 41.8%). This may imply that, for small dataset, generalization knowledge is more important than discrimination knowledge.

Causal-TimeBank. Table 2 shows results on CausalTimeBank, where we adopt two settings: 1) 10-fold crossvalidation (CV) as in [Mirza, 2014a], and 2) evaluating on an additional TemporalEval-3 datasets as in [Mirza and Tonelli, 2016]. Our models show a consistence performance as in EventStoryLine, which achieves the best performance (44.1% for CV, and 66.7% for TE). Moreover, MFULL and MMASK demonstrate high recall value, which is benefited from their generalization ability.

EventCausality. Table 3 shows results on EventCausality. Note this is an extremely tiny datasets, and [Do et al., 2011] adopts a weakly-supervised methods to retrieve additional examples for training. Nevertheless, in our approach, we use only 10 documents ([Do et al., 2011] use them as seed documents) for training. From the results, the superior performance of MFULL (45.4% on F1) demonstrates the applicability of our approach for small datasets.

5 Model Generalization Evaluation

Generalization refers to a model's ability to adapt to new, previously unseen data. We conduct 1) cross-topic adaptation, 2) unseen predicates, and 3) cross-task adaptation to estimate the generalization ability of our model.

5.1 Cross-Topic Adaptation

Different topics usually involve different events. In our crosstopic adaptation, we train our model on a source topic, but test our model on other topics. We use EventStoryLine to conduct our experiments. Specifically, we first randomly select a topic as the source topic (for model training and tuning), and then we rank the remaining topics based on their similarities with the source topic (The similarity of two topics t1 and t2 is defined as Et1∩Et2 Et1∪Et2 , where Et denotes the event set of t). Finally, we test how our model performs on topics with the lowest, medium, and highest similarity value with the source topic. We re-implement previous state-of-the-art system LIP to compare with our models.

From the results in Table 4, the performance of LIP is highly depended on the similarity of source and target topics. It achieves relative good performance when the target and source topics are of high-similarity, but behaves extremely poorly when the target and source topics are of low-similarity. While our approach, especially MMMR and MFULL, are robust in cross-topic adaptation, which achieve superior performance even in low-similarity cases.

5.2 Unseen Predicates

To further test the generalization ability of our model, we conduct experiments to explore unseen predicates. For the EventStoryLine corpus, we first randomly select 1/3 of documents as the training set. Then, we divide the remaining corpus as 1) 'Both Seen' set, where both of events exist in the training data (with a size of 3,464); 2) 'One Unseen' set, where only one of the events exists in the training data (with a size of 4,381); 3) 'Both Unseen' set, where both events are unobserved during training (with a size of 1,891). From the results in Figure 3, 1) LIP behaves relatively good on 'Both Seen', but poorly on 'One Unseen' and 'Both Unseen' (only 11.3% in F1). 2) Our full model achieves the best performance on all of the three sets. 3) MMMR achieves better performance than MKG on 'One Unseen' and 'Both Unseen'.

5.3 Cross-Task Adaptation

Finally, we investigate cross-task adaptation, where we train our model on ECI datasets but test the performance of our model in generalizing to other tasks. Specifically, we train our model on EventStoryLine, but we test our model on identifying causal relations in SemEval-8 (which focuses on causal relations between entities) and FrameNet (which focuses on causal relations between frame elements). From the results in Table 5, LIP performs relatively poor in cross-task adaptation. The reason might be that features adopted by LIP are not applied to entities and frame elements. MKG behaves better than MMMR in SemEval, but much worse than MMMR in FrameNet. The reason is that, SemEval focus on relations between entities, which are more likely to have definitions on a KG. But FrameNet focuses on frame elements, which can be any span of the sentence, and do likely to have definitions on a KG. Our full model achieves the best performance among all models regarding cross-task adaptation.

6 Further Discussion

Inductive Bias. To further explore the effectiveness of our model, we investigate the prediction bias of MKG and MMMR by inspecting their outputs. Accordingly, there are 685 cause relations only predicted by MKG, 655 relations only by MMMR and 382 relations by both of them in the experiments shown in Table 1 (for a specific fold). The values change to 102, 132 and 58 in the experiment of cross-topic adaptation (T18→T33). The relatively less of their common predictions indicate that MKG and MMMR focus on different aspects of features to identify the cause relations and they share complementary effects. This provides explanation for the good performance of our full model.

Case Study. We conduct case study to further investigate the effectiveness of our model. To simplify the discussion, we limit the experiments to a specific cross-topic adaptation, i.e., T18→T33 adaptation. Table 6 shows several cases showing the outputs of MKG and MMMR. Basically, MKG is good at finding commonsense causality that is usually contextindependent, such as killing cause ---→ die in a), and murder cause ---→ on trial in b), but cannot handle context depended cases as in c), d), and e). While MMMR is completely opposite. The full model can take advantage of MKG and MMMR to make a more accurate prediction.

7 Conclusion and FutureWork

In this paper, we propose a new approach for event causality identification. Our approach on the one hand can leverage background knowledge to enhance the reasoning, on the other hand cam mine event-agnostic context-specific patterns for reasoning, which greatly enhances its generalization ability. The effectiveness of our model is verified on three datasets with diverse settings. In the future, we would like to apply our model to other NLP tasks such as relation classification, event temporal relation extraction and others.

Acknowledgments

This work is supported by the Natural Key R&D Program of China (No.2018YFB1005100), the National Natural Science Foundation of China (No.61922085, No.U1936207, No.61976211, No.61806201) and the Key Research Program of the Chinese Academy of Sciences (Grant NO. ZDBS-SSW-JSC006). This work is also supported by Beijing Academy of Artificial Intelligence (BAAI2019QN0301), CCF-Tencent Open Research Fund and independent research project of National Laboratory of Pattern Recognition.