Strikingness-Aware Evaluation for Temporal Knowledge Graph Reasoning

1 Introduction

Recent advances in Temporal Knowledge Graph Reasoning (TKGR) have led to substantial progress, which can be broadly categorized into two classes according to whether they forecast future events: interpolation and extrapolation reasoning [Jin et al., 2020]. The former refers to inferring missing historical facts, while the latter involves predicting future events, also named temporal knowledge graph forecasting [Sun et al., 2021]. This work primarily focuses on extrapolation reasoning, which is essential for many high-risk applications like financial risk control [Aven, 2013].

Despite promising empirical results [Liang et al., 2024], many reported gains may largely arise from data biases, leading to misjudgment of advancements in the field [Kervadec et al., 2021]. A historical parallel exists in the static KGR field, potential data leakage in well-known benchmarks (WN18, FB15k) led to an overestimation of the reasoning capabilities of models [Toutanova and Chen, 2015; Dettmers et al., 2018]. Over 94% and 81% of queries, such as (A, hypernym, ?), in WN18 and FB15k can easily be mapped to a training triple (B, hyponym, A) if it is known that hyponym is the inverse of hypernym. Recently, a similar phenomenon is emerging in the TKGR field: over 80% of events have occurred in prior history in the ICEWS data [Zhu et al., 2021]. It may enable heuristic-based predictions, inflating state-of-the-art (SOTA) performance, under the existing TKGR evaluation framework, on common events while obscuring poor accuracy on fewer than 10% of truly challenging, striking cases. For example, given a query such as (A,MakeV isit, ?, Tq), they may output an answer B by selecting either the most frequently occurring historical event (A,MakeV isit,B, Ti) [Lee et al., 2023; Xu et al., 2023]. It raises doubts about the predictive quality of current TKGR methods and whether the current evaluation framework could reasonably reflect these models' forecasting capabilities [Gastinger et al., 2024b].

∗Corresponding author.

The above flaw in static KGR has been addressed by removing inverse relation triples from their training sets, i.e., creating WN18RR [Dettmers et al., 2018] and FB15k-237 [Toutanova and Chen, 2015]. However, this straightforward removal strategy is fundamentally inapplicable to TKGR, since all historical events, even repetitive ones, constitute essential evidence for forecasting the future. This dilemma raises a critical question: How to construct a more meaningful evaluation framework for TKGR without deleting data?

A principled feasible alternative is to re-weight testinstance gains, rather than treating all instances uniformly. Specifically, trivial events such as (A, MakeVisit, B) occurring across different timestamps frequently should be assigned lower weights. In contrast, rarer outstanding events like (A, Sign Agreement, B), which require deeper temporal reasoning, should be emphasized. Generally, accurately inferring outstanding events offers far greater practical value than merely predicting numerous trivial ones. However, measuring the weights and automatically identifying outstanding ones from a large volume of trivial events is nontrivial. While there have been some efforts to measure strikingness of facts in static KGs, there is a notable lack of studies in TKGs field. This gap stems from two challenges: First, beyond the statistical features, a comprehensive TKGR evaluation framework necessitates incorporating both semantic and temporal relevance. Second, since the ground-truth impact of a future event is unknowable in advance, any measure of its strikingness can only be derived from observable historical patterns.

To address this, we propose a Rule-based Strikingness Measuring Framework (RSMF) to measure the strikingness of future events based on historical evidence. RSMF first leverages first-order temporal rules to retrieve peer events for the target event. Subsequently, it computes the expected occurrence of candidate events with the semantic confidence of rules, the temporal characteristics of events, and the frequency of event repetition. The strikingness of the future event is derived by contrasting its expected occurrence with that of its peer events. Finally, we construct a strikingness-aware evaluation framework by using strikingness as a weighting factor. Experimentally, we evaluate eight representative baselines under the striking-aware evaluation framework across four widely adopted TKG datasets, including three path-based, three representation-based, and two large language model (LLM)-based approaches. Our contributions and key findings can be summarized as follows:

• We propose RSMF to quantify event strikingness in TKGs, and build a new corresponding striking-aware TKGR evaluation framework that re-weights test instances with their strikingness.
• We show that the reasoning performance of evaluated models decreas as event strikingness increases, i.e., events with higher strikingness are more difficult to predict.
• We find distinct performance patterns across baselines: path-based methods show stronger performance in lowstrikingness events, whereas representation-based approaches excel at high-strikingness events.
• We design an ensemble method combining path- and representation-based models, aiming to leverage their complementary strengths. Consequently, it separately obtains significant and marginal gains in the original and our proposed strikingness-aware framework. Analysis reveals that while the method's gains come from dominant low-strikingness events, whereas, performance on rare high-strikingness events decreases.

2 RelatedWork

Temporal Knowledge Graph Reasoning Evaluation In recent years, researchers have proposed various extrapolation TKGR methods, including graph neural networks-based [Li et al., 2021; Li et al., 2022; Chen et al., 2024], rulebased [Liu et al., 2022; Huang et al., 2024], reinforcement learning based [Sun et al., 2021; Zheng et al., 2023; Dong et al., 2023], and the increasingly popular large language models-based methods [Lee et al., 2023; Liao et al., 2024; Xia et al., 2024]. Alongside these advancements, the common rank-based evaluation, for link prediction like KGR, methods are undergoing continuous refinement. Initially, to address the issue of multiple answers for a single query, correct answers other than the target answer are filtered out during ranking to avoid underestimating model performance [Bordes et al., 2013]. Subsequently, to accommodate TKGR, time-aware filtering [Han et al., 2021] and time interval prediction evaluation were introduced [Jain et al., 2020]. Considerable efforts have also been made in reevaluating the performances of various models to establish fair comparisons [Sun et al., 2020; Ruffinelli et al., 2020; Gastinger et al., 2023]. In addition, a valuable baseline named Recurrency highlight flaws in datasets and offered significant insights [Gastinger et al., 2024b]. To explore the capability boundaries of TKGR models, some studies attempt to build new benchmark datasets tailored to different scenarios, such as context-aware [Ma et al., 2023], multi-modal [Li et al., 2024], and large-scale settings [Gastinger et al., 2024a]. However, they may also suffer from the above data biases, as the repetitive pattern is an inherent characteristic of TKGs.

Outstanding Facts Mining in Knowledge Graph Outstanding facts (OFs) mining focuses on quantifying the strikingness of facts. Early research focused on extracting OFs from unstructured data (e.g., text) [Angiulli et al., 2009; Hassan et al., 2014; Wu et al., 2012]. Maverick [Zhang et al., 2018] firstly measured event strikingness in static KGs with the specific attribute values of entities. FMINER [Yang et al., 2021] introduced context entity constraints and designed a pattern relevance model to optimize the process of event searching. The robustness of measured outstanding events is further explored using perturbation analysis [Xiao et al., 2024]. To the best of our knowledge, our framework is the first principled extension of the established Outstanding Fact Mining paradigm from static KGs to TKGs. Furthermore, we transform a mining technique into a comprehensive evaluation framework, creating weighted metrics that reorient the TKGR field towards valuing outstanding reasoning.

3 Strikingness-Aware Evaluation

3.1 Preliminaries

Temporal Knowledge Graph Reasoning A TKG can be represented as a sequence of timestamp KGs, denoted as G = {G1, G2, . . . , Gt}. Each KG at a specific timestamp t is defined as Gt = (E,R,Ft), where E is the set of entities, R represents the set of relations, and Ft = {(s, r, o, t)} refers to the set of events observed at timestamp t. Given a query (s, r, ?, t), a reasonable TKGR model is to infer the object o based on the facts observed before t, where s and o are subject and object entities, r is a relation, and t is a timestamp. For instance, the query (Markieff Morris, join, ?, 2025-02) requires the model to predict the Lakers based on events before 2025-02 to validate its forecasting capability in practice.

Strikingness of Events Strikingness quantifies how outstanding a target event f = (s, r, o, t) is, which is a continuous value in the range [0, 1]. The closer the value is to 1, the more outstanding the event, and vice versa. Events with low strikingness can be referred to as trivial events, while events with high strikingness can be referred to as outstanding events. Since it is not meaningful to compare two entirely unrelated events, such as Markieff Morris will join the Lakers in 2025 and The Federal Reserve will cut interest rates in 2026, the strikingness thus is defined by comparing it with peer events P.

Peer Events A peer event is a related event of the target event generated by replacing entities or relations. For a target future event f = (s, r, o, t), its peer events are defined as P s f = {(s′ , r, o, t)|s′ ∈ E}, P o f = {(s, r, o′ , t)|o′ ∈ E}, and P r f = {(s, r′ , o, t)|r′ ∈ R}.

3.2 Strikingness Measuring

To measure the strikingness of an event, three key challenges must be addressed: 1) Constructing a set of peer events that can be used for comparison with the target future event, 2) Computing the expectation of occurrence of the target event and its peer events, and 3) Calculating the strikingness score of the target event using the expectation of occurrence. The overall procedure for RSMF is outlined in Figure 1.

Peer Events Retrieval Peer events can be obtained by replacing the entities or relations of the target event. However, direct substitution may generate many meaningless peer events, such as (Markieff Morris, join, Microsoft, 2025-02). Therefore, we utilize temporal rules to constrain the generation of peer events from historical KGs.

For a target future event f = (s, r, o, t), we first obtain the rule set TR corresponding to the relation r through rule mining [Liu et al., 2022]. While higher-order rules could capture more complex patterns, they also introduce exponential computational complexity and risk of overfitting. Thus, we only use the length 1 rules as a practical measure. A temporal rule is defined as follows:

where T1 < T2, rh and rb denote rule head and body relation, Ei and Ti indicate entity and timestamp variables.

For ease of understanding, we take the retrieval of the object entity as an example. We first mask the object entity in the target event to convert it into a query fq = (s, r, ?, t), and then use the historical KG sequences and temporal rules to search for historical events that support this query. For a rule tr ∈ TR, we ground the rule body in the historical KG sequences {Gi}t-1 i=t-w to obtain the grounded historical events:

where rb represents the relation of the rule body for tr, while w controls the window of the historical KG sequences. The grounded events of all rule bodies for tr:

Then, we take the object entities in F o f as the candidate set C o f = {o′ |(s, rb, o′ , t′ ) ∈ F o f } for object entity replacement. Further, peer object events P o f = {(s, r, o′ , t)|o′ ∈ C o f } of the target event f can be generated by substituting the object entities o. The peer relation events P r f and the peer subject events P s f could be obtained similarly.

Expectation of Occurrence Strikingness is related to human expectations regarding the occurrence of an event. To this end, we employ a rule-based approach to compute the expected scores of the target event and its peer events. For a peer event f ′ , we ground the instances by applying each rule tr ∈ TR to obtain the rule grounding events pair:

Each rgtr f ′ consists of a rule body and a rule head means the body event could support the occurrence of head event.

Furthermore, we also consider the impact of event frequency on strikingness measurement. Intuitively, the more frequently the rule grounding observed, the higher the expectation of event f ′ . Therefore, we iteratively collect rule grounding to obtain the set of rule grounding:

where ty1 ≥ t - w, and n represents the number of rule groundings. The temporal constraint is utilized to avoid overlap and redundancy. Additionally, we set thn = t, indicating that in the temporally closest grounding, only the rule body is a historical event, which provides support for reasoning about the potential future event f ′ .

After obtaining the rule grounding set, we compute the expectation score of target event and its peer events. Following the rule-based reasoning methods [Ott et al., 2023; Huang et al., 2024], we design the expectation score from two aspects. First, the effectiveness of rule-based reasoning will decay over time, and we use an exponential distribution to model this decay. Second, since different rules contribute variably to expectation, we employ the confidence of each rule to reflect its contribution. The expectation score is calculated as follows:

where conf(tr) represents the confidence of rule tr, λ > 0 is the temporal decay coefficient, and ty denotes the timestamp of the rule body in the rule grounding rgtr f ′ .

According to the different elements being replaced, we obtain the corresponding score sets of the target event f and transform them to vectors, denoted as scsf ∈ R|Cs f | , scr ∈ R|Cr f | , and sco ∈ R|Co f | . These three sets of scores estimate the expectation of events from different perspectives.

Strikingness Calculation The expectation score reflects the prior perception of the likelihood of an event's occurrence. Strikingness measures the degree to which the event exceeds the prior expectation and is thus inversely correlated with the expectation score. That is, the higher the expectation score assigned to the event f ′ , the less striking the occurrence of the event. To constrain strikingness within the range [0, 1], the obtained score sets need to be normalized as follows:

where body element be ∈ {s, r, o} is a replaced element, and || · ||2 is L2 normalization.

Then, we compare the normalized scores of peer events f ′ to highlight the prominence of the target event f. The strikingness scoring function accounts for both the magnitude of peer event scores and the differences between them. Based on the strikingness measure proposed in [Angiulli et al., 2009], we adopt the following function to calculate the strikingness of the body elements:

where scbe f ′ ∈ scbe norm, and I(·) is the indicator function that returns the value 1 if the condition is true and 0 otherwise.

Finally, we weight the strikingness of all body elements to obtain the final strikingness of potential future event f:

where αs ; αo ; αr ∈ [0, 1] are weights of body elements and αs + αo + αr = 1.

3.3 Striking-aware Evaluation Framework

Given a test event f = (s, r, o, t), its rank of score is computed by corrupting candidate entities. Specifically, the object entity would be replaced by another candidate ec, and the candidate event fc = (s, r, c, t) would be scored. By adding the reverse events, i.e., f ′ = (o, r-1 , s, t) and replacing s by c, the subject entity prediction could be achieved. The two metrics are widely used to evaluate model performance:

Mean Reciprocal Rank (MRR) evaluates ranking quality by averaging the inverse rank of the correct result:

Hits@k (H@k) measures the proportion of queries where the correct result appears in the top-k positions:

where F ′ test = Ftest ∪ {(o, r-1 , s, t)|(s, r, o, t) ∈ Ftest}, and I{T rue} = 1 and I{False} = 0.

However, the approach assigns equal weight to all future events, making it unable to capture the model's ability to predict outstanding events. To address this limitation, we propose a striking-aware evaluation framework to evaluate existing TKGR baselines. Specifically, the computed strikingness scores are used as weighting factors to calculate the Weighted MRR (WMRR) and Weighted Hits@k (WHits@k) metrics, as described below:

where |N| is the size of the test set and si is the strikingness of the event, which ensures that high-strikingness events contribute more to the metric. Setting b is equivalent to assigning a higher cost of mis-prediction to outstanding events in the evaluation. Specifically, with b = 0.1, an event with strikingness sk = 1 receives approximately ten times the weight of an event with sk = 0 in the metric calculation, since (1 + b)/(0 + b) = 11. This reflects our value judgment that the utility of correctly predicting a critical outstanding event far outweighs that of correctly predicting a routine trivial one. The parameter b thus quantifies and incorporates this value judgment into the evaluation framework.

In addition, we introduce a simple ensemble method. Instead of constructing complex networks, we follow [Meilicke et al., 2021; Liu et al., 2023; Wang et al., 2024] to combine the output scores of the existing path- and representationbased methods straightforwardly to obtain the final score:

where η ∈ [0, 1] is a hyperparameter. We perform a hyperparameter search for the η on the validation set, and subsequently apply it to the test set. Our purpose in constructing the ensemble method is not to pursue higher performance. Throughout the paper, we focus on investigating the boundaries of TKGR models' reasoning capabilities.

4 Experiments

4.1 Implement Settings

Datasets Extensive experiments are conducted on four TKG datasets: ICEWS14, ICEWS18, ICEWS05-15, and GDELT. The datasets are divided in chronological order.

Baselines We conducted comparisons under a unified experimental framework, focusing on reproducible approaches, including path-based methods: Recurrency [Gastinger et al., 2024b], TITer [Sun et al., 2021], and TLogic [Liu et al., 2022], representation-based methods: RE-GCN [Li et al., 2021], TiRGN [Li et al., 2022], and LogCL [Chen et al., 2024], and LLM-based methods: ICL [Lee et al., 2023] and GenTKG [Liao et al., 2024]. We follow the time-aware filtering settings described in [Gastinger et al., 2023] 1 . The code and strikingness weights are available 2 .

ImplementationDetails For the ensemblemethod, we perform a grid search for the hyperparameter η over the range [0, 1] with a step size of 0.1, and determine its optimal value by evaluating performance on the validation set of each dataset. We set λ as 0.1 following TLogic [Liu et al., 2022]. And αs , αo , αr are set to 0.4, 0.4, 0.2. The recommended parameters setting is provided to establish a wellcalibrated evaluation framework to facilitate reproducibility and promote community adoption. We employ the rule mining framework from TLogic [Liu et al., 2022] to learn and extract temporal rules, with the distinction that we focus exclusively on rules of length 1.

4.2 TKGR Performance on Group Strikingness

Performance Across Groups Distinct from previous works that only report overall metrics for model performance, we group the data based on strikingness and compute the average performance within each group. Figure 2 presents the grouped MRR results for six baseline models and our proposed ensemble model on the ICEWS14. It could be observed that the volume of events decreases as strikingness increases. That is, low-strikingness events dominate the test set, whereas high-strikingness events are scarce, which aligns with human intuition. On performance, all models exhibit a decline with increasing strikingness, indicating that events with higher strikingness are more difficult to predict.

In overall trends, we find that path-based methods excel at predicting events with low strikingness, while representationbased methods demonstrate superior performance on highstrikingness events. This indicates that different categories of methods have distinct advantages in forecasting future events with different levels of strikingness. Based on this finding, we attempt to investigate whether a simple ensemble method can combine the advantages of both methods. Results on ICEWS0515 and GDELT exhibit consistent conclusions.

1 https://github.com/nec-research/TKG-Forecasting-Evaluation

2 https://github.com/PersimmonZ1/RSMF

For the ensemble method, we observe two phenomena: 1) Trade-off: For events with sk < 0.1 or sk > 0.5, the performance of the ensemble method lies between those of the individual methods. 2) Enhancement: For events with 0.1 < sk < 0.5, the ensemble method outperforms both individual methods. Based on these observations, it can be inferred that some models that leverage the ensemble method mainly improve the performance for low-strikingness events, which are predominant. For instance, comparing RE-GCN and TiRGN, which incorporates additional repeated events information, TiRGN significantly outperforms RE-GCN on low-strikingness events, while achieving similar performance on high-strikingness events.

Predictability To better understand the predictability of high-strikingness events, we introduce Neighborhood Overlap (NOf ), a structural metric that quantifies the richness of historical interactions between the subject and object entities.

The NOf is defined as follows:

where Ns = {o′ |(s, r′ , o′ , t′ )} ∪ {o′ |(o′ , r′ , s, t′ )} and No = {s′ |(s′ , r′ , o, t′ )} ∪ {s′ |(o, r′ , s′ , t′ )} denote the set of neighboring entities of s and o, and t - w ≤ t′ < t is consistent with the window for calculating strikingness.

As shown in Table 1, within each high-strikingness interval, events with higher NOf (richer historical evidence) are consistently more predictable than those with lower NOf . This confirms that even among outstanding events, those supported by sufficient historical evidence remain learnable. The findings validate that our strikingness measure aligns not only with event rarity but also with predictive difficulty rooted in evidence scarcity.

4.3 Strikingness-Aware Evaluation for TKGR

To mitigate the low-strikingness bias in the dataset and more comprehensively evaluate the ability of TKGR models to forecast future events, we propose a striking-aware evaluation framework. We evaluate existing models using weighted MRR and weighted Hits@k. The hyperparameter b determines the extent to which the evaluation results emphasize the model's ability to predict outstanding events. By adjusting b, the framework can place more or less weight on events, thereby controlling the balance performance between overall events and outstanding events. Figure 3 shows the WMRR of the models on ICEWS18 under different b. A small b indicates that WMRR focuses on the model's ability to predict outstanding events. When the b becomes very large, i.e., b ≥ 100, the results of WMRR are close to the original MRR. Additionally, the comparison of model performance reverses as the value of b changes. When b > 0.1, Ensemble outperforms LogCL, and TiRGN outperforms RE-GCN. However, when b < 0.1, the results change such that LogCL surpasses Ensemble, and RE-GCN outperforms TiRGN. This demonstrates that LogCL has a superior ability to predict outstanding events compared to the Ensemble method, and a similar conclusion holds for RE-GCN and TiRGN.

We empirically select b = 0.1 to provide a unified evaluation result and facilitate fair comparison. This setting balances the metric's emphasis between the model's ability to reason events with varying levels of strikingness. Table 2 shows the experimental results of the models under both the original and the striking-aware evaluation frameworks. Obviously, the absolute values of the striking-aware metrics are significantly lower than those of the original evaluation framework, which aligns more closely with the recognized challenges of the future event forecasting task. Nevertheless, it is important to emphasize that our motivation is not to lower the scores purposely. Instead, we aim to reveal the models' comprehensive predictive capabilities through the differences between the original and striking-aware metrics. Furthermore, adjustments to other hyperparameters have a negligible impact on model ranking evaluations, demonstrating the robustness of the strikingness-aware evaluation framework.

As shown, path-based methods exhibit a reduction of over 30% across all datasets, with the heuristic baseline Recurrency demonstrating a particularly striking decrease of 50%. In contrast, methods based on evolutionary representations experience a notably smaller decline, with all reductions remaining below 30%. Among them, LogCL achieves the most robust performance, with its reduction consistently maintained at around 20%. This indicates that the existing pathbased methods have not demonstrated the claimed multi-hop reasoning on TKGs. Methods based on LLMs exhibit similar behavior to path-based approaches because the context window size constrains their reasoning, typically limited to first-order neighborhood information as input.

For the ensemble method, it still achieves state-of-the-art performance under the striking-aware metrics. However, the improvement is limited compared to the original metrics. This is because the ensemble approach primarily enhances the model's predictive performance on events with low strikingness, whereas the striking-aware evaluation framework focuses on the model's ability to predict outstanding events. By restricting the improvements from low-strikingness events, the striking-aware evaluation allows researchers to more objectively assess progress in the field.

4.4 Human Evaluation

To validate the effectiveness of the proposed RSMF in measuring event strikingness, we conducted a human evaluation study involving six volunteers. The six volunteers are all graduate students (Master's or Ph.D. candidates) in artificial intelligence, with research backgrounds spanning temporal knowledge graphs, knowledge graphs, relation extraction, and named entity recognition. Specifically, we randomly sampled 3000 events and a peer event for each target event, providing the contextual information for all events. The volunteers were asked to evaluate which event in each pair exhibited higher strikingness based on the given context.

Table 3 reports the Cohen's Kappa coefficients between the evaluations of individual annotators and the proposed RSMF. Cohen's Kappa is a widely used measure for inter-rater agreement, with values ranging from -1 to 1, where -1 denotes "less than chance agreement" and 1 represents "almost perfect agreement." As shown in Table 3, the average Cohen's Kappa coefficient between RSMF and human evaluators is 0.696, indicating that RSMF achieves "substantial agreement" with human evaluators on the strikingness of events.

5 Conclusion

We observe that the current TKGR evaluation overweights trivial repetitive events, overshadowing models' ability to predict rare yet meaningful ones. To rectify this, we introduce a strikingness-aware evaluation framework that quantifies event strikingness through rule-based peer-event comparison and incorporates it as a dynamic weight into ranking metrics. Experiments on four benchmarks demonstrate: 1) a consistent performance drop as event strikingness increases, 2) a clear divide between path-based methods (strong on lowstrikingness events) and representation-based methods (superior on high-strikingness events), and 3) we design a simple ensemble method and find it mainly improves prediction of repetitive events, with limited gains on rare, striking events. Our framework recenters evaluation on the forecasting of outstanding events, offering a rigorous and meaningful benchmark for TKGR, and calls for future work to prioritize reasoning beyond repetition.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant No. 62276110 and in part by the fund of Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL). The authors would also like to thank the anonymous reviewers for their comments on improving the quality of this paper.