Local and Global: Temporal Question Answering via Information Fusion

1 Introduction

QA aims to answer questions expressed in natural language via specific answers and has a wide range of application scenarios. Recently, many studies have been devoted to the use of KGs containing facts in the form of (subject, relation, object) as external knowledge sources to improve the performance of QA [Lukovnikov et al., 2017; Zhang et al., 2018;

∗Corresponding Author

Liang et al., 2019a; Liang et al., 2019b; Huang et al., 2019]. Notably, some facts are associated with temporal properties (i.e., timestamps or time intervals), which are typically represented in the form of quadruples (subject, relation, objective, time), for example, (Cristiano Ronaldo, member of, Manchester United FC, [2003, 2009]). Studies on QA of KGs consisting of time-dependent facts have received increasing attention from both academia and industry. This line of work follows a dominant learning paradigm, in which questions are fed into large-scale pre-trained LMs to obtain the corresponding question representations, and then the representations are combined with entity embeddings obtained using KG embedding algorithms to infer the correct answers [Saxena et al., 2021; Mavromatis et al., 2021].

Despite the relative success of previous models in the task of temporal KGQA, existing efforts can be greatly compromised in practice, primarily due to their limitations in the following respects: (I) Existing approaches almost do not emphasize the graph structural information among entities in KGs and fail to model multi-hop relational paths explicitly, which are beneficial for reasoning, as demonstrated in previous research [Ren et al., 2020]. In essence, these models are retrieval-based approaches that performwell in simple question reasoning. For example, the question "Which team was Cristiano Ronaldo part of in 2006?" can be answered with a single fact (Cristiano Ronaldo, member of, Manchester United FC, [2003, 2009]) from a KG. However, such models struggle when answering the given questions requires multiple facts or multi-hop reasoning (i.e., complex question reasoning). Hence, incorporating the structural information of KGs can facilitate complex question reasoning, which remains unexplored in temporal KGQA tasks.

(II) The global information (i.e., sentence-level semantic information) related to the question and the local information (i.e., entity-level information) of the entities involved are essential to answer the question. However, in previous methods, question understanding is typically performed by pre-trained LMs that implicitly encode the corpus only. In other words, they consider only the global information and ignore the rich semantic information (i.e., local information) of the entities involved. For example, for the question "With whom did Cristiano Ronaldo play on the FC in 2006?", the local information (i.e., Manchester United FC) is not contained explicitly but exists in the sub-graph containing the entity Cristiano's æ-hop neighbors extracted from KGs. Therefore, it is beneficial to infer the answer if the entity-level semantic information of the extracted sub-graphs is captured. Moreover, these proposed methods lack a certain transparency about their predictions, since they do not model the reasoning paths well and the whole process is invisible. As a result, interpreting the reasoning process is challenging.

To address the aforementioned limitations, in this work, we propose a novel model, LGQA, for temporal KGQA. Our goal is to develop a reasoning model that can effectively infer answer entities for the given questions. Concretely, we first employ temporal KG embedding algorithms based on given temporal KGs to obtain the embeddings of entities, relations, and timestamps. Notably, to build the timestamp embeddings with prior knowledge of the temporal order, we employ an auxiliary task for each pair of timestamp embeddings, which is crucial for further improvements in the model performance. Then, to address limitation I, we explicitly leverage the structural information among entities of KGs via the graph neural networks (GNNs). Moreover, to directly model relational paths, we perform multi-hop message aggregation that allows each node to access its æ-hop neighbors within a single propagation layer, which is significantly superior to one-hop propagation. Next, to solve limitation II, we extract the æ-hop subgraph of the entities from KGs and then perform the above multi-hop message passing to obtain the entities' local information. At the same time, we feed the question into LMs to obtain its global information. Finally, we combine the local and global information into a sophisticated information fusion layer, followed by a model prediction layer. In modeling relational paths, we introduce an attention mechanism to score the reasoning path. In this way, our model can be interpreted according to this score when reasoning.

Overall, our contributions in this work are as follows:

• We propose a novel model named LGQA, which can effectively understand a question and infer the correct answer. To the best of our knowledge, we are the first to apply GNN layers with a multi-hop message passing paradigm for temporal KGQA.
• We leverage the structural information of KGs and combine global and local information for the given questions. Additionally, our model can provide trustworthy predictions based on the attention weights of the relevant reasoning paths.
• We perform extensive experiments on two widely used benchmarks, and the empirical results demonstrate the significant superiority of our model compared to other competitive baselines.

2 RelatedWork

Temporal KGQA. Generally, KG embedding algorithms [Bordes et al., 2013; Trouillon et al., 2017] are employed to initialize entity and relation embeddings to help answer a question in the task of KGQA [Saxena et al., 2020]. For temporal KGQA, we typically adopt temporal KG embedding approaches, such as TComplEx [Lacroix et al., 2020], for initializing and also obtain the timestamp embeddings. Recently, many researchers have focused on temporal KGQA and have proposed corresponding methods for this task. Among these models, there are three representative ones: CronKGQA [Saxena et al., 2021], TSQA [Shang et al., 2022], and TempoQR [Mavromatis et al., 2021]. CronKGQA utilizes recent advances in temporal KG embeddings and feeds the given questions to pre-trained LMs for answer prediction. TSQA is equipped with a time estimation module that allows unwritten timestamps to be inferred from questions, and presents a contrastive learning module that improves sensitivity to time relation words. TempoQR designs three modules to deepen the question understanding with context, entity, and time-aware information.

Graph Neural Networks. GNNs have attracted much attention due to their ability to model structured data and have been developed for various applications in practice [Liu et al., 2021a; Liu et al., 2021b; Liu et al., 2022]. Among these models, graph convolutional network (GCN) [Kipf and Welling, 2017] is a pioneering work that designs a local spectral graph convolutional layer for learning node embeddings. GraphSAGE [Hamilton et al., 2017] generates node embeddings by learning an aggregator function that samples and aggregates features from the nodes' local neighborhoods. Graph Attention Network (GAT) [Veliˇckovi´c et al., 2018] assigns different weights to different neighbors of a node to learn its representations by introducing self-attention mechanisms. Recently, several models [Feng et al., 2020; Yasunaga et al., 2021] have been designed to shift the power of GNNs to general QA tasks. However, these models use vanilla GNNs that adopt a one-hop neighbor aggregation mechanism, which may limit their expressiveness. Additionally, these models cannot be directly applied to our focused scenarios, i.e., temporal KGQA.

3 Definition

Temporal KGQA aims to find suitable answers from KGs G = (V, E, R, T ) for given free-text questions. The answer is either an entity from entity set V or a timestamp from timestamp set T. Here, R and E represent the union sets of relations and edges. Each edge represents a valid fact in the form of quadruples (s, r, o, t), where s, o ∈ V are the subject and objective entities, r ∈ R is the relation, and t ∈ T is the timestamp, respectively.

Following previous models [Saxena et al., 2021], we formalize temporal KGQA as a link prediction problem. The underlying idea is to regard the question as a virtual relation to infer the answer. For example, for the question q "What award did Cristiano Ronaldo receive in 2008?", we can answer it with the single fact (Cristiano Ronaldo, award received, Ball d'Or). In fact, we can infer the relation "award received" from the question's content, i.e., virtual relation. Thus, we can solve it by the link prediction manner, which can be transformed into (Cristiano Ronaldo, q, ?, 2008).

Temporal KG embeddings aim to learn low-dimensional embeddings based on the facts contained in the KG. Concretely, we embed s, o ∈ V , r ∈ R, and t ∈ T based on the predefined score function ϕ(·) to obtain the corresponding embeddings es, eo, er, et ∈ R2D . Typically, the valid fact (s, r, o, t) is scored much higher than invalid facts (s′ , r′ , o′ , t′ ), i.e., ϕ(s, r, o, t) ≫ ϕ(s′ , r′ , o′ , t′ ).

4 Method

In this section, we introduce our proposed model, LGQA, for temporal KGQA, which includes three key modules: timesensitive KG embedding, information fusion and answer prediction. To better describe the method, we present the overall framework in Fig. 1. Next, we will elaborate on each module.

4.1 Time-Sensitive KGEmbedding

We start by obtaining the embeddings of the entity, relation, and timestamp in the temporal KG using a time-sensitive KG algorithm. TComplEx, a prevalent method, can produce highquality temporal KG embeddings. Specifically, it is defined in the complex space and its score function is as follows:

where Re denotes the real part in the complex space and ⟨·⟩ represents the multi-linear product operation. Additionally, es, er, eo, et are complex-valued embeddings and ¯ eo is the complex conjugate of eo.

Due to the learning procedure of TComplEx, it is proficient at inferring missing facts in temporal KGs, such as (s, r, ?, t) and (s, r, o, ?), which is suitable for our scenarios. Therefore, in this work, we combine it with temporal order information to generate pre-trained temporal KG embeddings.

However, the vanilla TComplEx algorithm does not explicitly consider the sequential ordering information of timestamps, which is detrimental to reasoning based on temporal signals. For example, for the question "Who was awarded the Ballon d'Or after Lionel Messi?", the relevant facts are (Lionel Messi, award received, Ballon d'Or, [2009, 2009]) and (Cristiano Ronaldo, award received, Ballon d'Or, [2013, 2013]). In the embedding space, it is helpful to be aware that 2013 is later than 2009 when answering this question. Inspired by the usage of position embeddings [Vaswani et al., 2017; Jia et al., 2021], we inject temporal order information into timestamp embeddings via an auxiliary task while training temporal KGs. Specifically, we define the position embedding of the k-th timestamp tk as follows:

where 2d is the dimension of timestamps and c denotes the even or odd position in the 2d-dimensional vector. We can obtain the position embedding tk ∈ R2d via Eq. 2. This position encoding method has the properties of uniqueness (i.e., different timestamps have different position embeddings) and sequential ordering (i.e., it can reflect the relative positions among timestamps). Next, we adopt linear regression to obtain the probability of timestamp m being ahead of timestamp n for the given pair (m, n). A binary cross-entropy objective function is employed in this auxiliary task. The concrete formulas are as follows:

where σ(·) and Wts are the sigmoid function and learnable parameters. e∗ and t∗ are the trainable timestamp embeddings and the corresponding position embeddings. α(m, n)=1 if m < n and 0 otherwise, and ρ(m, n) is the predicted probability of the time order. The subsequent timestamp embeddings et are obtained with the corresponding position embeddings added (i.e., et,i = et,i + ti). And the final loss function of this module is combined with the loss function of the auxiliary task, i.e., Lts. We can obtain the desired trained embeddings of temporal KGs by performing joint training.

4.2 Information Fusion

This module aims to generate enhanced question representations by incorporating the local information of temporal KGs and the global information of pre-trained LMs.

(I) Local Information. Given the temporal KG G = (V,E,R, T), we initialize the node and edge features by the pre-trained time-sensitive KG encoder. Specifically, the value of a node is the corresponding entity embedding. The value of an edge is the concatenation of the relation and timestamp embedding, i.e., er||et. The idea is to propagate both relations and timestamps via graph structures, which is specific to temporal KGQA tasks.

Next, we obtain annotated entities {ent1, ent2, · · · , entw}, which are pre-annotated by hand-crafted templates, from each question q. For each entity enti, we then extract its æhop sub-graph Gi. The final relevant æ-hop sub-graph Gq for the question can be obtained by combining each entity's subgraph, i.e., Gq = ∪wi=1Gi. Note that we restrict the answer selection to Gq via the latent sub-graph extraction procedure, which can greatly reduce the search space and effectively facilitate the training process.

To directly leverage the structural information among entities of temporal KGs, we apply GNNs to the extracted subgraph. Typically, the classic message passing paradigm of GNNs can be formulated as:

where Nv is the set of node v's neighbors. aℓv is the aggregated message at layer ℓ, and hℓv is node v's embeddings at layer ℓ obtained by combining hℓ-1 v and aℓv . However, in the above framework, the nodes in the graph can only access their one-hop neighbors through a single graph layer. In other words, suppose two nodes are not directly connected, they can only interact with each other by stacking a sufficient number of layers, which severely limits the capability of GNNs to explore the relationships between disjoint nodes.

To address this problem, we adopt a multi-hop message passing mechanism that works on all possible paths between two nodes. The first step is to compute the normalized attention using Eq. 5.

where Wℓad is the learnable weight shared by the ℓ-th layer. hℓi is the embedding of node i, initialized by h0i = ei. hr and ht are the embeddings of relation r and timestamp t, respectively. δ denotes the ReLU activation function. A ℓ and Aℓ represent the attention matrix obtained by applying the edges appearing in G and the normalized attention matrix derived by performing a row-wise softmax function, respectively. || denotes the concatenation operation. In addition, since paths with different importance are assigned corresponding weights using Eq. 5, we can derive the reasoning path based on these weights, which is further discussed in Section 6.

To enable the aggregation of multi-hop messages to a target node within a single propagation layer, we employ a mechanism defined as follows:

where ξτ are trainable coefficients. A(τ) is the powers of A, which considers relational paths with length limits up to τ from neighboring nodes to the target node. In other words, the target node's context (i.e., intermediate neighbors) and its local graph structure are involved in attention calculation. This procedure successfully creates attentional interactions between a node and its disjoint neighbors beyond one-hop. In practice, we can achieve impressive performance when empirically setting the diffusion distance ℵ ∈ [2, 4] since many graphs have small-world properties with lower diameters.

Subsequently, the transition matrix D is leveraged to update the nodes' embeddings to obtain Hℓ+1 in Eq. 7.

Finally, we perform an average pooling operation on the nodes of the extracted sub-graph to acquire the question's local information Qloc, formulated as Eq. 8.

where Vq is the node set of the sub-graph and hLi is the node embeddings at the L-th layer.

(II) Global Information. To obtain the global information of the question, we feed the question to the pre-trained LMs, such as BERT, since such models implicitly encode world knowledge. Concretely, we first insert the [CLS] token into question q. Then, we identify all the entities in q and mask them with the [MASK] token. For example, for q "Who is the president of USA after Obama?", we identify the entities "president of USA" and "Obama" and transform q into "[CLS] Who is the [MASK] after [MASK]". Finally, the tokenized question is fed into BERT, and it can be expressed as:

where Wq is the projection matrix. In addition, ¯Q = [¯Q[CLS], ¯Q1, · · · , ¯Qo] is an embedding matrix. We adopt the [CLS] token embedding, ¯Q[CLS], as the representation of the entity-independent question q. For the masked entities, we use pre-trained temporal KG entity embeddings. In other words, if the question contains two annotated entities, the global information is Qglo = ¯Q[CLS] + e1 + e2. To further enhance question representation and to make full use of the available data, we retrieve the relevant facts of the annotated entities in the question from the temporal KG, so that we can obtain the question-specific time scope. If we retrieve multiple timestamps of relevant facts, we sort them and keep only the start time and end time. For example, for question q, we can retrieve the fact (Barack Obama, held position, president of USA, [2008, 2016]) and obtain two time embeddings t1 and t2 that correspond to the temporal KG

embedding for start time 2008 and end time 2016, respectively. Hence, the global information can be rewritten as Qglo = ¯Q[CLS] + e1 + e2 + t1 + t2.

To better integrate the question's local and global information, we employ a sophisticated knowledge fusion layer, Φ(·), that contains several Transformer encoder layers. After performing the Transformer-based information fusion layer, we obtain the final question representation, i.e., Qfin = Φ(Qloc||Qglo).

4.3 Answer Prediction

We use two-layer MLPs to transform Qfin into Qent and Qtim, which correspond to entity and timestamp prediction, respectively, and are defined in Eq. 10.

Next, we define an entity score function ϕent(·) and a timestamp score function ϕtim(·) to obtain the scores of candidate entities and timestamps, as shown in Eq. 11.

where ˜ ε ∈ Eq and ˜ t ∈ Tq, in which Eq ⊆ E and Tq ⊆ T are specified by the sub-graph Gq with respect to the given question q.

Finally, we concatenate the obtained scores for the entities and timestamps and perform the softmax function over them to obtain the answer probability. The objective function is the cross-entropy loss, as shown in Eq. 12.

where yi is the true answer to the question.

5 Experiment

Datasets. We employ two temporal KGQA benchmarks, i.e., CRONQUESTIONS [Saxena et al., 2021] and TimeQuestions [Jia et al., 2021]. CRONQUESTIONS is the largest known dataset, which has 410K unique questionanswer pairs, where each question contains annotated entities and timestamps. Moreover, this dataset can be divided into entity and time questions based on the type of answers. It can also be divided into simple reasoning (i.e., Simple Entity and Simple Time) and complex reasoning (i.e., Before/After, First/Last and Time Join) based on the questions' difficulty. TimeQuestions is another challenging dataset, which has 16k manually tagged temporal questions and is divided into four categories (i.e., Explicit, Implicit, Temporal, and Ordinal) according to the type of time reasoning. We present the statistical information of the datasets in Tables. 1 and 2.

Baselines. We select three types of baselines for comparison on CRONQUESTIONS: (I) pre-trained LMs, including BERT [Devlin et al., 2019], RoBERTa [Liu et al., 2019] and KnowBERT [Peters et al., 2019]; (II) general KG embeddingbased models, including EaE [F´evry et al., 2020] and EmbedKGQA [Saxena et al., 2020]; and (III) temporal KG

embedding-based models, including CronKGQA [Saxena et al., 2021], TMA [Liu et al., 2023], TSQA [Shang et al., 2022], TempoQA [Mavromatis et al., 2021], and CTRN [Jiao et al., 2022]. For another dataset, TimeQuestions, we use temporal KG embedding-based models for comparison.

Model Implementations. We set the weighted coefficient in the KG encoder stage as λ = 0.5. In the second stage, we extract a 3-hop sub-graph of the question, i.e., æ=3. Moreover, the hop is set to ℵ = 3. We perform 2-layer GNNs to obtain the updated node embeddings, i.e., L = 2. Furthermore, we use 3-layer Transformers with 4 heads per layer in the knowledge fusion layer Φ(·). We train our model for 20 epochs with Adam methods, and the validation performance determines its final parameters. We conduct all experiments ten times and take the average values as the final results. We leverage two popular evaluation metrics, i.e., Hits@1 and Hits@10, following previous studies.

6 Result

Model Performance. We present the results of our proposed LGQA and baselines on CRONQUESTIONS in terms of Hits@1 and Hits@10 in Table 3 and on TimeQuestions for Hits@1 in Table 4. LGQA achieves the best performance in all experimental settings, indicating its superiority on the temporal KGQA task. Remarkably, LGQA significantly outperforms the second-best model on both datasets. It achieves 7.6% and 4.9% absolute improvements on Hits@1 with respect to complex reasoning and time questions on CRONQUESTION, respectively. It also performs far better than other models for various types of questions in the TimeQuestions dataset. For example, it achieves absolute improvements of 6.3% and 6.0% on Hits@1 for questions involving 'Explicit' and 'Implicit' types. While in the 'Temporal' type of questions, our model gains an absolute improvement of 9.3% compared to the second best performing model. We attribute this to the use of the multi-hop propagation of knowledge fusion and the time-sensitive KG embedding.

We find that pre-trained LMs (e.g., BERT and RoBERTa) achieve unsatisfactory performance in this scenario, lagging far behind the general and temporal KG embedding-based models on CRONQUESTIONS. A plausible reason is that these models do not introduce KG into this task, which is detrimental to question understanding. Despite the relative success of general KG embedding-based models (e.g., EaE and EmbedKGQA) in common QA tasks, they still perform worse than temporal KG embedding-based models (e.g., TSQA, TempoQR and CTRN) in our focused scenario. A possible reason is that they do not explicitly leverage temporal KG and neglect temporal information, which is crucial for the temporal KGQA task.

We present the Hits@1 results of our model and other competitive baselines on different question types in Table 5. LGQA is significantly superior to other models, especially for complex questions. Our model gains 15.5%, 5.6%, and 9.4% absolute improvement over "Before/After", "First/Last" and "Time Join", respectively, due to the consideration of the timestamp order and multi-hop structural information of the temporal KG. Additionally, our model has comparable performance on simple questions.

Ablation Study. We conduct extensive ablation experiments on the crucial components by designing some model variants on the CRONQUESTION dataset. (I) w/o time order: We exclude the auxiliary task of encoding temporal order information and use the vanilla TComplEx method. (II) w/o multi-hop: We use the one-hop attention computed from the direct neighbors without multi-hop attention, similar to GAT. (III) w/o local: We remove the module for extracting local information. (IV) w/o global: We remove the module considering global information. The experimental results are presented in Table 6. We can obtain the following insights: First, after eliminating the global information module, the model's performance drops drastically, which is in line with our expectations. This result indicates that this module can provide helpful contextual information for accurately understanding the question. Second, since the local information can bring additional valuable information fromKGs, eliminating it can negatively affect the model. Moreover, the performance declines when we perform one-hop message passing instead of multi-hop, empirically demonstrating that multi-hop message passing is more expressive. Finally, complex questions require the temporal order information to be captured, thus removing this information inevitably harms the model.

Hyperparameter Sensitivity. We empirically explore the effects of different hyperparameters by observing the performance of LGQA on the CRONQUESTIONS dataset. We study the effect of the number of hops æ in the extracted subgraphs and the number of layers L in GNNs. The hits@1 results in terms of all questions and complex questions are presented in Fig. 2. We find that the model can achieve the best performance when extracting the 3-hop sub-graph. A possible reason for this is that smaller sub-graphs may exclude correct answers, while larger sub-graphs increase the search space for candidate answers but may bring exponential noise from KGs. Moreover, as illustrated on the right side of Fig. 2, the model's performance shows a trend of increasing and then decreasing as the number of GNN layers increases.

Impact of Training Data Size. We select several competitive models for comparison on complex questions of the CRONQUESTION dataset regarding hits@1 with different training data sizes. The experimental results are presented in Fig. 3. We find that our model consistently outperforms other baselines in all cases. Taking the 20% training data as an example, our model's hits@1 absolute improvement reaches 18.6% compared to the second-best-performing model. This demonstrates that, first, our proposed model exhibits superior expressive power in complex question reasoning. Second, it does not rely on large amounts of training data.

Model Interpretability. To interpret our model's reasoning process, we investigate the relational path attention weights induced by the attention layer of GNNs described in Eq. 5. Specifically, we trace high attention weights from entity nodes to the candidate answer nodes on the retrieved subgraph Gq by leveraging Best First Search (BFS). Fig. 4 illustrates one example. In this example, we note that the reasoning path contains "Cristiano Ronaldo" in the question and "Alex Ferguson" and "Manchester United F.C." in KGs. LGQA can make accurate predictions, i.e., "Alex Ferguson", given the question. Notably, LGQA promotes rational reasoning by introducing "Manchester United F.C.", which is not mentioned in the question, revealing the importance of background knowledge. It provides an interpretable reasonable path "Cristiano Ronaldo→Manchester United F.C.→Alex Ferguson".

7 Conclusion

In this work, we propose a novel model, LGQA, to perform temporal KGQA tasks. Three specific modules are introduced to significantly improve the model's performance. Specifically, the time-sensitive KG embedding module is employed to add temporal ordering information. Moreover, the information fusion module with multi-hop message passing during the extraction of the æ-hop sub-graphs combines the local information with global information to understand questions. Finally, we obtain the answer based on the answer prediction module. Extensive experiments on two widely used datasets imply that LGQA achieves satisfying performance.

Acknowledgments

Our work is supported by the National Key Research and Development Program of China No.2021YFF1201200, the National Natural Science Foundation of China No.62172187, No.61972175, and No. 61972174.

Contribution Statement

Note that the first three authors, Yonghao Liu, Di Liang and Mengyu Li, are equal contributions. Y.H.L., D.L., and M.Y.L. designed and developed the method and analysed the data. S.R.W. and W.W. contributed to the implementation and data analysis. Y.H.L., M.Y.L., and X.Y.F. drafted the paper. F.G., X.M.L., and L.H. revised the paper. X.Y.F., and R.C.G. supervised the project and contributed to the conception of the project. All authors read and approved the final manuscript.