A Canonicalization-Enhanced Known Fact-Aware Framework For Open Knowledge Graph Link Prediction

1 Introduction

Open knowledge graph (OpenKG) stores factual triples automatically extracted from text corpus by open information extraction (OpenIE) approaches, such as ReVerb [Fader et al., 2011] and CALMIE [Saha and others, 2018]. These triples are in the form of (head noun phrase, relation phrase, tail noun phrase), e.g., (Bill Gates, be the founder of, Microsoft). OpenKG is highly adaptable and can be easily exploited by new domain-specific corpora since no pre-specified ontologies are required. However, OpenKG may not be suitable for directly used by an end task owing to their sparsity and incompleteness. Therefore, tasks such as OpenKG canonicalization [Lin and Chen, 2019; Dash et al., 2021] and OpenKG linking [Dubey et al., 2018; Liu et al., 2021] are studied to reduce graph sparsity, while OpenKG link prediction task [Gupta et al., 2019; Chandrahas and Talukdar, 2021] is proposed to reduce graph incompleteness. Since OpenKG linking may be unavailable when bootstrapping OpenKG from a domain-specific corpus that lacks pre-defined entities and relations, we focus on utilizing OpenKG canonicalization for graph sparsity reduction to improve the performance of OpenKG link prediction.

* Corresponding author

Unlike curated knowledge graph (CKG), e.g., Freebase [Bollacker et al. ], OpenKG is extremely sparse since noun phrases (NPs) and relation phrases (NPs) are not canonicalized. As a result, severe long-tail problems exist in embedding-based link prediction methods, such as TransE [Bordes et al., 2013] and ConvE [Dettmers et al., 2018], where embeddings of infrequent phrases are poorly learned due to insufficient training. Recently, OpenKG link prediction starts to receive more and more attention owing to its efficient learning ability in high-sparsity situations. CaRe [Gupta et al., 2019] utilizes NPs' canonical clusters to share NP embeddings and encodes RPs' semantic meanings by GRU [Cho et al., 2014]. OKGIT [Chandrahas and Talukdar, 2021] employs a pre-trained BERT [Devlin et al., 2019] to provide type guidance for NPs. OKGSE [Xie et al., 2022] learns separate parts of embedding for NP and its cluster, and captures knowledge among NPs' neighboring triples. Nevertheless, there are still two limitations that hinder the performance of OpenKG link prediction.

First, previous methods have not taken into account RP canonicalization explicitly. The number of RPs can reach tens of thousands because of their diverse expressions, yet many of them may refer to the same semantic, such as "be a close friend of" and "become good friend with" shown in Fig. 1. In order to correlate these RPs, CaRe [Gupta et al., 2019] encodes tokens of RPs initialized with glove vectors [Pennington et al., 2014]. However, there is clearly a semantic gap between glove embeddings and randomly initialized NP embeddings, and it neglects to canonicalize RPs explicitly by sharing their semantics.

Second, triple canonicalization, which directly groups semantically similar triples, has been ignored by existing works. Prior approaches [Kolluru et al., 2021; Lovelace et al., 2021] use textual information from triples that consist of query and promising candidate tails for re-ranking in CKG. However, directly applying these methods to OpenKG fails to utilize knowledge from triples with similar semantics, where relevant known triples can provide helpful contextual information for predicting the missing tail. Taking Fig. 1(b) as an example, it will be easier to infer the tail NP of (Gates, be cochair of, ?) is "Bill &Melinda Gates Foundation" by sharing known triple knowledge from (Bill Gates, be the founder of, Microsoft) and (Bill Gates, ex-wife, Melinda).

To tackle these issues, we propose a CanonicalizationEnhanced Known Fact-Aware (CEKFA) framework, which improves OpenKG link prediction through sparsity reduction of RPs and triples. First, we propose a similarity-driven RP canonicalization method to collect semantically similar RPs for each RP as its canonical information. Second, we introduce a known fact-aware triple canonicalization method through retrieving known facts associated with the query from training data as its canonical triples. Finally, a general two-stage framework consisting of a canonicalizationenhanced encoder and a known fact-aware re-ranker is proposed to utilize these two kinds of canonical information for link prediction. The advantages of our approach are that the proposed canonicalization methods require neither predefined knowledge bases for OpenKG linking nor several external tools for generating various side information [Vashishth et al., 2018; Liu et al., 2021], and can be easily adapted to different query encoding models such as TransE [Bordes et al., 2013], ComplEx [Trouillon et al., 2016], ConvE [Dettmers et al., 2018] and so on. In summary, Our main contributions are:

• We propose a two-stage re-ranking framework named CEKFA to boost the performance of OpenKG link prediction through reducing the sparsity of RPs and triples.
• We develop novel and effective similarity-driven RP canonicalization and known fact-aware triple canonicalization methods for sparsity reduction.
• We achieve state-of-the-art results on two OpenKG datasets, ReVerb20K and ReVerb45K, and obtain different degrees of improvement when CEKFA is applied to five typical query encoding models. Abla-

tion studies and extensive analyses demonstrate the effectiveness and scalability of our framework. Source codes and datasets of this paper are available at https://github.com/wylResearch/CEKFA.

2 RelatedWork

2.1 OpenKGLink Prediction

Knowledge graph embedding (KGE) methods have been commonly explored for the link prediction problem in CKG, typically including distance-based models [Bordes et al., 2013; Lin et al., 2015; Sun et al., 2019], semantic matching models [Trouillon et al., 2016; Zhang et al., 2019], and neural network-based models [Dettmers et al., 2018; Schlichtkrull et al., 2018].

However, link prediction for OpenKG is a relatively underexplored area. CESI [Vashishth et al., 2018] focuses on obtaining multiple side information of NPs and RPs for embedding learning built on HolE [Nickel et al., 2016], so as to perform canonicalization task instead of link prediction. Based on ConvE [Dettmers et al., 2018], CaRe [Gupta et al., 2019] takes the canonicalization of NPs into account to enhance NP embeddings, and learns RP embeddings by GRU [Cho et al., 2014]. Instead of concentrating on canonicalization, OKGIT [Chandrahas and Talukdar, 2021] employs pre-trained BERT [Devlin et al., 2019] to improve the type compatibility of tail NPs. [Broscheit et al., 2020] provides an OpenKG benchmark called OLPBENCH, and employs ComplEx [Trouillon et al., 2016] to learn embeddings initialized by LSTM [Hochreiter and Schmidhuber, 1997]. OKGSE[Xie et al., 2022] learns cluster-segregated NP embeddings enhanced by neighboring contexts and type compatibility. Unlike the above methods, we focus on reducing the sparsity of RPs and triples through obtaining their canonical information to improve link prediction performance, which requires neither predefined relations for OpenKG linking nor additional side information for OpenKG canonicalization [Vashishth et al., 2018; Liu et al., 2021].

2.2 Re-ranking for Link Prediction

By exploiting textual information, re-ranking promising candidates through PLMs brings performance gains for link prediction in CKG [Kolluru et al., 2021; Lovelace et al., 2021]. CEAR [Kolluru et al., 2021] applies BERT [Devlin et al., 2019] as a re-ranker to score promising candidates together by concatenating textual forms of the query and all candidate tail entities into one input sentence. [Lovelace et al., 2021] develops a student network based on BERT [Devlin et al., 2019] with the textual concatenation of query and a candidate tail as input, which is distilled from their ranking model. Different from them, we not only utilize query along with tail candidates but also retrieve useful known facts related to the query from training set to assist in re-scoring, which is able to reduce the triple sparsity by sharing knowledge of similar triples and provide contextualized information.

2.3 Retrieving fromTraining Data

Lately, several methods show that explicitly retrieving from training data is useful for several NLP tasks [Gu et al., 2018; Khandelwal et al., 2020; Wang et al., 2022]. [Wang et al., 2022] develops a model pipeline that retrieves knowledge from training data for tasks such as summarization, machine translation, language modeling, and question answering, where the model input is concatenated with the retrieved content obtained by BM25 [Robertson and Zaragoza, 2009]. To the best of our knowledge, we are the first to retrieve training data for scoring promising candidates at the re-ranking stage for OpenKG link prediction.

3 CEKFA: The Proposed Framework

To solve the problems of failing to utilize the canonical information of RPs and triples for OpenKG link prediction, we propose the Canonicalization-Enhanced Known Fact-Aware (CEKFA) framework, shown in Fig. 2. First, we introduce two different canonicalization methods for sparsity reduction of RPs and triples in Sec. 3.2. To exploit the obtained canonical information, we design a canonicalization-enhanced encoder that utilizes canonicalization of both NPs and RPs to learn dense embeddings for scoring a triple in Sec. 3.3, and a known fact-aware re-ranker that adopts triple canonicalization to re-score promising candidates in Sec. 3.4.

3.1 ProblemFormulation

Let G = (E, R, F) be an OpenKG, where E is the set of noun phrases (NPs), R is the set of relation phrases (RPs), and F = {(eh, r, et)|eh ∈ E, r ∈ R, et ∈ E} is the set of facts. Here, eh and et are the head NP and tail NP in a factual triple. Each NP e ∈ E and RP r ∈ R is a sequence of tokens, denoted as xe = (xe1 , xe2 , . . . , xele ) and xr = (xr1 , xr2 , . . . , xrlr ), respectively, where le and lr are the numbers of tokens in NP e and RP r. Given a query (eh, r, ?), the link prediction task aims at inferring the tail NP et in missing fact (eh, r, et) by learning the triple score representing the holding probability.

3.2 Sparsity Reduction for RP and Triple

To reduce the sparsity of RPs and triples for OpenKG link prediction, two canonicalization methods, namely similaritydriven RP canonicalization and known fact-aware triple canonicalization, are proposed.

Similarity-driven RP Canonicalization

Prior methods like CaRe [Gupta et al., 2019] fail to explicitly canonicalize RPs, resulting in knowledge of similar RPs cannot be shared. Considering this, we propose to retrieve similar RPs for each RP as its canonical information, which does not require an existing knowledge base for OpenKG linking.

First, to obtain the semantic representation of RPs, each of them is constructed into an input sequence. Take RP rj as an example, its input sequence srj is set as "[ENT1] xrj [ENT2]", where "[ENT1]" and "[ENT2]" are two introduced tokens. Then, a pre-trained language model SBERT [Reimers and Gurevych, 2019] is employed to encode the input and obtain its dense sentence representation, denoted as function fSBERT . Finally, to get the canonical RPs for rj, a retrieval system Srp is adopted to match its representation against all other RPs in R and return the Top-Mrp most similar ones:

where Mrp is a hyperparameter and Nr(rj) is the canonical neighbor RPs of rj. The semantic similarity simrp j,u between rj and ru ∈ R is computed as the cosine similarity between their sentence representations:

where sru is the input sequence of ru.

Known Fact-aware Triple Canonicalization

Prior works ignore the sparsity reduction of triples in OpenKG, and they fail to utilize relevant known triples for providing useful hints. Triple canonicalization aims to retrieve these relevant triples with respect to the query, so as to deliver contextualized information for guiding the prediction of unseen queries. Motivated by [Wang et al., 2022], we propose to canonicalize a query by retrieving similarly-expressed known facts from training data.

First, we index all known factual triples in the training set to a dictionary, i.e., C = {(k, v)}, where the key k is the token concatenation of head NP and RP, and the value v is the token concatenation of head NP, RP, and tail NP. For instance, the k and v for a triple (eh, r, et) are expressed as:

where [] means concatenation, and xeh , xr , xet are tokens of eh, r and et, respectively.

For a query (eh, r, ?), we identify known facts with similar query as its canonical triples, so as to provide clues for predicting the missing tail et. We match the key of the query, written as kq = [xeh , xr ], against keys of all known facts, and return the Top-Mtriple most similar ones along with their values through a retrieval system Striple:

Striple(kq, C) = {(k1, v1), . . . , (kMtriple , vMtriple )} (4) where Mtriple is a hyperparameter. Striple is built based on SBERT [Reimers and Gurevych, 2019] as Srp (Eq. (1)), in which the similarity simtriple i,j between any two keys ki and kj can be formulated as follows:

At last, the canonical triples Ntriple(eh, r, ?) for (eh, r, ?) are set to the values in the retrieved results Striple(kq, C):

3.3 Canonicalization-Enhanced Encoder

The canonicalization-enhanced encoder aims to sufficiently exploit the canonical information of NPs and RPs to score a triple (eh, r, et), which consists of four modules: (1) pretrained semantic embedding, of which embeddings are initialized with rich semantics rather than at random; (2) NP canonicalization encoding, which updates NP embeddings utilizing NPs' canonical information; (3) RP canonicalization encoding, which applies RPs' canonical information to update RP embeddings; (4) query encoding, which captures the interaction of a query triple and obtains its score.

Pre-trained Semantic Embedding

We employ pre-trained language models to encode the textual forms of NPs and RPs as initialization, which can identify similar NPs/RPs and give meaningful representations for long-tailed ones. For each NP/RP, its textual sequence is sent to the pre-trained BERT [Devlin et al., 2019], and the output hidden states at the last layer are mean pooled to form the initialized embedding, denoted as function fBERT . Then, initializing NP ei ∈ E and RP rj ∈ R to get their embeddings ei ∈ Rd and rj ∈ Rd can be written as ei = fBERT (xei ) and rj = fBERT (xrj ), where d is the embedding dimension, xei and xrj are token sequences of ei and rj, respectively.

NP Canonicalization Encoding

In order to densify the OpenKG by leveraging NP canonicalization, we follow CaRe [Gupta et al., 2019] to employ a Local Averaging Network, which is a non-parametric message passing network, for updating NP embeddings. For each NP ei, let Ne(ei) be its canonical neighbor NPs, namely NPs belonging to the same entity as ei, which are first aggregated to obtain its canonical neighbor vector eni ∈ Rd as follows:

Then, ¯ ei ∈ Rd is obtained by averaging ei and the canonical neighbor vector eni , i.e., ¯ ei = (ei + eni )/2, which entails information about NPs linked to an entity.

RP Canonicalization Encoding

Taking one step further, we take into account the RP canonicalization Nr(rj) obtained in Eq. (1). The embedding of rj is updated in the same way as NPs, where its canonical neighbor vector rn j ∈ Rd of RP rj is first calculated fromNr(rj):

Then ¯ rj ∈ Rd is averaged from rj and rn j , i.e., ¯ rj = (rj + rn j )/2, so that it carries the knowledge of RPs similar to rj.

Query Encoding

Query encoding intends to encode a query triple (eh, r, et) to obtain its triple score ψ(eh, r, et), notated as the score function SF. We utilize the updated embeddings that are enhanced by canonicalization, i.e. ¯ eh, ¯ r, ¯ et, as inputs to SF:

Note that most KGE models could be leveraged to implement SF, such as distance-based models like TransE [Bordes et al., 2013], semantic matching models like ComplEx [Trouillon et al., 2016], and neural network-based models like ConvE [Dettmers et al., 2018]. For instance, the score function based on ConvE [Dettmers et al., 2018] is:

where g is a operation of reshaping embeddings into a 2D matrix, ω is convolutional filters, f is a non-linear function, vec is a operation of reshaping a tensor into a vector, and W is a linear transformation matrix.

Model Training and Inference

For training the canonicalization-enhanced encoder, we take the same loss function as the KGE method adopted in the query encoding module. Taking the encoder built on ConvE as an example, the logistic sigmoid function σ is first applied to triple score ψ to obtain its ranking probability p1:

Then the loss for triple (eh, r, et) is calculated following Binary Cross Entropy (BCE):

where y is the label of (eh, r, et), which is 1 if (eh, r, et) ∈ F and 0 otherwise.

During the inference stage, the Top-K potential candidate tail NPs to query (eh, r, ?) are obtained by calculating the scores of triples consisting of (eh, r, ?) and all NPs.

3.4 Known Fact-Aware Re-Ranker

Prior CKG link prediction methods perform re-ranking with the query and predicted candidates as inputs, but fail to utilize canonical information of triples. To reduce triple sparsity in OpenKG, the known fact-aware re-ranker is proposed to re-score candidates by additionally exploiting query-related canonical triples for providing explicit contextualized hints.

We employ DistilBERT [Sanh et al., 2019] followed by a classification head, which is a two-layer fully-connected neural network, to implement re-ranking, namely modelM. For query (eh, r, ?) and one of its candidate tail NPs et, they are concatenated into a query triple sentence xq,t, i.e., xq,t = [xeh , xr , xet ]. Instead of using the query triple sentence alone for prediction, we adopt canonical triples Ntriple(eh, r, ?) obtained in Eq. (6) to help re-score by concatenating them into a known factual sentence xv, i.e., xv = [v1; . . . ; vMtriple ]. Then it is combined with the query triple sentence xq,t as an input pair fed intoM:

where p2 is the re-ranking probability of triple (eh, r, et).

Model Training and Inference

For each query (eh, r, ?) in the training set, we use the ground-truth tail NPs to form the sentence pairs as positive samples, and extract the Top-Mneg negative candidate tail NPs predicted by canonicalization-enhanced encoder to form negative samples, where Mneg is a hyperparameter. During training, we filter out the query itself from the retrieved results. The known fact-aware re-ranker is fine-tuned by minimizing the BCE loss of p2 and its label as in Eq. (12).

At inference stage, we re-rank Top-K candidate tail NPs through ensembling scores produced from canonicalizationenhanced encoder and known fact-aware re-ranker. For query (eh, r, ?), the ranking probability p1 for one of its TopK candidate tail NPs et is first scaled to p′1 by the maximum ranking probability among the Top-K candidates:

where p(1:K) 1 is the ranking probabilities of the Top-K candidates for query (eh, r, ?). At last, the final score ˆ p(eh, r, et) for (eh, r, et) is merged as:

where α ∈ [0, 1] is a score composition weight.

4 Experimental Setup

4.1 Dataset

We evaluate our approach on ReVerb20K and ReVerb45K datasets following CaRe [Gupta et al., 2019], which are created through ReVerb Open KB [Fader et al., 2011]. Details of these datasets are shown in Tab. 1, where "Gold NP Clusters" is the gold canonical clusters of NPs extracted through Freebase entity linking information [Gabrilovich et al., 2013]. Inverse RPs are introduced for both datasets by adding the phrase "inverse of" to each RP. For more details, we refer readers to the related papers.

4.2 Experimental Settings

We use canonical clusters of NPs provided in CaRe [Gupta et al., 2019]. The pre-trained all-mpnet-base-v2 model is applied for fSBERT in sparsity reduction for RP and triple. The pre-trained BERT-base-uncased model is employed for fBERT in pre-trained semantic embedding. The pre-trained distilbert-base-uncased model is utilized to implement M in known fact-aware re-ranker. Note that we experiment with multiple pre-trained models and choose the above settings because of their good performance and high efficiency. We set the embedding dimension d = 768, and the number of reranking tail NPs K = 10. The value of Mrp, Mtriple, and Mneg are all selected from {1, 3, 5, 7, 10}. The optimal settings of hyperparameters for both datasets are listed in Tab. 2.

For training the canonicalization-enhanced encoder (Sec. 3.3), we use the self-adaptive optimization method Adam [Kingma and Ba, 2014]. Here we report the training settings of the best-performing CEKFA[ResNet] framework, which will be introduced later. Both datasets use a batch size of 128 and an initial learning rate of 0.0001 with a scheduler that reduces the learning rate by a decay factor of 0.5 when the value of MRR has stopped improving for 2 epochs. The model is trained for a maximum of 500 epochs, which would be terminated if the MRR metric of the validation set has not improved within 10 epochs.

After training the canonicalization-enhanced encoder (Sec. 3.3), we freeze its parameters and then fine-tune the known fact-aware re-ranker (Sec. 3.4) for a maximum of 5 epochs with a batch size of 16, where the best model is chosen based on the accuracy of validation set. We employ AdamW [Loshchilov and Hutter, 2017] optimizer using a linear decay learning rate scheduler with a weight decay of 0.01 and a linear warm-up with 10% of the training data, where the initial learning rate is 2×10-5 .

At inference stage, Top-10 candidate tail NPs predicted by the canonicalization-enhanced encoder are re-scored by known fact-aware re-ranker, and the score composition weight α is picked from [0,1] that achieves the best MRR on validation set with an increment of 0.1. All experiments are conducted in Pytorch and on one 16G Tesla V100 GPU.

4.3 Baselines

We implement CEKFA with five typical KGE methods in CKG, including TransE [Bordes et al., 2013], ComplEx [Trouillon et al., 2016], ConvE [Dettmers et al., 2018], RotatE [Sun et al., 2019], and PairRE [Chao et al., 2021], denoted as "CEKFA[*]". We also adopt ResNet [He et al., 2016] following [Lovelace et al., 2021] as the query encoding module to implement CEKFA, notated as "CEKFA[ResNet]".

We compare CEKFA with: (1) KGE methods in CKG: TransE [Bordes et al., 2013], ComplEx [Trouillon et al., 2016], R-GCN [Schlichtkrull et al., 2018], ConvE [Dettmers et al., 2018], KG-BERT [Yao et al., 2019], RotatE [Sun et al., 2019], SpacESS [Nayyeri et al., 2020], PairRE [Chao et al., 2021], the two-stage method [Lovelace et al., 2021] (denoted as BertResNet-ReRank), and the query encoding module in it (denoted as ResNet); (2) KGE methods in OpenKG: CaRe [Gupta et al., 2019], OKGIT [Chandrahas and Talukdar, 2021], and OKGSE [Xie et al., 2022].

4.4 EvaluationMetrics

We evaluate CEKFA on the link prediction task following CaRe [Gupta et al., 2019], which evaluates the rank of canonical clusters of NPs under filtered setting [Bordes et al., 2013]. Standard ranking metrics are utilized: Mean Rank (MR), Mean Reciprocal Rank (MRR), Hits at 1 (Hits@1), Hits at 3 (Hits@3), and Hits at 10 (Hits@10).

5 Experimental Results

5.1 Main Results

Link prediction results of CEKFA against various baselines are shown in Tab. 3. CEKFA[ResNet] outperforms all baselines on MRR, Hits@1, and Hits@3, indicating that it performs best on accurate metrics. Compared to methods in OpenKG (the second field in Tab. 3), CEKFA[ResNet] achieves absolute improvements of 1.5% and 2.7% for MRR, 1.9% and 2.0% for Hits@1 on ReVerb20K and ReVerb45K respectively, which confirms the effectiveness of canonicalizing RPs and triples for OpenKG link prediction.

Unsurprisingly, most KGE methods in CKG show relatively poor performance when applied directly to OpenKG (the first field in Tab. 3), with the best MRR and Hits@1 on ReVerb20K and ReVerb45K implemented by BertResNetReRank and ConvE, respectively. Notably, KG-BERT performs moderately in MRR and Hits@1, but well in MR metric, with a similar phenomenon observed when applied to CKG [Wang et al., 2021]. This is mainly because KG-BERT only encodes the textual knowledge of triples and neglects to learn the structural information, which makes it good at selecting NPs/entities with the right semantics as candidates, but unable to accurately predict the target owing to ambiguity problem. Therefore, when N is slightly large, it achieves good Hits@N (i.e., Top-N recall), but performs poorly for small N, resulting in poor MRR and good MR.

Comparing the first and third fields in Tab. 3, it is clear that the results of KGE models are improved by different degrees after being integrated with CEKFA. Interestingly, after applying CEKFA, ResNet obtains greater improvements than ConvE and achieves the best results, with absolute improvements of 16.3% and 18.8% for MRR, 12.2% and 14.4% for Hits@1 on ReVerb20K and ReVerb45K respectively. This indicates the effectiveness of employing deeper convolutional networks to capture the interactions between dense embeddings with rich semantics. Moreover, CEKFA obtains up to 29.0% and 23.5% absolute improvements for MRR metric on ReVerb20K and ReVerb45K (CEKFA[ComplEx]). Even with the smallest boost, CEKFA obtains 12.4% (CEKFA[ConvE]) and 6.3% (CEKFA[TransE]) in MRR on these two datasets respectively, suggesting the effectiveness and generalization ability of the proposed framework.

5.2 Ablation and Variants Study

Ablation study of the proposed canonicalization methods in CEKFA is presented in Tab. 4. We find that: (1) For CEKFA[ResNet], removing the known fact-aware re-ranker (-KFARe) decreases the MRR by 1.5% on both datasets. And for CEKFA[*] constructed from different KGE methods, the removal of KFARe leads to maximum decreases of 3.1% and 3.8% for MRR, 4.1% and 5.5% for Hit@1 on ReVerb20K and ReVerb45K respectively, both of which are built on TransE. It illustrates the crucial role of triple canonicalization to share knowledge of related triples. (2) For CEKFA[ResNet]-KFARe, removing RP canonicalization encoding (-RPCE) further reduces the MRR by 1.0% and 2.1% on ReVerb20K and ReVerb45K. For CEKFA[*]-KFARe, the absence of RPCE leads to maximum reductions of 5.1% and 2.1% for MRR, 4.9% and 2.2% for Hit@1 on ReVerb20K and ReVerb45K, which are integrated with TransE and ResNet, respectively. This implies the effectiveness of RP canonicalization in sharing knowledge of similar RPs to reduce their high sparsity. Overall, the results show that leveraging RP

and triple canonicalization helps to reduce their sparsity and enhance the performance of CKG methods.

In addition, to investigate the effectiveness of known factaware re-ranker, two variants are further developed. From the results shown at the bottom of Tab. 4, we can observe that: (1) Re-ranking by query triple sentence xq,t without known factual sentence xv (CEKFA[ResNet] TripleOnlyReranking) gives no gains, and even results in a drop of 0.1%∼0.2% in MRR on both datasets. It indicates that known facts composed of canonical triples are indispensable to the triple re-ranking stage which performs triple canonicalization and provides contextualized hints. (2) Retrieving relevant known facts by BM25 (CEKFA[ResNet] BM25Retrieving) that learns sparse representations performs worse than by SBERT that learns dense representations. It is primarily owing to the ability of SBERT to better handle complex sentences with the same semantics but different expressions.

5.3 Sparsity Analyses

Sparsity of RPs

To intuitively understand the sparsity reduction of RPs, we compare the canonical neighbors of several randomly chosen RPs obtained from CEKFA[ResNet] with CESI [Vashishth et al., 2018], which performs OpenKG canonicalization task. Results are displayed in Tab. 5, where only the Top-3 canonical neighbor RPs are reported for each RP. It can be seen that RP canonicalization obtained from CEKFA[ResNet] is semantically similar, rather than only superficially similar, e.g., the canonical neighbor RPs obtained for "was an inmate at". Besides, we obtain specific canonicalization for each RP instead of clustering them, which is flexible and highly faulttolerant. For example, in CESI, "was an engineer at" is incorrectly clustered with "was an inmate at", resulting in all RPs in this cluster having wrong canonicalization information.

Furthermore, for comparison of the learned RP embeddings with and without canonicalization, we project the highdimensional embedding to a 2-dimensional space using tSNE [Van der Maaten and Hinton, 2008] for visualization. Owing to the lack of gold canonicalization of RPs, we randomly select six RPs with different semantics in experiments, and for each RP we manually choose two more RPs with the same semantics. Fig. 3 displays the results of CEKFA[ResNet] compared against CaRe [Gupta et al., 2019] and OKGIT [Chandrahas and Talukdar, 2021], where RP canonicalization is not considered in CaRe and OKGIT. It can be seen that RPs with the same semantic are well clustered by CEKFA[ResNet]. Besides, CEKFA[ResNet] can distinguish RPs that have similar tokens but different semantics, e.g., "split from" and "is basically split into".

Sparsity of Triples

To visualize the effect of triple sparsity reduction for predicting NPs with different sparse levels, Fig. 4 shows the improvements brought by the known fact-aware re-ranker based on the in-degree of tail NPs in test triples. We find that the performance across all sparse levels of tail NPs can be improved by re-ranking with known facts to reduce the sparsity of query triples. Noticeably, re-ranking with triple canonicalization brings a significant performance boost of 23.10% in ReVerb45K when tail NPs are "extremely sparse", i.e., their in-degree in the training data is 0. It indicates that canonical triples can provide query-related contextualized hints, especially in predicting unseen NPs.

To analyze the improvements to known fact-aware reranker brought by known factual sentences consisting of canonical triples, we present in Tab. 6 the Top-3 candidate tail NPs predicted by CEKFA[ResNet] before and after reranking for several query pairs, as well as the Top-3 retrieved canonical triples. It can be found that known facts in the training data related or similar to the query pair can be retrieved as canonicalization, even if their tokens are superficially less similar, e.g., "simon was growing to be simon peter" can be retrieved for query (simon, inverse of was called, ?). Moreover, better inferences can be made with the support of canonical triples. For instance, "phil gramm is also an officer of ub" provides contextual information for query (phil gramm, went to work for, ?), which leads to correct prediction.

5.4 Time Complexity

We analyze the additional time complexity of CEKFA compared to baselines during the inference process in each stage. For canonicalization-enhanced encoder, the increased time complexity is O(T(Mnp +Mrp)), coming from the canonicalization encoding of NP and RP in each triple, where T is the number of triples, Mnp and Mrp are the average number of canonical neighbors for each NP and RP (Mnp < 2, Mrp ≤ 10). As for the known fact-aware re-ranker, the time complexity is O(TK), stemming from encoding each query with one of its top-K candidate tails.

Tab. 7 compares the inference times of three baseline methods before and after applying the canonicalizationenhanced encoder, as well as the relative changes in time and performance. It can be seen that the canonicalizationenhanced encoder built on baseline method without reranking stage (CEKFA[*]-KFARe) achieves performance gains by 12.37%∼276.47%, which only decreases the inference time by 4.67%∼11.72%. Although the time cost increases, which comes from the canonicalization encoding of NP and RP, it is much less than the performance benefits brought by the canonicalization-enhanced encoder. As for known fact-aware re-ranker (KFARe), we compare its efficiency in CEKFA[ResNet] with the re-ranking stage of BertResNet-ReRank [Lovelace et al., 2021], where the number of samples predicted per second on ReVerb45K is 61 and 151, respectively. The relatively slow re-ranking time in CEKFA is reasonable since we include the canonical triples as part of the input, which increases the encoding time.

6 Conclusion

In this paper, we propose a Canonicalization-Enhanced Known Fact-Aware (CEKFA) framework for OpenKG link prediction by reducing the sparsity of RPs and triples. For RP sparsity reduction, a similarity-driven RP canonicalization method is introduced to collect semantically similar RPs as canonical information. To reduce the sparsity of triples, a known fact-aware triple canonicalization method is developed by retrieving query-related known facts to provide contextualized hints. These two kinds of canonical knowledge are integrated into the canonicalization-enhanced encoder and known fact-aware re-ranker for link prediction. Experiments show that CEKFA outperforms previous baselines on two OpenKG datasets, ReVerb20K and ReVerb45K, and extensive analyses suggest the effectiveness and generalization ability of the proposed approach.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No. 62006243, Grant No. 62025208, and Grant No. 61932001), and the Xiangjiang Laboratory Foundation (Grant No. 22XJ01012).