Improving Inductive Link Prediction Using Hyper-Relational Facts (Extended Abstract)

1 Introduction

Knowledge graphs are known for their incompleteness [Nickel et al., 2016], therefore predicting missing links is one of the most important applications of machine learning over KGs [Nickel et al., 2011; Bordes et al., 2013]. A flurry [Ali et al., 2021b; Ji et al., 2020] of approaches has been developed over the years. Most of them operate over triple-based KGs in the transductive setup, where all entities are known at training time. Such approaches can neither operate on unseen entities, which might emerge after updating the graph, nor on new (sub-)graphs comprised of completely new entities. Those scenarios are often unified under the inductive link prediction (LP) setup. A variety of NLP tasks building upon KGs have inductive nature, for instance, entity linking or information extraction. Hence, being able to work in inductive settings becomes crucial for KG representation learning algorithms. For instance (see Fig. 1 in [Ali et al., 2021a]), the directorgenre pattern from the seen graph allows to predict a missing genre link for The Martian in the unseen subgraph.

∗Published at International Semantic Web Conference 21 - Received Best Research Paper Award

† equal contribution

Several recent works [Teru et al., 2020; Daza et al., 2021] investigate inductive LP but usually focus on a custom inductive setting. Further, their underlying KG structure is based on triples only. On the other hand, new, more expressive KGs like Wikidata [Vrandecic and Kr¨otzsch, 2014] exhibit a hyper-relational nature where each triple (a typed edge in a graph) can be further instantiated with a set of explicit relationentity pairs, known as qualifiers in the Wikidata model. Recently, it was shown [Galkin et al., 2020] that employing hyperrelational KGs yields significant gains in the transductive LP task compared to their triple-only counterparts. However, the effect of such KGs on inductive LP is unclear. Intuitively (Fig. 1 in [Ali et al., 2021a]), the (nominee: Matt Damon) qualifier provides a helpful signal to predict Best Actor as an object of nominated for of The Martian given that Good Will Hunting received such an award with the same nominee.

Therefore, in our ISWC 2021 paper [Ali et al., 2021a], we systematically study hyper-relational KGs in different inductive settings:

• We propose a classification of inductive LP scenarios that describes the settings formally and, to the best of our knowledge, integrates all relevant existing works. Specifically, we distinguish fully-inductive scenarios, where target links are to be predicted in a new subgraph of unseen entities, and semi-inductive ones where unseen nodes have to be connected to a known graph.
• We adapt two existing baseline models for the two inductive LP tasks probing them in the hyper-relational settings.
• Our experiments suggest that models supporting hyperrelational facts indeed improve link prediction in both inductive settings compared to strong triple-only baselines by more than 6% Hits@10.

2 Background

We assume the reader to be familiar with the standard link prediction setting (e.g. from [Nickel et al., 2011]) and introduce the specifics of the setting with qualifiers.

Let G = (E,R, S) be a hyper-relational KG where E is a set of entities, R is a set of relations, and S a set of statements. Each statement can be formalized as a 4-tuple (h, r, t, q) of a head and tail entity 1 h, t ∈ E, a relation r ∈ R, and a set of qualifiers, which are relation-entity pairs q ⊆ P(R × E) where P denotes the power set. For example, see Fig. 1 in [Ali et al., 2021a] contains a statement (Good Will Hunting, nominated for, Best Actor, {(nominee, Matt Damon)}) where (nominee, Matt Damon) is a qualifier pair for the main triple. We define the set of all possible statements as set

with a set of relations R, a set of head, tail and qualifier entities EH, ET , EQ ⊆ E. Strain is the set of training statements and Seval are evaluation statements. We assume that we have a feature vector xe ∈ Rd associated with each entity e ∈ E. Such feature vectors can, for instance, be obtained from entity descriptions available in some KGs or represent topological features [Belkin and Niyogi, 2001].

In this work, we focus on the setting with one fixed set of known relations. That is, we do not require xr ∈ Rd features for relations but learn relation embeddings during training.

3 Inductive Link Prediction

Recent works have pointed out the practical relevance of different inductive LP scenarios. However, there exists a terminology gap as different authors employ different names for describing conceptually the same task or, conversely, use the same inductive LP term for practically different setups. We propose a unified framework that provides an overview of the area and describes the settings formally.

Let E· denote the set of entities occurring in the training statements Strain at any position (head, tail, or qualifier), and E◦ ⊆ E \E· denote a set of unseen entities. In the transductive setting, all entities in the evaluation statements are seen during training, i.e., Seval ⊆ S(E·,R, E·, E·). In contrast, in inductive settings, Seval, used in validation and testing, may contain unseen entities. In order to be able to learn representations for these entities at inference time, inductive approaches may consider an additional set Sinf of inference statements about (un)seen entities; of course Sinf ∩ Seval = ∅.

The fully-inductive setting (FI) is akin to transfer learning where link prediction is performed over a set of entities not seen before, i.e., Seval ⊆ S(E◦,R, E◦, E◦). This is made possible by providing an auxiliary inference graph Sinf ⊆ S(E◦,R, E◦, E◦) containing statements about the unseen entities in Seval. For instance, in Fig. 1 in [Ali et al., 2021a], the training graph is comprised of entities Matt Damon, Good Will Hunting, Best Actor, Gus Van Sant, Milk, Drama. The inference graph contains new entities The Martian, Alien, Ridley Scott, Blade Runner, Sci-fi with one missing link to be predicted. The fully-inductive setting is considered in [Teru et al., 2020; Daza et al., 2021].

1 We use entity and node interchangeably

In the semi-inductive setting (SI), new, unseen entities are to be connected to seen entities, i.e., Seval ⊆ S(E·, R, E◦, E·) ∪ S(E◦,R, E·, E·). Illustrating with Fig. 1 in [Ali et al., 2021a], The Martian as the only unseen entity connecting to the seen graph, the semi-inductive statement connects The Martian to the seen Best Actor. Note that there are other practically relevant examples beyond KGs, such as predicting interaction links between a new drug and a graph containing existing proteins/drugs [Bagherian et al., 2020; Gaudelet et al., 2020]. We hypothesize that, in most scenarios, we are not given any additional information about the new entity, and thus have Sinf = ∅; we will focus on this case in this paper. However, the variation where Sinf may contain k statements connecting the unseen entity to seen ones has been considered too [Albooyeh et al., 2020; Bhowmik and de Melo, 2020; Clouˆatre et al., 2021] and is known as k-shot learning scenario.

A mix of the fully- and semi-inductive settings where evaluation statements may contain two instead of just one unseen entity is studied in [Daza et al., 2021; Baek et al., 2020; Wang et al., 2020]. That is, unseen entities might be connected to the seen graph, i.e., Seval may contain seen entities, and, at the same time, the unseen entities might be connected to each other; i.e, Sinf ̸= ∅.

Our framework is general enough to allow Seval to contain new, unseen relations r having their features xr at hand. Since research so far has focused on the setting where all relations are seen in training, we will do so, too.

We hypothesize that qualifiers, being explicit attributes over typed edges, provide a strong inductive bias for LP tasks. In this work, for simplicity, we require both qualifier relations and entities to be seen in the training graph, i.e., EQ ⊆ E· and RQ ⊆ R, although the framework accommodates a more general case of unseen qualifiers given their respective features.

4 Inductive Link Prediction with Qualifiers

Both semi- and fully-inductive tasks assume node features to be given. Recall that relation embeddings are learned and, often, to reduce the computational complexity, their dimensionality is smaller than that of node features.

Encoders. In the semi-inductive setting, an unseen entity arrives without any graph structure pointing to existing entities, i.e., Sinf = ∅. This fact renders message passing approaches [Gilmer et al., 2017] less applicable, so we resort to a simple linear layer to project all entity features (including those of qualifiers) into the relation space: ϕ : Rdf → Rdr .

In the fully inductive setting, we are given a non-empty inference graph Sinf ̸= ∅, and we probe two encoders: (i) the same linear projection of features as in the semi-inductive scenario which does not consider the graph structure; (ii) GNNs which can naturally work in the inductive settings [Chami et al., 2020]. However, the majority of existing GNN encoders for multi-relational KGs like CompGCN [Vashishth et al., 2020] are limited to only triple KG representation. To the best of our knowledge, only the recently proposed STARE [Galkin et al., 2020] encoder supports hyper-relational KGs which we take as a basis for our inductive model. Its aggregation formula is:

where γ is a function that infuses the vector of aggregated qualifiers xq into the vector of the main relation xr. The output of the GNN contains updated node and relation features based on the adjacency matrix A and qualifiers Q:

Finally, in both inductive settings, we linearize an input statement in a sequence using a padding index where necessary: [x′h , x′r , x′qr 1 , x′qe 1 , [PAD], . . .]. Note that statements can greatly vary in length depending on the amount of qualifier pairs, and padding mitigates this issue.

Decoder. Given an encoded sequence, we use the same Transformer-based decoder for all settings:

In this work, we evaluated several aggregation strategies and found a simple mean pooling over all non-padded sequence elements to be preferable.

Here and below, we denote the linear encoder + Transformer decoder model as QBLP (that is, Qualifier-aware BLP, an extension of BLP [Daza et al., 2021]), and the STARE encoder + Transformer decoder, as STARE.

In order to compare results with triple-only approaches, we trained the models, as usual, on subject and object prediction.

5 Experiments

We design our experiments to investigate whether the incorporation of qualifiers improves inductive link prediction. In particular, we investigate the fully-inductive setting (Section 5.1) and the semi-inductive setting (Section 5.2). We analyze the impact of the qualifier ratio (i.e., the number of statements with qualifiers) and the dataset's size on a model's performance.

5.1 Fully-Inductive Setting

In the full inductive setting, we analyzed the effect of qualifiers for four different datasets (i.e., WD20K (100) V1 & V2 and WD20K (66) V1 & V2, which have different ratios of qualifying statements, the numbers in brackets, and are of different sizes). As triple-only baselines, we evaluated CompGCN [Vashishth et al., 2020] and BLP [Daza et al., 2021]. To evaluate the effect of qualifiers on the fullyinductive LP task, we evaluated StarE [Galkin et al., 2020] and QBLP. It should be noted that StarE without the use of qualifiers is equivalent to CompGCN.

The main findings (from all experiments) are that (i) for all datasets, the use of qualifiers leads to increased performance, and (ii) the ratio of statements with qualifiers and the size of the dataset has a major impact on the performance. CompGCN and StarE apply message-passing to obtain enriched entity representations while BLP and QBLP only apply a linear transformation. Consequently, CompGCN and StarE require Sinf to contain useful information in order to obtain the entity representations while BLP and QBLP are independent of Sinf. Table 1 shows the results obtained for two of the four datasets.

We observe that the performance gap between BLP/QBLP (0) and QBLP (2,4,6) is considerably larger than the gap between CompGCN and StarE. This might be explained by the fact that QBLP does not take into account the graph structure provided by Sinf, hence is heavily dependent on additional information, i.e., the qualifiers compensate for the missing graph information. The overall performance decrease observable between V1 and V2 could be explained by the datasets' composition, in particular, in the composition of the training and inference graphs: Sinf of V2 comprises more entities than V1, so that each test triple is ranked against more entities, i.e., the ranking becomes more difficult. At the same time, the training graph of V1 is larger than that of V2, i.e., during training more entities are seen which may improve generalization.

5.2 Semi-inductive Setting

In the SI setting, we evaluated BLP as a triple-only baseline and QBLP as a statement baseline (i.e., involving qualifiers) on the WD20K SI datasets. We did not evaluate CompGCN and StarE since message-passing-based approaches are not directly applicable in the absence of Sinf. The results highlight that aggregating qualifier information improves the prediction of semi-inductive links despite the fact that the ratio of statements with qualifiers is not very large (37% for SI WD20K (33), and 30% for SI WD20K (25)). In the case of SI WD20K (33), the baselines are outperformed even by a large margin. Overall, the results might indicate that in semi-inductive settings, performance improvements can already be obtained with a decent amount of statements with qualifiers.

5.3 Qualitative Analysis

We obtain deeper insights on the impact of qualifiers by analyzing the StarE model on the fully-inductive WD20K (66) V2 dataset. In particular, we study individual ranks for head/tail prediction of statements with and without qualifiers (see Fig. 2 in [Ali et al., 2021a]) varying the model from zero to four pairs. First, we group the test statements by the number of available qualifier pairs. We observe generally smaller ranks which, in turn, correspond to better predictions when more qualifier pairs are available. In particular, just one qualifier pair is enough to significantly reduce the individual ranks. Note that we have less statements with many qualifiers.

We also study how particular qualifiers affect ranking and predictions (see Fig. 3 in [Ali et al., 2021a]). For that, we measure ranks of predictions for distinct statements in the test set with and without masking the qualifier relation from the inference graph Sinf . We then compute ∆MR and group them by used qualifier relations. Interestingly, certain qualifiers, e.g., convicted of or including, may deteriorate performance, which we attribute to the usage of rare, qualifier-only entities. Nevertheless, other qualifiers largely reduce ranks and hence very positively impact prediction accuracy.

Finally, we study the average impact of qualifiers on the whole graph, i.e., we take the whole inference graph and mask out all qualifier pairs containing one relation and compare the overall evaluation result on the test set, we count ranks of all test statements, not only those which have that particular qualifier) against the non-masked version of the same graph. We then sort relations by ∆MR and find top 3 most confusing and most helpful relations across two datasets (cf. Table 3). On the smaller WD20K (100) V1 where all statements have at least one qualifier pair, most relations tend to improve MR. For instance, qualifiers with the distributed by relations reduce MR by about 29 points. On the larger WD20K (66) V2 some qualifier relations, e.g., statement is subject of, tend to introduce more noise and worsen MR which we attribute to the increased sparsity of the graph given an already rare qualifier entity. That is, such rare entities might not benefit enough from message passing.

6 Conclusion

In our ISWC 2021 paper [Ali et al., 2021a], we present a study of the inductive link prediction problem over hyper-relational KGs. In particular, we propose a theoretical framework to categorize various LP tasks to alleviate an existing terminology discrepancy pivoting on two settings, semi- and fully-inductive LP. Probing statement-aware models against triple-only baselines, we demonstrated that hyper-relational facts considerably improve LP performance in both inductive. Moreover, our qualitative analysis show that the achieved gains are consistent across different setups and still interpretable.

Our findings open up interesting prospects for employing inductive LP and hyper-relational KGs along several axes, e.g., large-scale KGs of billions statements, new application domains including life sciences, drug discovery, and KG-based NLP applications like question answering or entity linking.

Acknowledgements

This work was funded by the German Federal Ministry of Education and Research (BMBF) under Grant No. 01IS18036A and Grant No. 01IS18050D (project "MLWin"). The authors of this work take full responsibilities for its content.