Fast and Continual Knowledge Graph Embedding via Incremental LoRA

1 Introduction

Knowledge graph embedding (KGE) [Wang et al., 2017; Rossi et al., 2021] aims to embed entities and relations in knowledge graphs (KGs) [Dong et al., 2014] into vectors, which is crucial for various downstream applications, such as question answering [Bordes et al., 2014], recommendation systems [Zhang et al., 2016], fact detecting [Shang et al., 2024], and information extraction [Li et al., 2022; Xu et al., 2023; Li et al., 2024]. Traditional KGE methods [Bordes et al., 2013; Sun et al., 2019; Liu et al., 2020; Pan and Wang, 2021; Shang et al., 2023; Liu et al., 2023] primarily focus on dealing with static KGs. However, realworld KGs, such as Wikidata [Vrandeˇci´c and Kr ¨otzsch, 2014] and YAGO [Suchanek et al., 2007], are dynamic and continuously evolving with emerging knowledge. For instance, Wikidata expanded from 16M entities to 46M between 2014 and 2018 [Wikidata, 2023]. A challenge with traditional KGE is that updating the embeddings of entities and relations requires retraining the entire KG, leading to heavy training costs in large-scale KGs. To address this issue, the continual knowledge graph embedding (CKGE) task has received growing attention to fine-tuning with only new knowledge [Song and Park, 2018; Daruna et al., 2021].

∗Corresponding author.

The primary challenge in CKGE lies in alleviating catastrophic forgetting [Kirkpatrick et al., 2017; Liu et al., 2024] while simultaneously reducing training costs. One solution for CKGE, known as the full-parameter fine-tuning paradigm, memorizes old knowledge by replaying a core old data set [Lopez-Paz and Ranzato, 2017; Wang et al., 2019; Kou et al., 2020] or introducing additional regularization constraints [Zenke et al., 2017; Kirkpatrick et al., 2017; Cui et al., 2023]. Although this paradigm effectively mitigates catastrophic forgetting, it significantly increases training costs, especially when handling large-scale KGs. Another solution adopts the incremental-parameter fine-tuning paradigm, with only a few parameters to learn emerging knowledge [Rusu et al., 2016; Lomonaco and Maltoni, 2017]. Despite eliminating explicit knowledge replay, the straightforward alignment of new and old parameter dimensions may still result in an unacceptable increase in parameters and training time. Meanwhile, in recent years, to efficiently reduce the training time of large language models (LLMs) [Radford et al., 2019; Brown et al., 2020] for downstream tasks, some work has employed low-rank adapters, such as LoRA [Hu et al., 2022], to lower the parameter dimension and enable efficient parameter fine-tuning. Namely, LoRA freezes the pre-trained model weights and injects trainable rank decomposition matrices into original model architectures. In this paper, we try to innovatively adapt this mechanism to address continual learning problems, i.e., CKGE.

Inspired by low-rank adaptation [Hu et al., 2022] in parameter fine-tuning for LLMs, we are among the first to store new knowledge in KGs via low-rank adapters (LoRAs) to reduce training costs. As shown in Figure 1, in contrast to the old entity Ron Weasley, new incremental LoRAs should be assigned to the new entity Remus Lupin. Moreover, recent work has discovered that new parameters of different layers should be adaptively assigned distinct dimensions [Ansuini et al., 2019], which can also be applied to emerging knowledge with different structural features in CKGE. As depicted in Figure 1, Remus Lupin connects with more entities than Lord Voldemort. Consequently, it retains a broader influence, requiring more parameters for effective learning. In light of this consideration, we propose a fast CKGE framework (FastKGE), incorporating a novel incremental low-rank adapter (IncLoRA) mechanism, to learn new knowledge both effectively and efficiently. To alleviate catastrophic forgetting, FastKGE isolates new knowledge to specific layers. Concretely, FastKGE sorts and divides the new entity representations into explicit layers according to their distance from the old KG and the degree centrality. To reduce training costs, IncLoRA embeds the specific layers into incremental low-rank adapters with fewer training parameters. Specifically, the rank scales of each layer are allocated adaptively with the awareness of the KG structure.

To make up for the deficiency of small initial KG size in current CKGE datasets [Hamaguchi et al., 2017; Kou et al., 2020; Daruna et al., 2021; Cui et al., 2023], we construct two new CKGE datasets: FB-CKGE and WNCKGE based on FB15K-237 [Dettmers et al., 2018] and WN18RR [Toutanova et al., 2015]. FB-CKGE and WN-CKGE allocate 60% of total triples to generate the larger initial KG. Results on four traditional datasets show that FastKGE reduces training time by 34%-49% while still achieving competitive MRR scores in link prediction tasks against SOTAs (average in 21.0% vs. 21.1%). Meanwhile, on the new datasets FB-CKGE and WN-CKGE, FastKGE reduces 51%- 68% training time while improving link prediction performance by 1.5% in MRR on average.

To sum up, the contributions of this paper are three-fold:

• To the best of our knowledge, we are among the first to introduce low-rank adapters to CKGE, namely, emerging knowledge can be stored in low-rank adapters to reduce training costs and preserve old knowledge well.
• We devise a fast CKGE framework (FastKGE), which isolates knowledge into specific layers to alleviate catastrophic forgetting and utilizes an incremental low-rank adapter (IncLoRA) mechanism to reduce training costs.
• Experimental results demonstrate that FastKGE significantly reduces training time with competitive performance in link prediction tasks compared with SOTAs. Additionally, two new open CKGE datasets with largescale initial KGs are released.

2 RelatedWork

In contrast to traditional knowledge graph embedding (KGE) approaches [Bordes et al., 2013; Trouillon et al., 2016; Kazemi and Poole, 2018], continual knowledge graph embedding (CKGE) methods [Song and Park, 2018; Daruna et al., 2021] enable KGE models to acquire new knowledge while retaining previously learned knowledge. Existing CKGE methods can be categorized into two main groups. First, full-parameter fine-tuning methods preserve learned knowledge by replaying old data [Lopez-Paz and Ranzato, 2017; Wang et al., 2019; Kou et al., 2020], or introducing constraints on weight updates in neural networks [Zenke et al., 2017; Kirkpatrick et al., 2017; Cui et al., 2023]. Second, incremental-parameter fine-tuning methods [Rusu et al., 2016; Lomonaco and Maltoni, 2017] adaptively adjust architectural properties to accommodate new information while preserving old parameters, thus facilitating continual learning. However, these methods only focus on preserving knowledge while ignoring training efficiency when KGs evolve.

In the field of large language models (LLMs), some work tries to utilize low-rank adapters (LoRAs) to fine-tune LLMs efficiently [Hu et al., 2022; Zhang et al., 2023]. Based on it, recent work tries to utilize LoRAs to separately store knowledge to alleviate catastrophic forgetting in continual learning [Wang et al., 2023]. In the field of KGE, recent work tries to learn a small set of reserved entities to represent all entities for parameter-efficient learning [Chen et al., 2023]. However, few works focus on efficient fine-tuning for CKGE.

3 Methodology

3.1 Preliminary and Problem Statement

Growing Knowledge Graph. A growing knowledge graph (KG) is denoted as snapshot sequences, i.e., G = {S0, S1, ..., Sn}. Snapshot Si is a triplet (Ei,Ri, Ti), where Ei, Ri and Ti denote the set of entities, relations, and triples at time i, respectively. Furthermore, we denote ∆Ti = Ti - Ti-1, ∆Ei = Ei - Ei-1 and ∆Ri = Ri - Ri-1 as new triples, entities, and relations, respectively.

Continual Knowledge Graph Embedding. Continual knowledge graph embedding (CKGE) aims to embed entities and relations into vectors in the growing KG G = {S0, S1, ..., Sn}. Specifically, when new triples ∆Ti emerge in time i, CKGE learns the representations of new entities ∆Ei and relations ∆Ri, and updates the representations of old entities Ei-1 and relations Ri-1 to adapt ∆Ti. Finally, all representations of entities Ei and relations Ri are obtained.

3.2 Framework

The framework of FastKGE is illustrated in Figure 2. Overall, as a KG grows in each snapshot, we utilize incremental lowrank adapters (LoRAs) for different KG layers to learn and preserve new entities and relations. First, during the knowledge graph layering stage, we divide new entities and relations into several layers based on their distances from the old graph and node degrees. Second, in the IncLoRA learning stage, embeddings of entities and relations in each layer are represented by incremental LoRAs with adaptive rank allocation. Finally, in the link predicting stage, we compose all new LoRAs into a LoRA group and concat all LoRA groups and initial embeddings for inference.

3.3 Graph Layering

In order to achieve separate storage of KGs and assign different ranks to LoRAs of different layers, we divide new knowledge into several layers according to a graph structure. For new triples ∆Ti emerging in a new snapshot i > 0, we firstly get new entities ∆Ei and relations ∆Ri and initiate embeddings of them as shown in Stage 1 of Figure 2. Then, to sort and divide embeddings of ∆Ei in snapshot i, we calculate the importance of ∆Ei by the distance from the old graph and degree centrality. Specifically, we use the breadth-first search (BFS) algorithm to progressively expand ∆Ei in snapshot i from Si-1. Then, we get the sorted entity sequence sentity as follows:

where for ej, ek ∈ sentity, if j ≤ k, the distance of ej from the old graph is closer than ek. To further sort entities with the same distance, we denote fdc(e) as the degree centrality of e in the new graph composed of new triples ∆Ti as follows:

where fneighbor(e) denotes the number of the neighbors of e in ∆Ti. For entities with the same distance from the old graph in sentity, we use fdc for further sorting. Then, we divide the entities in sentitiy, previously sorted with importance, equally into N distinct layers E = {E1,E2, ...,EN}, where Ek denotes the k-th layer of entities, and N is a hyper-parameter. For relations, we only put all new relations into a layer R rather than layering. That is because the number of entities is increasing more significantly than relations in the dynamic evolution of KGs, and the number of total embedding parameters linearly to the number of entities in practice [Chen et al., 2023]. Therefore, we focus on storage and training optimization for new entities. Finally, we get the layering entities and relations E and R.

3.4 IncLoRA Learning

Incremental Low-Rank Decomposition

To accelerate learning, we propose an incremental low-rank adapter learning mechanism (IncLoRA) to reduce training costs. Specifically, we obtain entity and relation layers E and R from graph layering stage. Then, for each layer in E and R, we use an incremental low-rank adapter to learn and store knowledge. Take E as example, the embeddings of k-th layer Ek in E can be denoted as Ek. For learning the embeddings Ek ∈ Rn×d , we learn matrices Ak ∈ Rn×r and Bk ∈ Rr×d , making it meet the following condition:

where n denotes the number of entities in a layer Ek, d stands for embedding dimension and r refers to the rank of (Ak,Bk). We denote (Ak,Bk) as an incremental LoRA. To make sure that parameters of low-rank learning is fewer than normal training for faster learning, the decomposition should meet the following condition:

By this way, we compose Ek to an incremental LoRA (Ak,Bk). Finally, we get a LoRA group GEi for all embeddings of ∆Ei as follows:

where concat(·) denotes the concatenation of several matrices. We can also get the LoRA group GRi for new relations by the same decomposition.

Adaptive Rank Allocation

In order to preserve more information for more important entities, we utilize an adaptive rank allocation strategy to assign different ranks to LoRAs in different layers. Specifically, instead of assigning a fixed base rank rbase to all incremental LoRAs in GEi , we assign more ranks to more important lowrank LoRAs with higher fdc. Firstly, we denote the total sum of degree centrality in the k-th layer Sumk and the average degree centrality Avgdc in all layers as follows:

Then, the rk of the k-th LoRA in GEi is denoted as follows:

Finally, the rank of all adapters in GEi is determined. Since there is only one layer in R, we do not utilize adaptive rank allocation for GRi .

IncLoRA Training

From above, we obtain the LoRA group GEi and GRi , and adaptively determine the rank of each LoRA. Then, the embeddings of all entities Eall and all relations Rall can be denoted as follows:

where Eorigin and Rorigin denote the origin embeddings of entities and relations in snapshot 0, respectively. Finally, we train all LoRAs inGEi andGRi with new triples∆T . We take TransE [Bordes et al., 2013] as the base KGE model, and the loss can be denoted as follows:

where (h′ , r, t′ ) is the negative triple of (h, r, t) ∈ ∆Ti, and f(h, r, t) = |h+r-t|L1/L2 is the score function of TransE. h ∈ Eall, r ∈ Rall, and t ∈ Eall denote embeddings of h, r, and t, respectively. We only train the parameters in GEi and GRi , and fix parameters in all other LoRA groups and origin embeddings. Finally, all representations of entities and relations, i.e., Eall and Rall are obtained.

3.5 Link Predicting

In the stages of link predicting, we compose all LoRA groups as Equation 9 and 10. Taking link prediction as an example, we freeze all parameters of LoRA groups and initial embeddings. For a given query (h, r, ?), we calculate the tail entity t that gives the highest triple score of TransE as the prediction result. Notably, FastKGE only involves assembling LoRAs into a comprehensive embedding module before the inference stage without requiring additional operations, resulting in no additional time consumption in inference.

4 Experiments

4.1 Experimental Setup

Datasets

We use six datasets in the main experiments: ENTITY, RELATION, FACT, HYBRID, FB-CKGE, and WN-CKGE. ENTITY, RELATION, FACT, and HYBRID are traditional datasets for CKGE [Cui et al., 2023], in which the number of entities, relations, triples and mix of them are growing equally in each snapshot, respectively. However, realworld KGs are typically built on a substantial foundational graph, to which a small increment of new knowledge merges in each snapshot. To make up for the deficiency of small initial KGs in current CKGE datasets [Hamaguchi et al., 2017; Kou et al., 2020; Daruna et al., 2021; Cui et al., 2023], we construct two new datasets for CKGE: FB-CKGE and WNCKGE, which are based on two widely-used KGE datasets FB15K-237 [Dettmers et al., 2018] and WN18RR [Toutanova et al., 2015]. In FB-CKGE and WN-CKGE, we assign 60% of the total triples to the initial snapshot, and 10% of the total triples to each next snapshot. Compared to traditional datasets ENTITY, RELATION, FACT, and HYBRID based on FB15K-237, FB-CKGE has 2 to 4 times triples in base KGs in snapshot 0. We set the snapshots for all datasets to 5. The ratio of train, valid, and test sets for all datasets is 3:1:1. The details of the datasets are shown in Table 1. The datasets are available at https://github.com/seukgcode/FastKGE.

Baselines

We choose two standard baselines: incremental-parameter fine-tuning methods and full-parameter fine-tuning methods. For incremental-parameter fine-tuning methods, we choose PNN [Rusu et al., 2016] and CWR [Lomonaco and Maltoni, 2017]. For full-parameter fine-tuning methods, we choose replay-based methods GEM [Lopez-Paz and Ranzato, 2017], EMR [Wang et al., 2019], DiCGRL [Kou et al., 2020], and regularization-based methods SI [Zenke et al., 2017], EWC [Kirkpatrick et al., 2017], LKGE [Cui et al., 2023].

Metrics

We assess the performance of our model in the link prediction task. Specifically, we substitute the head or tail entity of the triples in the test set with all other entities and then calculate and rank the scores for each triple. Subsequently, we measure the Mean Reciprocal Rank (MRR), Hits@1, Hits@3, and Hits@10 as evaluation metrics. The higher MRR, Hits@1, Hits@3, and Hits@10 indicate better model performance. For each snapshot i, we compute the average of the above metrics evaluated on all the test sets of current and previous snapshots as the final metric. The main results are derived from the model generated at the final time. In addition, we calculate the total training time of all snapshots to evaluate the time efficiency.

Settings

All experiments are conducted on the NVIDIA RTX 3090Ti GPU. The codes of the experiments are supported by PyTorch [Paszke et al., 2019]. The number of snapshots is set to 5. We choose the batch size from [258, 512, 1024] and the learning rate from [1e-1, 2e-1, 3e-1]. We choose the Adam as the optimizer. In our experiments, we set the entity base rank of LoRA from the range [10, 50, 100, 150, 200] and the relation base rank to 20. Also, we set the number of LoRA layers N from the range [2, 5, 10, 20]. We set the embedding size for all methods to 200. To fairness, all the results are from the average of random five running times, and the patience of early stopping is 3 for all methods to compare time efficiency.

4.2 Results

Main Results

The main results are shown in Table 2 and Table 3. Overall, our method FastKGE achieves competitive performance compared to the state-of-the-art methods on all datasets. Furthermore, it outperforms all other baselines regarding time efficiency, highlighting its superior training speed.

Firstly, FastKGE outperforms all other baselines across all datasets regarding training time efficiency. Specifically, on the four traditional datasets ENTITY, RELATION, FACT, and HYBRID, FastKGE can save 34%-49% training time compared to the fastest baselines. Notably, on the two newly constructed datasets FB-CKGE and WN-CKGE, FastKGE can save 51%-68% training time compared to the fastest baselines. This demonstrates that our method with low-rank adapters will be more efficient in larger initial graphs.

Secondly, FastKGE has strong competitiveness in performance compared to the best baselines. Specifically, FastKGE achieves the best performance on two traditional datasets (ENTITY and HYBRID), and two newly constructed datasets (FB-CKGE, and WN-CKGE). Compared to the best baselines, FastKGE achieves 0.4%-1.5%, 0.7%-1.8%, 0.2%- 2.8%, and 0.2%-0.9% higher in MRR, Hits@1, Hits@3, and Hits@10, respectively. Notably, on two newly constructed datasets (FB-CKGE and WN-CKGE) with extensive initial triples, FastKGE can significantly improve performance by 1.5%, 1.3%, 2.4%, and 0.6% in MRR, Hits@1, Hits@3, and Hits@10 on average compared to the best methods, respectively. This proves that FastKGE performs better on KGs with larger initial scales, which is more in line with the changes in real life. Besides, FastKGE also demonstrates strong competitiveness on two other traditional datasets (RELATION and FACT), with only 0.7%, 0.2%, 0.6%, and 0.5% decrease compared to the optimal baselines in MRR, Hits@1, Hits@3, and Hits@10 on average, respectively. This performance decline on RELATION and FACT can be attributed to the limited number of entities they contain. Since our method addresses this issue by employing low-rank decomposition and knowledge storage for representing new entities. Therefore, the small changes in entity number on RELATION and FACT pose a challenge in showcasing the benefits of our method.

Ablation Results

This section investigates the effectiveness of incremental lowrank adapter (IncLoRA) learning and graph layering strategy (GL). The results are shown in Table 4 and Table 5. Firstly, we find that if we remove IncLoRA and fine-tune the KGE model directly, the model performance will significantly decrease by 2.8%-9.1%, 1.3%-6.8%, 3.3%-9.4%, and 2.5%-13.8% in MRR, Hits@1, Hits@3, and Hits@10 on all datasets, respectively. This proves that IncLoRA can effectively preserve learned knowledge with incremental low-rank adapters. Meanwhile, with IncLoRA the training time will decrease obviously by 36%-65% on all datasets. This proves that using IncLoRA can significantly improve training efficiency with low-rank composition compared to direct finetuning. Secondly, if we remove GL and use one LoRA for all entities, the performance will decrease slightly by 0.4%- 3.1% on all datasets. This proves that our multi-layer LoRAs are more effective than one LoRA with adaptive rank allocation. Also, the training time will decrease by 3%-18%, indicating that training and assembling multiple LoRAs require additional time. Notably, the additional training time only makes the 3%-9% time of the total training time on FBCKGE and WN-CKGE with more initial triples, where the additional time is much smaller than the whole training time.

Effectiveness of Base Ranks

This section investigates the effectiveness of different base ranks on the model performance and newly added parameters. Since the number of new entities is much more significant than new relations, the number of total embedding parameters is linear to the number of entities. Therefore, we only analyze the rank rbase for entities. We conduct our experiments on ENTITY, HYBRID, FB-CKGE, and WN-CKGE. The results are shown in Figure 3. Firstly, we find a significant increase in model performance from 1.5% to 5.2% in MRR when the rank changes from 10 to 150 in all datasets. However, when the rank size exceeds 150, there is only a slight improvement from 0.1% to 0.2% in MRR on ENTITY, FB-CKGE, and WN-CKGE. Notably, on HYBRID, there is a slight decrease of 0.1% in MRR when the rank changes from 150 to 200. This demonstrates that when the base rank is small, the larger the base rank, the better the performance. When the size of the base rank reaches an upper limit, the model performance hardly improves. Secondly, we find that the number of newly added parameters in the model increases continuously with the increase of rank. Therefore, the model performance does not continually improve with the increase of parameter quantity, but there is a specific upper limit. Combining Equation 4, we find that when the number of new entities is much greater than the dimension, the upper-rank limit is approximately equal to the dimension, which is consistent with the upper limit of the rank 200 shown in the experimental results.

Effectiveness of Layer Numbers

In this section, we investigate the effectiveness of different layer numbers N on model performance. We conduct experiments on two traditional datasets (ENTITY and HYBRID) and two new datasets (FB-CKGE and WN-CKGE), as shown in Figure 4. Firstly, on ENTITY and HYBRID, the model performance peaks when layer number equals five and then decreases by 0.4%-0.5% in MRR from 5 to 20 layers. This indicates that the layer should be set to a small number for the datasets with small initial graphs. Secondly, we observe that on FB-CKGE and WN-CKGE, the model performance increases by 0.4%-0.6% in MRR when the layer number ranges from 2 to 20. It proves that graph layering can achieve more significant results with larger initial KGs.

Different Base Models

In this paper, although we choose the most typical model, TransE, as the base KGE model, our method can still be extended easily to other base KGE models. To verify the scalability of our method FastKGE, we extend FastKGE to two other different types of KGE models, a standard bilinearbased model ComplEx [Trouillon et al., 2016] and a typical roto-translation-based Model RotatE [Sun et al., 2019]. We conduct the experiments on FB-CKGE, and the results are shown in Figure 5. Firstly, we observe that FastKGE outperforms direct fine-tuning by 2.6%-3.1% in MRR on ComplEx and RotatE, respectively. Meanwhile, for KGE models with better performance, applying our method will also lead to better performance. This indicates that FastKGE can effectively store the learned knowledge for different KGEs, alleviating catastrophic forgetting in continual learning. Secondly, our method can significantly shorten the time from 61.3% to 70.1% for different KGEs. This demonstrates that FastKGE has robust scalability for different KGE models to accelerate the training process.

4.3 Case Study

We conduct a case study to verify that FastKGE learns more critical entities well by graph layering with dynamic rank allocation. As shown in Table 6, we select two new entities with different importance, namely, with the same distance from the old graph (dis=1) but different fdc, and test the prediction results with layering and non-layering strategies. Results show that for entity Sedona with small fdc = 4.7e - 4, both strategies can rank it first. However, for entity Robert Morse with larger fdc = 1.18e - 3, the rank of the correct answer will increase after layering. It proves that graph layering learns critical new entities effectively without affecting others.

5 Conclusion

This paper proposes a novel fast learning framework for CKGE, FastKGE, which utilizes an IncLoRA mechanism to preserve learned knowledge well and accelerate fine-tuning. Firstly, to alleviate catastrophic forgetting, we conduct graph layering for new knowledge to achieve separate storage of old and new knowledge. Moreover, to reduce training costs, we propose incremental low-rank adapter learning to learn new knowledge efficiently with adaptive rank allocation. In the future, we will explore how to conduct CKGE learning when the knowledge of growing KGs is forgotten or modified.

Acknowledgments

We thank the reviewers for their insightful comments. This work was supported by National Science Foundation of China (Grant Nos.62376057) and the Start-up Research Fund of Southeast University (RF1028623234). All opinions are of the authors and do not reflect the view of sponsors.