How to Mitigate Information Loss in Knowledge Graphs for GraphRAG: Leveraging Triple Context Restoration and Query-Driven Feedback

1 Introduction

Large Language Models (LLMs) augmented with Knowledge Graphs (KGs) have achieved remarkable successes across diverse domains, from social sciences to biomedicine [Pan et al., 2024; Peng et al., 2024; Yang et al., 2024; Soman et al., 2024]. By harmonizing the structured information in KGs and the sophisticated language understanding and processing capabilities of LLMs, such hybrid systems enable more accurate and context-aware reasoning for complex tasks.

Despite these advances, the performance of current KGLLM integration methods is often hindered by the underlying assumption that the KG is complete. Typical integration strategy involves retrieving relational data from a constructed KG and feed it into LLMs via prompt augmentation [Peng et al., 2024; Sun et al., 2023; Edge et al., 2024; Zhang et al., 2025b; Dehghan et al., 2024], assuming that critical entities and relationships relevant to the query are already captured within the KG. In practice, however, KG construction itself is beset by inherent constraints, where vital contextual information can be discarded in the process of converting unstructured text into structured triples, leading to missing or incomplete relations [Zhu et al., 2024; Zhong et al., 2024]. Such missing information in KGs can significantly degrade the LLM reasoning capabilities.

∗Corresponding author.

To wit, Figure 1 illustrates two primary sources of information loss. First, information sparsity arises when information extraction falls short, omitting potentially important triples and thus failing to provide sufficient coverage for specific queries [Biswas et al., 2024; Xu et al., 2024b; Li et al., 2023; Zhang and Soh, 2024; Chen et al., 2024a; Sun et al., 2024; Cohen et al., 2023]. This sparsity can be exacerbated by data noise, long-tail entities, and complex relationships, where extraction algorithms often falter. Second, context loss occurs when transforming rich yet unstructured text into discrete triples, sacrificing crucial semantic nuances and relational dependencies [Trisedya et al., 2019; Paulheim, 2017; Xu et al., 2024a]. While prior studies attempt to mitigate this issue by refining graph structures or retrieval algorithms [Liang et al., 2024; Chen et al., 2024b; Panda et al., 2024; Munikoti et al., 2023; Cohen et al., 2023], their subgraphs still lack the broader contextual information that is vital for robust reasoning, resulting in suboptimal performance in downstream tasks.

To address these challenges, we propose the Triple Context Restoration and Question-driven Feedback (TCRQF) framework, which aims to restore the missing contextual information and dynamically enrich the KG during the reasoning process. Specifically, our TCR-QF approach presents a triple context restoration component that retrieves the original text passages associated with each triple, thereby recapturing the semantic details often lost during KG construction. We further enhance KG coverage through a querydriven feedback mechanism, which iteratively identifies missing information relevant to the query and enriches the KG accordingly. These two components together form a synergistic cycle in which contextual fidelity and KG completeness are continuously reinforced, resulting in more accurate and context-aware responses from the LLM. Empirical study on five benchmark question-answering datasets substantiates that TCR-QF significantly outperforms the state-of-the-art GraphRAG methods in both response accuracy and completeness, demonstrating its effectiveness.

Specific Contributions of this paper are as follows:

1. We provide a systematic analysis of the key challenges in KG-LLM integration, highlighting the loss of contextual information and incomplete information extraction during KG construction, both of which hinder an advanced LLM reasoning performance.
2. We propose the TCR-QF framework, which restores the semantic context associated with triples and employs a query-driven feedback mechanism to iteratively enrich the KG, thereby significantly enhancing the LLM reasoning capabilities.
3. Extensive experiments on five benchmark questionanswering datasets are carried out, showing that TCRQF achieves an average 29.1% improvement in Exact Match and a 15.5% improvement in F1 over its GraphRAG competitors. These results validate the merit of restoring contextual information and dynamically updating KGs for effective KG-LLM integration.

2 RelatedWork

GraphRAG has emerged as a powerful paradigm for integrating knowledge graphs (KGs) with large language models (LLMs) to advance complex reasoning tasks [Pan et al., 2024; Peng et al., 2024; Yang et al., 2024; Zhang et al., 2024; Zhang et al., 2025a]. A widely adopted strategy involves retrieving relevant subgraphs from preconstructed KGs to augment LLMs during inference [Yasunaga et al., 2021; Taunk et al., 2023], with techniques such as extracting hop-k paths around topic entities [Yasunaga et al., 2021] or focusing on the shortest paths relevant to query entities [Delile et al., 2024]. More sophisticated methods optimize subgraph retrieval by assigning edge costs [He et al., 2024] or leverage LLMs themselves to generate new relations or invoke function calls [Kim et al., 2023; Jiang et al., 2023].

While these approaches have demonstrated effectiveness, most remain limited by their dependence on the initial completeness of the KG and often overlook the contextual information lost during KG construction. In reality, KGs are frequently incomplete due to information loss during construction and the difficulties in extracting all relevant triples, especially in noisy or complex scenarios [Biswas et al., 2024; Cohen et al., 2023]. These constraints can hinder the ability of LLMs to formulate coherent and context-rich reasoning paths. Addressing these gaps calls for a more dynamic strategy that restores missing contextual details and continuously refines the KG, ensuring that the retrieved and generated knowledge is both accurate and semantically complete.

However, the lack of essential data negatively impacts the inference results of LLMs. To address this, efforts have been made to enhance KG comprehensiveness through refined indexing methods and innovative graph structures for retrieving both triples and texts [Chen et al., 2024b; Munikoti et al., 2023; Liang et al., 2024; Cohen et al., 2023], as well as using LLMs to improve automated KG construction [Zhang and Soh, 2024; Xu et al., 2024b; Li et al., 2023]. These methods may retrieve texts related to the query without fully meeting its requirements. Additionally, the retrieved subgraphs can result in the loss of crucial information due to the absence of contextual data within triples, which is essential for maintaining semantic integrity. As a result, the constructed KG may lack critical information necessary for accurate reasoning, leading to suboptimal performance in downstream tasks.

The proposed TCR-QF framework addresses these limitations by dynamically enriching the KG during the reasoning process. By restoring the original textual context of triples, TCR-QF recovers lost semantic information. Additionally, it employs a query-driven feedback mechanism to identify and fill in missing information relevant to a query, enabling the KG to continuously update. This mutual enhancement between KG and LLM improves reasoning performance and better adapts to task requirements.

3 ProposedMethod

Task Definition. Given a set of documents D = {d1, d2, ..., dn} and a question Q, the task requires the model to read and reason over multiple relevant documents, extract and aggregate the necessary information, and finally generate the answer A based on the information from the documents.

In this section, we present the TCR-QF framework, designed to mitigate the loss of contextual information when building knowledge graphs (KGs) from unstructured text and to dynamically enrich these graphs during the reasoning process. As shown in Figure 2, the framework comprises four key components: (1) Knowledge Graph Construction, which builds a unified KG from textual sources; (2) Subgraph Retrieval, responsible for extracting task-relevant subgraphs composed of potential reasoning paths; (3) Triple Context Restoration, which traces back the original textual context of each triple to recover lost semantic nuances; and (4) Iterative Reasoning with Query-Driven Feedback, where an iterative cycle that both generates answers and identifies missing knowledge, thereby refining the KG on-the-fly. Together, these components ensure that contextual details are preserved and the KG remains up-to-date, ultimately enhancing the quality and depth of the system reasoning.

The above steps establishes a synergistic cycle for two-way knowledge enhancement, namely,

Forward Flow: The KG informs the LLM during answer generation, represented as

In the i-th iteration, a subgraph is retrieved from the KG G(i) and enhanced through triple context restoration to formG′(i) . The LLM then infers the answer A(i) LLM based on G′(i) .

Feedback Flow: Themissing knowledge in the KG is identified and subsequently integrated back into the KG:

where ∆K(i) represents the knowledge increment corresponding to the missing knowledge. This increment is updated in the KG as triples, resulting in the more comprehensive KG G(i+1) .

3.1 Knowledge Graph Construction

An initial KG was constructed from raw textual data using LLMs to extract entities and relations as triples (eh, r, et), where entities include names, types Type(e), and descriptions Desc(e). Source document information is retained in each node for provenance. The construction involves:

Document Splitting Each document D of length L is divided into overlapping chunks Ci with maximum length MAX_LEN = 512 and overlap OVERLAP = 64 tokens:

The overlap ensures entities and relations spanning across chunks are captured.

Triple Extraction From each chunk Ci, the LLM extracts

triples:

where Ti is the set of triples from Ci. A specialized prompt guides the LLM to output structured information, including entity types and descriptions.

3.2 Subgraph Retrieval

The subgraph retrieval phase focuses on extracting pertinent information from the KG in response to the query. Specifically, given a query Q expressed in natural language, the retrieval stage aims to extract the most relevant elements (e.g., entities, triplets, paths, subgraphs) from KGs, which can be formulated as:

where G∗ = {(h0, r0, s1), (h1, r1, s2), . . . , (ht, rt, st+1)} is the optimal retrieved graph elements and Sim(·, ·) is a function that measures the semantic similarity between user queries and the graph data. R(G) represents a function to narrow down the search range of subgraphs, considering efficiency. The retrieval method employed in the TCR-QF builds upon existing KG retrieval method [Sun et al., 2023], which utilize LLMs to perform a beam search over the KG, with iterative pruning guided by the LLM.

3.3 Triple Context Restoration

The structuring of unstructured text into triples can lead to a loss of semantic context. To address this issue, a triple context restoration mechanism was implemented in TCR-QF to restore semantic integrity by tracing back to the original textual context of the triples.

Context Retrieval For each triple (eh, r, et) in the retrieved subgraphs, the source documents associated with the head and tail entities were retrieved:

These sources were the documents which the entities were originally extracted during KG construction. This set encompassed all documents potentially containing contextual information about the relationship between eh and et.

Triple Context Tracing To trace the context of the triple (eh, r, et), the most relevant sentence from source documents were identified. A template T(eh,r,et) was used, such as:

A pretrained embedding model fembed was used to generate embeddings for both the template and candidate sentences. The context relevance was assessed via cosine similarity:

where S is the set of all sentences extracted from Sources(eh,et). The sentence with the highest similarity score was selected to provide contextual information into the triple.

Triple Augmentation Each triple was augmented with its associated contextual sentence:

This augmentation restored the contextual information of the triples, improving the accuracy and depth of inference tasks that rely on the KG.

3.4 Iterative Reasoning with Query-driven Feedback

To generate accurate answers to the original queries Q, an iterative reasoning process incorporating a query-driven feedback mechanism was implemented. This approach dynamically enriches the KG by identifying and updating missing information during the reasoning process, thereby enhancing the LLM's capability to produce more accurate responses.

Initially, the enriched subgraph G′(0) obtained from triple context restoration was used to prompt the LLM:

The LLM then generated an initial answer A(0) LLM by processing this prompt:

where LLM_Generate refers to generating a response based on the formatted input I (0) .

Missing Knowledge Identification The initial answer and contexts were analyzed to identify missing information required for the query:

where ∆K(0) represents the set of missing knowledge, formalized as a series of sub-questions. The function IdentifyMissing utilizes the LLM to compare Q with A(0) LLM and G′(0) , effectively harnessing its understanding to identify gaps in knowledge.

Knowledge Graph Enrichment For each missing component kq ∈ ∆K(0) , a dense retriever interacted with the original text sources D to retrieve relevant textual information and extract the missing knowledge:

where ExtractTriples employs the LLM to find and extract the needed information, resulting in triples k corresponding to the missing knowledge. The KG was then updated:

Duplicate relationships were filtered based on edit distance from elements in G(0) to maintain uniqueness in G(1) . A dense passage retriever, implemented using OpenAI's text-embedding-small, was employed due to its effectiveness in retrieving semantically relevant passages.

Iterative Reasoning and Update The updated KG G(1) was used to generate a new answer by following the reasoning steps:

This iterative process continued, repeating the steps of Missing Knowledge Identification and Knowledge Graph Enrich-

for i = 2, 3, . . ., until ∆K(i) = ∅ or a predefined maximum number of iterations Imax = 20 was reached. By analyzing retrieved contexts and generated responses at each iteration, gaps in the KG were detected and addressed, continuously optimizing the KG and enhancing the reasoning capabilities of the LLM.

Due to space constraints, the detailed prompts used for the LLM at each step are provided in the appendix 1 .

4 Experiments

To evaluate the effectiveness of the TCR-QF on questionanswering tasks, experiments were conducted on 5 questionanswering datasets: 2WikiMultiHopQA [Ho et al., 2020], HotpotQA [Yang et al., 2018], ConcurrentQA [Arora et al., 2023], MuSiQue-Ans and MuSiQue-Full [Trivedi et al., 2022] . Followed the settings outlined in [Yang et al., 2018], utilizing a collection of related contexts for each pair as the retrieval corpus. Exact Match (EM) and F1 score were presented as the evaluation metrics across all datasets.

We compared TCR-QF with representative methods from LLMs and RAG:

(1) LLMOnly: Methods that directly use LLMs for obtaining answers, including models such as gpt-4o-mini and gpt-4o, as well as chain-of-thought (CoT) [Wei et al., 2022] prompting strategies.

1 The appendix is available in the arXiv version: https://arxiv.org/ abs/2501.15378.

1. Text-based RAG: Methods that employ a dense retriever to retrieve relevant text chunks from a text corpus and generate answers by leveraging this information. For this category, LangChainQ&A 2 was used as a representative naive RAG method, which is well-known and widely used.
2. Graph-based RAG: Methods that retrieve subgraphs from KG to enhance LLM. ToG [Sun et al., 2023] was selected as a representative for comparison in this category.
3. Hybrid RAG: Methods like GraphRAG [Edge et al., 2024] that retrieve information from both KG and textual documents to augment LLM.

Experimental Settings: For all comparison methods and the TCR-QF, unless otherwise specified, the gpt-4o-mini-2024-07-18 model was utilized. Due to the high computational costs associated with inference on the full dataset, 1,200 samples were randomly selected from each of the larger datasets-2WikiMultiHopQA, HotpotQA, MuSiQue-Full, and MuSiQue-Ans-for testing to conserve computational resources. For ConcurrentQA, 1,600 samples from the complete test set were evaluated.

4.1 Results and Findings

Table 1 presents the comparative results, from which we answer the following Research Question (RQ).

RQ1 How does the TCR-QF improve the completeness and accuracy of information retrieval in question answering tasks compared to the existing GraphRAG methods?

Table 1 demonstrates the superiority of the TCR-QF compared to different methods on five benchmark question answering datasets. TCR-QF consistently achieves the highest EM and F1 scores across all datasets, demonstrating its superior effectiveness in enhancing LLMs for complex reasoning tasks. Compared to the LLM-only approaches (GPT4o-mini, GPT-4o and CoT), TCR-QF shows substantial improvements. For instance, on the HotpotQA dataset, TCRQF attains an EM score of 0.558, which is 0.207 higher than GPT-4o's score of 0.351, representing a relative improvement of approximately 59%. This indicates that while LLMs possess strong language understanding capabilities, integrating external knowledge as TCR-QF does markedly enhances their accuracy in answering complex questions.

When contrasting TCR-QF with the text-based method (Naive RAG) and the graph-based method (ToG), TCRQF exhibits notable performance gains. Specifically, on the 2WikiMultiHopQA dataset, TCR-QF achieves an EM score of 0.598, which is an absolute increase of 0.259 over Naive RAG's EM score of 0.339-a relative improvement of approximately 76.4%. Similarly, TCR-QF surpasses TOG's EM score of 0.400 by an absolute margin of 0.198, reflecting a 49.5% improvement. This significant enhancement indicates that TCR-QF's approach of enriching the LLM with more comprehensive knowledge markedly improves reasoning, outperforming methods that rely solely on retrieved texts or static KGs.

2 https://python.langchain.com/docs/tutorials/rag

Furthermore, TCR-QF outperforms the hybrid method (GraphRAG), which combines text and graph information. On the MuSiQue-Full dataset, TCR-QF achieves an EM score of 0.303, compared to GraphRAG's EM score of 0.189. This represents an absolute increase of 0.114, amounting to an improvement of approximately 60.3%. These significant gains demonstrate that TCR-QF effectively leverages knowledge to enhance the LLM's performance beyond what is achieved by simply combining text and graph data. By dynamically restoring lost semantic information and enriching the KG during reasoning, TCR-QF provides a more comprehensive context for the LLM, leading to better reasoning and answer generation in complex tasks.

The consistent superiority of TCR-QF across multiple datasets-ranging from general question-answering to those requiring multi-hop reasoning-highlights TCR-QF's robustness and general applicability. TCR-QF effectively addresses the challenges posed by incomplete KGs and information loss, leading to more accurate and complete responses.

4.2 Ablation Study

To evaluate the individual contributions of the proposed components, namely triple context restoration (TCR) and querydriven feedback (QF), to the overall performance of the TCRQF, an ablation study was conducted on the 2WikiMultiHopQA and HotpotQA datasets to answer the question:

RQ2 In what ways does each component in TCR-QF enhance the reasoning of the LLM?

Table 2 presents the results of the ablation experiments. The full TCR-QF is compared with several ablated variants:

• ToG (w/o TCR & QF): The baseline method operating on the KG.
• TCR (w/o QF): Incorporates triple context restoration alone to address contextual information loss.
• QF (w/o TCR): Employs query-driven feedback alone to approach incomplete information extraction.
• TCR-AF: Integrate triple context restoration with answer-driven feedback (AF) which involves directly extracting triples from the LLM's answer and adding them to the KG.

From the results we can draw the following insights.

Effectiveness of Triple Context Restoration (TCR). Comparing the baseline ToG method with the TCR variant, it can be observed that introducing triple context restoration leads to significant performance improvements. On the 2WikiMultiHopQA dataset, the EM score increases from 0.400 to 0.481, representing an improvement of 20.25%, while the F1 score rises from 0.476 to 0.561. Similarly, on the HotpotQA dataset, the EM score improves from 0.420 to 0.494 (a 17.62% improvement), and the F1 score increases from 0.555 to 0.642. These enhancements confirm that triple context restoration effectively mitigates contextual information loss by reconnecting structured triples with their original textual context, thereby enriching the semantic information available for reasoning.

Effectiveness of Query-Driven Feedback (QF). The QF variant, which focuses on dynamically updating the KG based on the requirements of the query, shows even greater improvements over the baseline. The EM scores rise to 0.568 (a 42.00% improvement) on 2WikiMultiHopQA and 0.522 (a 24.29% improvement) on HotpotQA. These substantial gains indicates that query-driven feedback significantly addresses the issue of incomplete information extraction. By dynamically enriching the KG based on the specific requirements of the query, the model fills in the missing knowledge that static KGs might overlook due to limitations in initial extraction algorithms. This adaptive approach continually enhances the relevance and comprehensiveness of the knowledge graph throughout the reasoning process.

Synergy of TCR and QF. The full TCR-QF method, which combines both triple context restoration and querydriven feedback, achieves the highest performance. EM scores reach 0.598 on 2WikiMultiHopQA and 0.558 on HotpotQA, with relative improvements of 49.50% and 32.86% over the baseline, respectively. These results underscore a synergistic effect when combining TCR and QF, as the model benefits from both restored contextual semantics and a dynamically enriched KG. The integration of both components effectively addresses the dual challenges of information loss, leading to more accurate and complete reasoning.

Comparison with Answer-Driven Feedback (TCR-AF). The TCR-AF variant replaces query-driven feedback with answer-driven feedback, where triples are extracted from the model's answers to update the KG. While TCR-AF outperforms ToG, achieving EM scores of 0.538 on 2WikiMultiHopQA and 0.523 on HotpotQA, it falls short compared to TCR-QF. TCR-QF scores 0.598 on 2WikiMultiHopQA, an 11.12% increase over TCR-AF. This suggests that enriching the KG proactively based on the query is more effective than reactively updating it based on answers, likely because it prevents error propagation from incomplete initial reasoning.

Performance Trends Across Rounds. Figure 3 illustrates the EM performance across multiple reasoning rounds for each method. It is evident that TCR-QF consistently outperforms other variants from the initial rounds and maintains its lead as the reasoning progresses. The performance gain from TCR-QF is additive, with TCR-QF achieving the highest accuracy. The diminishing returns after a few rounds indicate that the most significant knowledge enrichment occurs early in the reasoning process, emphasizing the performance of proposed method.

In conclusion, the ablation study corroborates our initial hypotheses, demonstrating that both triple context restoration and query-driven feedback are vital in addressing the inherent limitations of integrating KGs with LLMs. Individually, each component contributes significantly to performance improvements by targeting specific sources of information loss-triple context restoration restores essential contextual semantics lost during the structuring process, while querydriven feedback dynamically enriches the KG to address incomplete information extraction. These results highlight the effectiveness of restoring semantic integrity and continuously updating the KG during reasoning, fulfilling our research objectives and underscoring the importance of a bidirectional knowledge flow in optimizing reasoning outcomes.

4.3 Statistical and Convergence Analysis

To evaluate the effectiveness and convergence of the TCRQF, statistical analyses were conducted over multiple inference rounds. Table 3 presents key metrics from the initial round to the 10th round, including the numbers of nodes and edges in the KG, as well as the EM and F1 scores on the 2WikiMultiHopQA dataset. These experiments and results provide answers to the following question:

Table 3: Statistics from the initial round to the 10th round on 2WikiMultiHopQA dataset, where ∆ denotes the cumulative increase.

From the results we can draw the following insights.

Continuous Improvement of KG Completeness and Model Reasoning Performance. As demonstrated in Table 3, the TCR-QF significantly enriches the KG over successive inference rounds. Specifically, on the 2WikiMultiHopQA dataset, the number of nodes in the KG increased by 4,090 (from 74,571 to 78,661), and the number of edges increased by 10,606 (from 69,866 to 80,472) over 10 rounds. This enrichment directly addresses the issue of information sparsity by incorporating previously missing triples and expanding the KG's coverage to meet query demands. Correspondingly, the model's reasoning performance improved substantially. The Exact Match (EM) score increased from 0.481 to 0.598, a 24.3% improvement, and the F1 score rose from 0.562 to 0.680, a 21.0% improvement. These significant performance gains indicate that the enriched KG provides the LLM with more comprehensive and contextually rich information, directly mitigating the effects of context loss and enhancing reasoning accuracy.

Alignment of KG Completeness and Reasoning Performance Enhancement. As depicted in Figure 4, the parallel upward trends in KG metrics and performance scores affirm a strong correlation between the enriched KG and the model's improved reasoning ability. By restoring the contextual information associated with triples and integrating new, relevant knowledge through query-driven feedback, the TCR-QF enhances the semantic integrity of the KG. This comprehensive knowledge base enables the LLM to perform more accurate and context-aware reasoning, directly addressing the limitations posed by information sparsity and context loss.

Convergence of KG Enrichment and Performance Improvements. The TCR-QF not only enriches the KG but also exhibits convergence over inference rounds, ensuring efficient use of computational resources. As illustrated in Figure 4, both the growth of the KG and the improvement in performance metrics begin to plateau after several rounds, specifically between the 8th and 10th iterations. The incremental increases in nodes and edges diminish, and the EM and F1 scores stabilize at 0.598 and 0.680, respectively. This convergence suggests that the TCR-QF effectively enriches the KG to an optimal level, beyond which additional iterations yield minimal benefits.

The experimental results validate the effectiveness of the TCR-QF in overcoming the foundational challenges outlined in the introduction. By continuously and efficiently enhancing the KG's completeness and restoring lost contextual nuances, the TCR-QF significantly boosts the model's reasoning performance. These findings confirm that addressing information loss through dynamic KG enrichment and context restoration is a viable and efficient strategy for advancing the integration of KGs and LLMs in complex reasoning tasks.

5 Conclusion

This paper introduces TCR-QF, a novel framework that integrates knowledge graphs (KGs) with large language models (LLMs) to enhance complex question answering. By mitigating context loss through triple context restoration (TCR) and addressing incomplete extraction with query-driven feedback (QF), TCR-QF recovers key semantic details and dynamically expands the KG during reasoning. Experiments on five benchmarks show that TCR-QF outperforms state-of-the-art methods, demonstrating the benefits of contextualized triples and iterative KG updates. These results highlight the potential of TCR-QF to bridge the gap between structured and unstructured knowledge, paving the way for more accurate and robust AI-driven reasoning across diverse domains.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 62120106008), the Anhui Provincial Science and Technology Fortification Plan (Grant No. 202423k09020015), and the Youth Talent Support Program of the Anhui Association for Science and Technology (Grant No. RCTJ202420). The authors gratefully acknowledge these supports.