GE-Chat: A Graph Enhanced RAG Framework for Evidential Response Generation of LLMs

1 Introduction

Large Language Models (LLMs) have shown exceptional performance in tasks such as multi-round conversational interactions, question understanding, response generation, and reasoning based on provided contexts [Hu et al., 2024], [Yuan et al., 2024], [Zhang et al., 2024], etc. These advancements have enabled diverse applications across domains, including customer support [Følstad and Skjuve, 2019], virtual assistance [Wei et al., 2024], and augmented agents with toolusage capabilities [Da et al., 2024].

Despite their extensive training on large-scale expert corpora, LLMs are prone to generating incorrect or misleading information [Ji et al., 2023]. Prominent LLM providers like OpenAI ChatGPT and Claude caution users with disclaimers such as "LLMs can make mistakes. Check important information carefully", emphasizing the ongoing challenge of ensuring response reliability [Liu et al., 2025].

Existing solutions to reduce the generation of misinformation primarily focus on grounding LLM outputs in factual data through either fine-tuning [Wang et al., 2024] or retrieval-augmented methods [Shuster et al., 2021]. While fine-tuning reduces hallucinations, it demands significant computational resources and is impractical for proprietary black-box models. Retrieval-augmented approaches rely on external sources for factuality verification [Ding et al., 2024], requiring multiple queries to external APIs or knowledge bases, which can increase latency and operational complexity.

To address these limitations, recent efforts have explored evidence-based response generation by matching model outputs to relevant source documents. For example, when users upload a document and pose a query, systems highlight raw contextual segments most relevant to the response, thus guiding the user to understand where the conclusion is drawn from the original document. However, the current method, such as [Lin, 2024] and [Han et al., 2024] use the direct LLM-responsed source of the evidence [Saad-Falcon et al., 2023]. It has two shortcomings, First, when faced with a redundant response, it can only perform on the chunk-level resource highlight, which gives a whole paragraph of relevant context without fine concentration. Second, the ability of LLMs to reflect their source evidence varies significantly on their instruction-following capability. Smaller models that lack instruction fine-tuning often struggle to highlight relevant information for users to refer to effectively.

To resolve the above issues, this paper proposes a framework named GE-Chat that provides users with evidence of an LLM's response in a more accurate and generalizable way. Different from existing work, this method not only poses constraints on the derived source that it must come from the original context, but also provides sentence-level fine-grained identification to accurately mark out the evidence supporting the LLMs' conclusion. This framework can also be applied to any LLM with outstanding evidence retrieval ability (related to the requirement of instruction-following ability). We compare with the baseline that gets source evidence by direct reflection, GE-Chat improves on retrieval performance.

2 Approach

We discuss the details of the GE-Chat framework, which builds upon the graph-based retrieval augmented generation agent (Graph-RAG), and then innovates on a three-step paradigm for better evidential response generation; we introduce from constructing the RAG agent Sec. 2.1), and then explain three components in our framework (Sec. 2.2, 2.4).

2.1 Graph-RAGConstruction

The Graph-RAG [Larson and Truitt, 2024] is unique in its ability to integrate external information through a structured knowledge graph, thus supporting graph-based queries and allowing for relational reasoning. Besides, it also does well in handling multi-hop reasoning, this feature helps to find more than one-hop-related entities in the knowledge base, extract sub-graphs, and capture the relational semantics clusters. After a user uploads a document in (.TXT/.DOC/.PDF) format, the metadata will be cut into corpus chunks to temporarily store the file, then we construct a knowledge graph G by two steps: ➀ extract entities from the chunks, and ➁ probe the relations among the entities. The LLM is used to achieve the knowledge graph (KG) and KG is used back to the LLM as external information to make responses. We constructed a fast and lightweight Graph-RAG following the implementations [Guo et al., 2024].

2.2 CoT Reasoning Elicitation

This section introduces the Chain-of-Thought (CoT) reasoning inducer, which serves as a primary step in deriving the reasoning process. It is well acknowledged that the majority of the LLMs can automatically perform CoT [Wei et al., 2022] to elicit the reasoning process, i.e., how they draw the conclusion step by step. We follow the same idea to induce the logic steps Logic steps{step1 , step2 , ..., stepn } from the LLM given a question Q on a document. By designing a CoT template, we tend to achieve the following:

The template is inspired by work [Zhang et al., 2022] and is shown in the above green block. This step corresponds to the upper right part of Fig. 1, CoT Reasoning Elicitation, where each answer is associated with a CoT chain that explains the reasoning process step by step. These CoT chains, generated by RAG models, provide a logical structure to responses, but may not inherently align with the raw content of the submitted document. To ensure the evidence strictly originates from the provided source, a critical grounding step is introduced through sub graph searching based on entity matching. This process anchors CoT reasoning to specific entities and relationships within the knowledge graph, bridging the gap between generated content and its original context. By doing so, we enhance the trustworthiness and accuracy of the responses while maintaining a clear traceability to the original document.

2.3 Efficient Sub-Graph Searching

The sub-graph searching is conducted based on two resources: Derived KG G and CoTs as in Figure 2. For each of the CoT results: ci ∈ {c1, c2, c3, ...., cn}, the ci will be used to match the most relevant graph entities (Entity Matching), as shown in the output part, the blue boxes are the identical ci, and the · is connected by blue edges, which is the first hop most relevant entities, this is what ci has been involved in the LLM's answer, then we relax this relation to further second-hop as shown in · green dot. This search is efficient for the CoT guidance and pre-calculated G for n-hop relationships probing (in contrast to the whole document-range global search). Finding this entity sub-graph is like finding an anchor that leads to the original source chunks, we can perform Source Chunk Retrieval to get several chunks for each ci in CoTs. This step bridges the made-up CoT content with the source content of documents by finding the anchor entity. However, chunk-level descriptions leave space for more fine-grained evidence sentences [Zhao et al., 2024], so we employ an optimization objective in the next section.

• (a) Abstraction for Sub-graph Searching
• (b) Prompt Template for CoT Elicitation

2.4 Evidence Content Optimization

As in Figure 1, based on the chunk sources (e.g.,chunk1,2,6 ), for each of the ci in CoTs, we expect to find a finer sentence from a certain chunk that best supports the answer sentence S′ with minimal redundant information. We first formalize this problem, and then provide a solution in corresponds to 4. Optimize in the Fig. 1 uses the entailment probability.

Problem Setting: Given a chunk of text containing n sentences, S = {s1, s2, . . . , sn}, and a target sentence S′ which represents the sentence in the answering content, the objective is to find the best sentence sbest ∈ S that: 1. Maximizes the entailment probability prob(sn | S′ ), which measures how strongly sn entails S′ , 2. Minimizes the sentence length len(sn), encouraging concise representations.

In order to balance these two criteria, we define an objective function F(sn), which assigns a score to each sentence sn based on its contained meaning and conciseness. The score for each sentence sn is given by: F(sn) = α · prob(sn| - S′ ) - β · len(sn), where α and β are set as 0.5 to control the weight of the entailment probability, and penalty for longer sentences, respectively. We want to measure how much the generated evidence means similarly to the answer, a rational way is to calculate the entailment probability prob(sn| - S′ ). We achieve this by using NLI model 1 , which provides a three-element tuple by taking two text pairs sn and S′ : [logitcont , logitneut , logitent ] = --→ NLI (sn, S′ ). The output is processed by transforming into the probability through p = Softmax(logitcont , logitneut , logitent ). Then we can calculate the --→ pent(sn, S′ ) = p(sn ⊢ S′ ) = p3 as the entailment probability. The optimal sentence sbest is the one that maximizes F(sn): sbest = argmaxsn∈S (F(sn)) Using this objective, we can find the best evidence that supports the answers in the LLM's responses, and this action is performed in a small chunk, which is not computationally expensive and can be deployed in real time.

The sbest will be calculated for each of the answers, such as in Figure 1, the best evidence output for Ans1 is a combination of sentences Sent.1, 2, 6. This will be returned back to users together for users to understand which part of the answer comes with the evidence supported and which part lacks such trustworthy information, helping practitioners understand the reliability of generated content.

1 we use off-the-shelf DeBERTa-large model [He et al., 2021]

3 Experiment

Dataset: To address the scarcity of evidence sources in prior research, we created a dataset with 1000 cases to evaluate evidence generation quality across 10 categories: Biology, Chemistry, etc, with three dimensions: (1) PDF length-short (<10), medium (10-100), and long (>100 pages); (2) Question types-synthesis, and (3) human-annotated answers with corresponding evidence sentences, ensuring reliability and comprehensiveness. We tested our method on this dataset and have released the dataset and videos 2 for public use with standard questions, groundtruth answers, and evidence for reference. Evaluation Metric: Following existing work [LangChain, 2023], we use the cosine similarity to evaluate the relevance of the generated text (evidence) with the correct evidence and use the conciseness score [Ragas authors, 2024] to quantify how precise response is. In combination, we have the following score(↑):

where the Ei is the embedding of the generated evidence for question i, and Egti is the groundtruth evidence. The first termmeasures the cosine similarity of two evidence (↑), and Lgti is the length of text for groundtruth evidence while the Li is the generated evidence, Lgti Li measures conciseness.

3.1 Experiment Results and Conclusion

In the experiment, the direct evidence retrieval ability of GPT4o is the best, while other models perform worse, especially lacking conciseness. After applying GE-Chat to existing models except for GPT4o (GPT4o is involved in the process of ground-truth 'reference' generation). The results are shown in Fig. 2 (c). In summary, we presented GE-Chat, a novel framework addressing the trustworthiness of LLMs by evidence retrieval and verification. Through constraints on source derivation and sentence-level highlight capabilities, GE-Chat significantly enhances the reliability of LLMgenerated responses. By offering a transparent and userfriendly approach, GE-Chat contributes to making AI systems more reliable and trustworthy, paving the way for responsible deployment in critical decision-making processes.

2 Click the link to the testset and videos.

Acknowledgments

The work was partially supported by NSF awards #2421839, NAIRR #240120 and used AWS through the CloudBank project, which is supported by NSF grant #1925001. The views and conclusions in this paper are those of the authors and should not be interpreted as representing any funding agencies. Thank OpenAI for providing API credits under the Researcher Access program and Amazon Research Awards.