EVICheck: Evidence-Driven Independent Reasoning and Combined Verification Method for Fact-Checking

1 Introduction

Fact-checking involves verifying the accuracy of claims or information, often to determine whether they are true or false. Traditionally, experts manually assess claims using authoritative sources and their expertise, with platforms like PolitiFact 1 leading the way. Automated approaches, such as FEVER [Thorne et al., 2018], scale the verification process using knowledge bases like Wikipedia. However, manual methods are limited by scale, and automated techniques, while scalable, struggle with complex claims and with ensuring verification accuracy without human oversight.

∗Corresponding author.

1 https://www.politifact.com/

Recent advancements in Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) have improved automated fact-checking systems [Guu et al., 2020; Izacard and Grave, 2021]. With strong language comprehension and efficient use of external knowledge bases, these systems offer notable advantages in tackling fact-checking tasks [Ostrowski et al., 2021; Chen et al., 2024; Lewis et al., 2020]. However, existing methods face persistent challenges in verifying complex disinformation [Chern et al., 2023; Khaliq et al., 2024]. One major limitation lies in evidence utilization [Atanasova et al., 2020; Kotonya and Toni, 2020]. While evidence is often collected through various approaches, most methods rely on overall validation instead of independently analyzing each piece of evidence, resulting in incomplete exploration of available information [Khaliq et al., 2024; Yue et al., 2024]. We term this problem Insufficient Evidence Utilization (IEU). Furthermore, many methods lack clear verification standards, often using simple prompts or few-shot examples for credibility verification, which undermines their reliability when dealing with complex or ambiguous claims [Zhang and Gao, 2023; Yue et al., 2024]. Addressing these gaps requires more effective evidence utilization and the establishment of robust verification standards.

Inspired by structured reasoning and evidence integration used by humans to address complex issues, recent research has sought to improve the reliability and transparency of automated systems [Chern et al., 2023; Zamani and Bendersky, 2024]. However, many approaches struggle to integrate evidence verification and reasoning effectively, limiting their capacity to process complex information [Khaliq et al., 2024; Zhang and Gao, 2023]. To overcome these challenges, we propose EVICheck, a novel method that enhances evidencebased reasoning and introduces fine-grained truthfulness criteria to improve fact-checking performance.

EVICheck performs independent reasoning for each piece of evidence, ensuring that each piece is thoroughly analyzed and utilized rather than being overshadowed by collective aggregation, as shown in Figure 1. It also introduces finegrained truthfulness criteria, making the evaluation process more structured and reliable. We also integrate search engine APIs (e.g., SerpApi 2 ) into the RAG process to ensure the system can access the most up-to-date relevant information, enhancing the model's real-time capability and accuracy. Experimental results show that EVICheck performs best on the public RAWFC dataset, particularly demonstrating superior potential compared to existing methods when handling complex, ambiguous, or controversial statements.

Our contributions can be summarized as follows:

• We propose the EVICheck method, which fully utilizes each piece of evidence by performing independent reasoning, enhancing the comprehensiveness and effectiveness of evidence utilization.
• We construct fine-grained truthfulness criteria within the framework, offering a more structured and reliable factchecking process and improving the model's decisionmaking accuracy, particularly on complex statements.
• We demonstrate the superiority of EVICheck through experiments on the public RAWFC dataset, showing its significant application potential in fact-checking.

2 RelatedWork

Fact-Checking Based on LLMs and RAG. Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) methods are effective tools for improving factchecking accuracy. While LLMs embed extensive knowledge through pretraining, relying on static data limits their timeliness and knowledge coverage [Izacard and Grave, 2021; Wang et al., 2023]. RAG methods address this by incorporating external knowledge sources to enhance LLMs' knowledge acquisition [Borgeaud et al., 2022; Wu et al., 2022]. For example, LLM-Augmenter [Peng et al., 2023] combines local knowledge bases with automatic feedback. However, its performance is constrained by delays in updating the knowledge base. To improve timeliness, researchers have utilized search engine APIs, such as RAGAR [Khaliq et al., 2024] with the DuckDuckGo Search. However, these methods often focus on aggregating multiple pieces of evidence into a single decision-making process without independently analyzing each piece, leading to insufficient utilization of the available evidence. This limits their ability to explore and validate contributions of evidence in fact-checking tasks fully.

2 https://serpapi.com/manage-api-key

Evidence-Based Interpretable Fact-Checking. Evidence-based fact-checking methods verify claims by retrieving relevant evidence, typically using external knowledge sources like knowledge graphs or document fragments [Shang et al., 2022; Wang and Shu, 2023; Zhao et al., 2023]. With the advent of LLMs (e.g., GPT), generative methods have gained traction in providing transparent, human-like explanations. These methods combine extractive and abstractive summarization to extract key information and generate coherent, contextually connected explanations, thereby improving interpretability [Atanasova et al., 2020; Kotonya and Toni, 2020]. However, most existing methods lack fine-grained truthfulness criteria to evaluate claims systematically. This absence of structured guidance limits the reliability of these methods, particularly when dealing with complex or ambiguous claims where precise decision-making is crucial.

In summary, fact-checking methods using LLMs and RAG improve accuracy and interpretability by integrating external knowledge and generating human-like explanations. However, they struggle with insufficient evidence utilization, finegrained truthfulness analysis, and timely updates. To bridge these gaps, we propose a novel method that optimizes evidence utilization and refines verification criteria for greater accuracy and robustness in real-world applications.

3 Our EVICheckMethod

Task Definition. The task is to perform automated factchecking of claims by evaluating their truthfulness using a multi-step reasoning process. Given a claim x, the goal is to determine its validity by retrieving relevant evidence, performing reasoning, and providing a conclusion along with an explanation. This task can be formally defined as follows:

where x is the input claim to be validated, yˆ is the truthfulness judgment (True, False, or Half), and e is the corresponding explanation. The function fvalidate(x) encompasses all the steps, including question generation, evidence retrieval, reasoning, and final aggregation of results.

As shown in Figure 2, the EVICheck method has two main modules: evidence acquisition with preliminary reasoning and combined verification based on fine-grained truthfulness criteria. The first module follows a four-step loop: generating verification questions, selecting the best one, retrieving relevant information for preliminary reasoning, and generating new questions to gather additional evidence. This process ensures comprehensive evidence collection. The second module, combined verification, integrates the collected evidence and uses fine-grained truthfulness criteria for structured evaluation, enabling accurate decision-making. The algorithm of our method is shown in Algorithm 1.

3.1 Evidence Acquisition &Preliminary Reasoning

Element Extraction and Question Generation

In this step, we aim to identify the core elements of the claim and generate verification questions from multiple aspects of the claim, as shown in Step 1 of Figure 2. For claim x, we first extract core elements, identifying key facts and verifiable information. This step ensures that the subsequent verification questions {qi}mi are focused on the aspects of the claim that require validation, preventing the process from diverging to irrelevant details. Based on these core elements, we generate targeted verification questions from multiple perspectives, ensuring alignment with the claim's content and verifiability through reliable evidence (e.g., official documents, news reports, or eyewitness accounts).

Best Question Selection

In this step, the goal is to select the most relevant validation question from the set of generated questions for retrieval, ensuring that the question chosen aligns closely with the claim's context and is effective for fact-checking, as shown in Step 2 of Figure 2. Once validation questions are generated, the large language model fselect selects the most relevant question qbest from the set {qi}mi for retrieval. The model assigns a relevance score to each question based on predefined evaluation criteria, selecting the question with the highest score:

where argmax selects the question qi with the highest score fselect(qi, x), i.e., the most relevant question to the claim x.

The rationale behind choosing the best question includes reducing noise, improving efficiency, and addressing context length limitations. By choosing the most relevant question, irrelevant or noisy queries are avoided, which enhances the accuracy of verification and reduces unnecessary API calls. This approach conserves computational resources and improves overall efficiency. Additionally, pre-selecting the best question ensures the context window stays within its limits, allowing the model to process all relevant information.

Furthermore, in practice, we observed the following deficiencies in LLMs when selecting optimal questions: First, in causal relationship claims, the models tend to focus on the veracity of events A and B while neglecting the critical importance of their causal linkage. Second, when handling specific claims, the models tend to rely on official sources for evidence retrieval but struggle to obtain useful information for statements that are informally recorded or released through unofficial channels. Finally, for claims involving multimodal information such as images or videos, the models attempt to retrieve the corresponding multimodal data directly but often fail. To address these issues, we designed a series of special prompt strategies to assist the models in more accurately selecting and generating appropriate verification questions.

Information Retrieval & Preliminary Reasoning

In this step, the objective is to retrieve relevant information from external sources based on the selected question and then conduct preliminary reasoning to form an initial verification, as shown in Step 3 of Figure 2. After selecting the optimal question qbest, the search engine API is used to retrieve web content {wi}ni=1 related to the selected question. Subsequently, the model freasoning is used to perform integrated reasoning and summary analysis on the retrieved web content, generating a preliminary conclusion yˆ1 and reasoning process e1:(yˆ1, e1) = freasoning({wi}ni=1 , qbest, x). (3)

Each retrieved piece of information is analyzed individually, extracting key evidence to make a preliminary judgment. The overall preliminary reasoning result is then presented in a structured format containing the following elements:

• Background Information: Provides context to aid in understanding the background of the statement.
• Evidence: Lists key information and data extracted from each source.
• Causal Relationship: Analyzes the causal logic of the statement, assessing its rationality and consistency.
• Conclusion: Makes a preliminary verification regarding the truthfulness based on the analysis and evidence.

Loop Inference and Validation

In this step, the goal is to refine the verification process by repeatedly generating new validation questions based on the claim and the current evidence and reasoning, as shown in Step 4 of Figure 2. After the preliminary inference, new verification questions {q′ i}mi are generated by combining the statement x with the previous inference results yˆi-1 and ei-1. The generation of new questions is centered around the claim

Algorithm 1 Multi-Round Reasoning and Validation

Input: claim x, the number of iterations max loops. Output: Final Prediction yˆ, Explanation e. {qi}mi ←GenerateQuestions(x) qbest ← SelectBestQuestion({qi}mi , x) EvidenceSet ← [] counter ← 0 while counter < max loops do w ← RetrieveWebContent(qbest) yˆcurrent, ecurrent ← Reasoning(w, qbest, x) EvidenceSet.append({yˆcurrent, ecurrent}) {q′i }mi ←GenFollowQ(yˆcurrent, ecurrent, x) qbest ← SelectBestQuestion({q′i }mi , x) counter ← counter + 1 end while (yˆ, e) ← CombinedValidate(EvidenceSet, S) return yˆ, e

rather than being entirely dependent on the questions or reasoning results from the previous round. This strategy avoids the issue of subsequent reasoning deviating from the core due to an initial question that is off-target. Each round of question generation ensures comprehensive validation by considering both the statement's key elements and prior reasoning results.

Once the loop counter reaches max loops, the loop terminates and proceeds to the next module to start the final step.

3.2 Evidence Aggregation and Combined Verification

In this step, the goal is to aggregate the evidence from multiple rounds of reasoning and validation and then combine them to make a final, accurate verification, as shown in the Final Step of Figure 2. The process begins with several rounds of reasoning, where question-reasoning result pairs {(yˆi, ei)}mi=1 , which include validation questions and their corresponding background information, evidence, causal analysis, and preliminary conclusions-are generated and iteratively refined. After multiple rounds, these evidence and preliminary judgments are aggregated and validated, guided by a set of fine-grained truthfulness criteria S, which define evaluation rules for conclusions (True, False, Half) and provide clear guidelines for the model, as shown in Table 1. The conclusion yˆ and explanation e are derived, ensuring both high credibility and accuracy.

To further improve the accuracy of the model's verification, we fine-tuned the model fcombined verification using a supervised learning approach on the Xtrain dataset. In the fine-tuning process, we used the LLaMA-Factory framework 3 [Zheng et al., 2024] along with the LoRA (Low-Rank Adaptation) method [Hu et al., 2022], freezing all parameters except for the LoRA adapters. The fine-tuning was performed for three epochs. The objective is to maximize the accuracy of the final prediction by minimizing the loss function, which quantifies the difference between predicted and actual outcomes:

3 https://github.com/hiyouga/LLaMA-Factory/tree/main

where θ represents the model parameters, x is the input statement, yˆi and ei are the predicted conclusions and explanations from the previous reasoning rounds, Xtrain is the training dataset, and L is the loss function that quantifies the difference between the model's prediction and the ground truth. The goal is to minimize the loss, ensuring that the model's final prediction yˆ is as accurate as possible based on the aggregated evidence and the criteria S.

4 Experimental Setting

Dataset. We adopt the English fake news dataset RAWFC [Yang et al., 2022] for experiments. The dataset was created by collecting claims from Snopes 4 and retrieving the relevant raw reports. It includes three categories of labels: True, False, and Half, with each data entry provided with a manually annotated "golden label" explanation. The data distribution is shown in Table 2.

Experimental Details. The experiments were conducted using three different models: GPT-3.5, GPT-4, and the Llama-3-8B-Instruct model 5 . To make more accurate verification we fine-tuned Llama-3-8B-Instruct model as fcombined verification. SerpApi was used as the search API in the experiment. To reduce computational overhead was conducted based on M = 5 validation questions and performing N = 2 rounds of loop inference. Although we found that increasing the number of training rounds and questions could lead to improved results, further details are shown in Figure 5.

Baseline. We employ two categories of baselines:

· Supervised methods

1. GenFE [Atanasova et al., 2020]: multi-task explanation generation.
2. SentHAN [Ma et al., 2019]: hierarchical attention over sentence-level evidence.

4 https://www.snopes.com/

5 https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct

1. SBERT [Kotonya and Toni, 2020]: interpretable fact-checking for public health claims.
2. CofCED [Yang et al., 2022]: cascading evidence distillation for report-based detection.

· LLM-based methods (GPT-3.5)

1. CoT [Wei et al., 2022]: chain-of-thought prompting for complex reasoning.
2. Standard Prompt [Brown et al., 2020]: few-shot GPT-3 prompting.
3. Hiss [Zhang and Gao, 2023]: hierarchical hinting for statement breakdown.
4. RAFTS [Yue et al., 2024]: retrieve-and-compare using LLM synthesis.

Evaluation Metrics. To comprehensively evaluate the model's performance, three evaluation metrics were used: Macro-average Precision (P), Recall (R), and F1-score. In addition to the metrics above, we used the confusion matrix to analyze further the model's performance in the three-class classification task. The matrix displays the relationship between true labels and predicted labels, helping identify misclassification patterns. By analyzing it, we gain insights into the model's performance, particularly in recognizing categories with significant misclassification issues, guiding future optimizations, as shown in Figure 3.

5 Experimental Results

This section presents the experimental results of the proposed method, starting with an evaluation of overall performance, including comparisons with existing approaches. Ablation experiments assess the impact of individual components on performance, while manual evaluation confirms the method's practical effectiveness. Finally, a detailed analysis of the validation process and a typical case is provided.

5.1 Performance Outcome

We conducted experiments separately using GPT-3.5 and GPT-4, and then replaced the fcombined verification model with the fine-tuned Llama-3-8B-Instruct model for further experimentation. The experimental results of our method, comparing them with the current SOTA results, are shown in Table 3.

The experimental results demonstrate that under the conditions of using GPT-3.5 and fine-tuned Llama 3, our approach outperformed traditional SOTA methods across all evaluation metrics. When using GPT-4 and fine-tuned Llama 3, compared to traditional SOTA methods, our method improved accuracy by 3.5%, precision by approximately 8.6%, and F1 score by about 6.0%. These results indicate a significant improvement in the accuracy and reliability of verification in the fact-checking task. The observed performance improvements can be attributed to several key innovations in our approach. First, we incorporated a preliminary reasoning step after each information retrieval, ensuring that every piece of evidence is fully utilized. Second, we introduced a set of fine-grained truthfulness criteria to guide the model in making final verification, which enhanced its performance in determining the veracity of statements. Finally, we fine-tuned the Llama 3 model specifically to perform better in fact-checking tasks, optimizing its ability to discern factual accuracy. Together, these innovations contributed to the significant performance gains observed in our experiments.

We analyzed the biases of GPT-4 and the fine-tuned Llama 3 model during verification (Figure 3). We found that GPT-4 exhibited a negative bias, such as a higher occurrence of False judgments and a tendency to classify Half claims as False, reflecting GPT-4's cautious bias, which influenced the model's performance. In contrast, the fine-tuned Llama 3 model displayed a more balanced bias, particularly improving accuracy when judging Half and True statements. These observations suggest that through fine-tuning, the Llama 3 model can better adapt to multidimensional verification tasks when handling complex statements, thereby enhancing the stability and reliability of the inference process.

5.2 Ablation Study

An ablation study was conducted to assess the impact of different configurations on EVICheck, as shown in Figure 4.

Effect of Model Fine-Tuning. We first evaluated the effect of fine-tuning on EVICheck. Fine-tuning the Llama 3 model using Xtrain improved final prediction accuracy by 3.47% compared to the untuned model, highlighting fine-tuning's key role in enhancing performance.

0.4

Figure 4: Experimental results of ablation study.

Effect of PreliminaryReasoning. In the original setup, the model performed preliminary reasoning-generating background, extracting evidence, analyzing causal relationships, and concluding-before combining and validating the results at the final verification node. Without preliminary reasoning, the model only gathered evidence, leading to a 7.77% performance drop. This performance drop highlights the preliminary reasoning's key role in integrating multi-round inference and improving accuracy.

Effect of Fine-Grained Truthfulness Criteria. We tested the influence of fine-grained truthfulness criteria. Removing these, along with preliminary reasoning, led to poor performance. However, incorporating our fine-grained prompt words boosted EVICheck performance by 10.08%, showing that detailed criteria significantly enhance the model's ability to perform accurate verification.

5.3 Optimal Solution for the Number of Loop Rounds and Verification Questions

To explore the optimal solution for the number of loops and verification questions, we randomly selected 12 samples (4 from each category). A total score of 12 points was assigned, with 1 point awarded for correct answers, 0 points for incorrect answers, and a 0.3-point deduction for significant errors (such as true-false or false-true discrepancies). The final score rate was computed as the ratio of the obtained score to the total score.

Number of Loop Rounds. As shown in Figure 5 (left), the number of loops was increased while fixing 2 validation questions per round. It was observed that the score rate increased with the number of loops. However, each additional loop also reduced the inference speed by 50% and increased the API call error rate. Therefore, considering both performance and efficiency, we set the number of loops to 2.

Number of Verification Questions. As shown in Figure 5 (right), the impact of varying the number of validation questions on model accuracy. With the number of loops fixed at 2, an inverted U-shaped curve in accuracy was observed.

Specifically, the score rate was highest when 6 questions were posed. However, as the number of questions increased, the score rate decreased after reaching its peak. Notably, when only 1 question was asked, the score remained relatively high. We conducted a deeper analysis. Specifically, we analyzed performance for Half, True, and False declarations across different question numbers:

1. True: As shown in Figure 6 (left), the score for True decreased as the number of questions increased. With only 1 question, queries were broad, making counterevidence harder to find, leading to more True classifications. With more questions, the queries became more specific, reducing the number of True classifications.
2. Half: As shown in Figure 6 (right), the score for Half statements followed by an inverted U-shape. More questions made the queries more specific, helping the model assess complex statements better. However, too many questions introduced noise, reducing accuracy.
3. False: Accuracy for False statements remained at 100%, demonstrating the model's robustness in identifying false statements.

Additionally, fluctuations in experimental results may be influenced by data size, and future studies with larger sample sizes could further validate these conclusions.

5.4 Human Evaluation

To evaluate the performance of EVICheck in generating judgments and explanations, three experts in NLP and public opinion analysis manually reviewed the results. Each expert rated the judgments and explanations on a scale from 1 (poor) to 5 (good) based on predefined criteria.

Twelve data points were randomly selected, and two types of explanations were compared: those from the RAWFC dataset and those generated by EVICheck. Experts rated both based on the following criteria:

• Coverage: Whether the explanation covers the key information needed for judgment verification.
• Readability: Whether the language of the explanation is concise and well-structured.
• Accuracy: Whether the explanation accurately reflects the data or facts and whether the reasoning is correct.
• Conciseness: Whether the explanation is succinct and contains only necessary information.
• Credibility: Whether the explanation is reasonable and convincing.

To reduce bias, we randomized the order of the explanations in each questionnaire. Expert ratings were then summarized, and the comparison of scores for each data point across criteria is shown in Table 4.

The results showed that EVICheck received a lower score in Conciseness, as manual explanations tend to be more concise. EVICheck includes reasoning steps for each loop, providing transparency but adding unnecessary details.

5.5 Case Study

This case study demonstrates our method's application in verification tasks. The target claim is shown in Figure 7.

The First-Round Verification Questions. Figure 8 shows five generated questions, with the most relevant selected for web retrieval. GPT-4 then performs preliminary inference using the retrieved information.

The Second-Round Verification Questions. As shown in Figure 9, five questions are posed based on the claim and the optimal question from the first round. The most relevant question is selected for web retrieval, and GPT-4 provides further inferences and answers.

Combined Verification and Final Judgment. Figure 10 compiles the preliminary inferences from both rounds and their evidence, then applies combined verification to make the final judgment based on fine-grained truthfulness criteria.

6 Conclusion

In this paper, we propose EVICheck, a method that enhances automated fact-checking by addressing the limitations of insufficient evidence utilization and the lack of clear verification standards. EVICheck analyzes each piece of evidence independently, conducts detailed reasoning, integrates the results, and applies fine-grained truthfulness criteria to improve reliability. Experiments on the RAWFC dataset show that EVICheck outperforms existing approaches, demonstrating its potential. Nevertheless, it still struggles with informal social-media statements and multimodal claims. Future work will integrate additional social-media APIs and strengthen multimodal reasoning. In conclusion, EVICheck offers an innovative, practical solution for combating fake news.

Acknowledgments

This work was supported by the Joint Fund Key Program of the National Natural Science Foundation of China (No. U23B2029), by the Beijing Natural Science Foundation (No. 4252035), by the Key R&D Program of Zhejiang (No. 2024C01164), by the Zhejiang Provincial Natural Science Foundation of China (No. LMS25F020005), by the National Natural Science Foundation of China (No. 62002027), by the Fundamental Research Funds for the Central Universities (No. CUC25SG013), and by the Public Computing Cloud, CUC.