Can We Verify Step by Step for Incorrect Answer Detection?

1 Introduction

Recent development in large language models (LLMs) [OpenAI, 2023; Bubeck et al., 2023; Yang et al., 2024a] has showcased their remarkable aptitude for tackling diverse downstream tasks. Given several demonstrations with reasoning steps, LLMs exhibit a formidable capability to address reasoning tasks, commonly referred to as chain-ofthought (CoT) [Wei et al., 2022]. There have been significant advancements in improving the reasoning abilities of LLMs in terms of end-task performance, such as intricate math word reasoning [Shum et al., 2023; Diao et al., 2023;

∗Corresponding author.

Zheng et al., 2023; Yu et al., 2023; Xu et al., 2024; Yang et al., 2024b], and tasks that involve extensive search and tactical planning [Yao et al., 2023]. Another line of research lies in the analysis of the reasoning steps themselves with or without human-annotated ones [Clinciu et al., 2021; Prasad et al., 2023; Xia et al., 2024].

[Wei et al., 2022; Ye and Durrett, 2022] have manually inspected whether reasoning steps align with the correctness of ultimate answers in both correct and incorrect answer groups through case studies. Moreover, [Prasad et al., 2023; He et al., 2023] find that elevating the quality of the rationales could potentially enhance task performance. Since aggregate task performance is a summation of the accuracy of final answers across individual instances, these observations offer an initial qualitative scrutiny of the reasoning chains and their impact on the accuracy of the final predictions. Despite these insights, there remains a shortage of quantitative proof to substantiate whether the evaluation of reasoning chains can reliably affirm the validity of the final outcomes. To bridge this gap, we introduce the R2PE (Relationship of Rationales to Performance Evaluation) benchmark, a test bed designed to quantitatively investigate this question.

The susceptibility of LLMs to generate incorrect information has been underscored by [OpenAI-Blog, 2022; Zhao et al., 2023]. Meanwhile, [Ye and Durrett, 2022] associate false predictions with nonfactual explanations. Furthermore, [Wei et al., 2022] have demonstrated that, even in arithmetic reasoning tasks, incorrect rationales can occasionally yield correct outcomes, and a variety of errors may occur at intermediate reasoning steps. To perform a qualitative evaluation of rationales to validate final predictions, the R2PE benchmark is established to integrate a diverse spectrum of reasoning tasks, covering mathematical, commonsense, physical, and textual reasoning (including fact verification and open-domain question answering). These tasks extend across domains such as mathematics, common knowledge, physics, literature, and general world knowledge. Responses are collected from six distinct LLMs to promote a broad generality of the findings. The characteristics and the creation steps of the R2PE benchmark are illustrated in Figure 2.

Each evaluated question or claim begins with the generation of multiple reasoning chains, leading to an answer aggregation based on majority voting [Wang et al., 2022] after extraction and manual cleaning. Once the final outcomes are labeled as true or false using an exact match, our goal is to derive a discernibility score (DS), which is intended as an indicator of the credibility of the final answer. To fully exploit the information contained in all reasoning chains, we present a process discernibility score (PDS) that substantially exceeds the answer discernibility score (ADS) baseline that counts the number of the same answers. As shown in Figure 1, PDS can detect conflicting information about the actor and predict the incorrectness of the final answer. Figure 2 presents a succinct overview of both our benchmark and our method.

In summary, our contributions are as follows:

• We propose R2PE, the first benchmark that quantitatively assesses the relationship between reasoning chains and end-task performance across a spectrum of reasoning tasks, multiple domains, and an array of LLMs.
• We introduce a process discernibility score (PDS) framework that aggregates the information in different reasoning chains for CoT verification.
• Comprehensive experiments of PDS reveal its superiority over the answer discernibility score (ADS) in predicting final answer correctness, leading to consistent increases in the F1 score and AUC-PR.
• Further experiments show the effectiveness of our approach in improving open domain question answering

performance when combined with verify-and-edit [Zhao et al., 2023].

2 R2PE Benchmark

R2PE serves as a comprehensive platform for verifying LLMgenerated answers in CoT reasoning, which is meticulously constructed with the following critical characteristics: (i) Assessability: every instance within R2PE can be verifiable as true or false based on a certain criterion. (ii) Task Diversity: The benchmark should encompass a wide range of reasoning datasets, featuring various answer formats, and spanning different task categories across multiple domains. (iii) Generalizability across LLMs: Responses should be elicited using different LLMs to ensure broad applicability. (iv) High Quality: To minimize instances where correct answers are inaccurately labeled as false due to extraction failures, it is crucial that answers are precisely extracted from responses.

2.1 Construction Process

As shown in Figure 2, the creation of R2PE consists of the following five steps:

Data Collection. we utilize a total of eight datasets including benchmarks for mathematical reasoning such as GSM8K [Cobbe et al., 2021] and MATH [Hendrycks et al., 2021], common sense reasoning tasks like StrategyQA [Geva et al., 2021] and Play dialogue from BIG-bench collections [Srivastava et al., 2022], physical reasoning tasks (Physics from BIG-bench collections [Srivastava et al., 2022]); fact verification (FEVER [Thorne et al., 2018]) and open-domain question answering (HotpotQA [Yang et al., 2018], 2WikiMultihop [Ho et al., 2020]). Each selected dataset meets the criterion of assessability through verifiable answers. These tasks not only vary in answer formats, including numerical, yes/no, multiple choice, and free form, but also span an array of domains such as mathematics, commonsense, literature, physics, and general world knowledge (see Appendix A.1).

Response Generation. For each question (or claim in the FEVER dataset) Q, the LLM is prompted using CoT to produce n responses (R1,R2, ...,Rn), respectively. The full prompts are given in Appendix A.3. The responses are aggregated from six distinct LLMs to ensure the generalizability: text-davinci-003, GPT-3.5-turbo [OpenAI-Blog, 2022], its instruct-trained variant GPT-3.5-turbo-instruct, Gemini Pro, Mixtral-8x7b, and mistral-medium. A concise exposition along with detailed settings, is deferred to Appendix A.2.

Extraction. Rationale ri and the corresponding anwser ai need to be isolated from each response Ri. During the pilot extraction period, we use the prompt cue "The answer is" as a delimiter to segregate responses. We find that almost all LLMs yielded some outputs that did not conform to the expected answer format. Given the varying responses styles across different LLMs, the answer trigger words are identified for different LLMs and datasets after observing the original responses to facilitate extraction (see Appendix A.3). This strategy effectively segments most of the responses.

Manual Cleaning. To maintain a high-quality benchmark, manual inspection and cleaning are performed to handle unusual cases. For atypical responses that deviate from the recognized patterns, we either manually separate the response to derive the answer or assign a special marker when separation is not feasible. A detailed description of the extraction and manual cleaning procedure is provided in Appendix A.3.

Labeling. Upon completion of extraction and manual cleaning,R = {R1 = (r1, a1), ...,Rn = (rn, an)} are collected for each question Q. Then we can aggregate the outputs to obtain the final answer by the majority votes: a = argmaxai freq(ai), where freq(ai) denotes the frequency in which ai appears. The final result a of each question Q is then compared with its ground truth; A match results in a "TRUE" label, while a mismatch is assigned as "FALSE".

2.2 R2PEOverview

Our R2PE benchmark comprises a diverse collection of data derived from six LLMs on eight reasoning tasks. These tasks encompass a variety of types, answer formats, and domains, making the benchmark rich and comprehensive. In total, it includes approximately 38.3K instances with around 20.2K labeled as TRUE and 18.1K as FALSE. The structure of R2PE allows for the organization of instances into 45 distinct subsets based on the original dataset and the LLM used to generate responses. Hereinafter, subsets are denoted as (dataset name from LLM name), with detailed statistics and concrete examples available in Appendix B.

Each instance e within R2PE incorporates a series of structured data fields: question or claim Q, the associated dataset name, the queried LLM name, five responses alongside their rationales and answers R = {R1 = (r1, a1), R2 = (r2, a2), ...,R5 = (r5, a5)}, the final output a, the groundtruth answer of question Q, and the label L ∈ {T, F} to indicate whether the generated answer a matches the ground truth. Detailed explications of the constituent data fields and illustrative examples are presented in Appendix B.

We aim to predict the correctness of final outcomes based on the intermediate reasoning steps. The response set R serves as an input for predicting the label L for every instance e. We introduce a numerical criterion, the discernibility score (DS), to encapsulate the quality of R.A low DS might suggest a potential mismatch between the final output a and the ground-truth answer. Hence, we will classify the example as false: ˆL = F , if its DS falls below a certain threshold H.

3 Process Discernibility Score

3.1 The Fallacy of Answer Agreement

From [Wang et al., 2022; Zhao et al., 2023], the degree of consensus among answers, is posited as a diagnostic tool to gauge the instances in which LLMs may make wrong predictions. We refer to the answer agreement number as Answer Discernibility Score (ADS) hereinafter. A high ADS often suggests a high likelihood that the proposed answer is correct. If the ADS is below that midpoint (i.e., the threshold H = n/2), ˆL = F is predicted.

However, LLMs may generate substantially consistent but incorrect answers in reasoning tasks. Figure 1 showcases one such example, where the LLM is queried about the nationality of an actor from "Hostage for a Day". The LLM gives the answer "American" four times and "Canadian" once. Here, the ADS stands at 4, which erroneously points to a seemingly accurate but ultimately false consensus. This example serves to warn against overreliance on ADS as the sole indicator of the veracity of the final answer in reasoning tasks.

3.2 Process Discernibility Score

In our R2PE benchmark, each response Ri constitutes a rationale ri paired with an answer ai. The ADS, however, limits its analysis to the answers (a1, ..., an), which takes up only a small proportion of the information contained in R. Inspired by [Yoran et al., 2023], a DS that makes use of rationales would be preferable.

To quantify the similarity between two responses, we employ Pairwise Process Supervision Score PPSS(Rk, Rl) to measure the degree to which the content of Rl is both included in and opposes that of Rk. PPSS is confined within the interval (-1, 1), where a positive score indicates a higher level of similarity relative to contradiction. For a set of n responses, we can calculate the average of PPSS across all ordered response pairs, culminating in the Process Supervision Score (PSS): PSS = meani̸=jPPSS(Ri,Rj). This allows PSS to evaluate the information across all responses. To underscore the final answers, we get

wherein ADS = 2(ADS - n/2)/n represents the normalized ADS, adjusted to align with PSS's range. The overview of PDS is shown in Figure 2.

Subsequently, we classify instances where the PDS falls below the threshold H as ˆL = F , whereas all remaining instances are designated as ˆL = T .

Compared to ADS, our PDS is capable of overseeing reasoning processes as well as doing implicit answer checking (ADS term in Equation (1)). As depicted in Figure 1, the PDS can detect contradictive information about the name of the actor playing a Russian hostage taker in "Hostage for a Day" among multiple reasoning chains (highlighted by red) and give a correct label ˆL = F , while the ADS only assesses the agreements among the answers.

3.3 Experimental Setup

We will detail the experimental setup in this section. Dataset. All subsets of the R2PE benchmark.

Baselines. From [Wang et al., 2022; Zhao et al., 2023], ADS can act as the baseline in our setting. We also include SelfCheckGPT [Manakul et al., 2023] and HaloCheck [Elaraby et al., 2023] as our baselines.

Metrics. We will employ the F1 score and AUC-PR as our evaluation metrics. As we aim to detect potential incorrect answers, False labels are treated as positive. Therefore, true positives are examples with L = F and ˆL = F. We do not compare with SelfCheckGPT and HaloCheck in terms of F1 score, as they do not provide thresholds for prediction.

PDS Implementation. For PPSS(·, ·), we adopt the SAUMMAC zero-shot model [Laban et al., 2022],which is a consistency detection method for text summarization that leverages the out-of-the-box natural language inference model to compute pairwise entailment score. As suggested by [Laban et al., 2022], probabilities of sentence-level entailment and contradiction are used. To detect all potential inconsistencies in any sentence, we use min operation to aggregate across sentence pairs:

where ai, bj are sentences of two responses Rk,Rj correspondingly, ent represents the probability of entailment, con is the probability of contradiction, the min is taken over all sentences of Rj, and max is taken over all sentences of Rk. Note that the PPSS metric lacks symmetry; however, this is not a problem in our implementation because the PSS calculation averages across all possible ordered response pairs.

Threshold Selection. As ADS ∈ {0, 1, 2, ..., 5}, threshold H is set to be the midpoint H = 2.5. Another reason for this selection is that ADS > 2.5 means the majority agrees, which aligns the choice in [Zhao et al., 2023]. The range of this PPSS spans in (-1, 1). A negative value implies the potential presence of conflicting information. Consequently, it is reasonable to set the threshold H for the PDS to be 0. Following [Li et al., 2023], the threshold of ADS is raised to 4.5 for discriminative tasks (yes/no, multiple-choice questions), and the PDS threshold is adjusted to 0.4 according to Equation (1). Our choice of threshold H is not task-specific. We regard this as a benefit because held-out data for threshold tuning may not always be available in the real scenario. A detailed discussion is given in Appendix C.3.

4 Results and Analysis

4.1 PDS Substantially Outperforms All Baselines

As shown in Table 1, Across all 45 subsets in our R2PE, PDS consistently improves ADS performance, yielding an average improvement of 5.10% in terms of F1. Although ADS demonstrates greater precision, it suffers from significantly lower recall, indicating its propensity to overlook numerous examples with L = F. In contrast, PDS adeptly balances precision and recall, culminating in superior overall performance relative to ADS (see Table 1, 2, and Appendix C.2). From Table 2, PDS improves ADS in terms of AUC-PR in the majority of cases and yields comparable results (less than 1% decrease) in the remaining ones. On average, PDS results in 3.01% absolute improvement in AUC-PR. Notably, ADS is a fairly strong baseline compared with SelfCheckGPT and HaloCheck in our setting, and PDS can further enhance the performance of ADS. We believe that the performance of SelfCheckGPT [Manakul et al., 2023] and HaloCheck [Elaraby et al., 2023], which lags behind ADS, can be attributed to their operation in different settings and the lack of explicit verification of the majority voting proportion of the final answer, which highlights the importance of jointly considering both the reasoning process and the final answer to validate CoT, demonstrating the superiority of PDS. For discriminative tasks, ADS has relatively low F1 scores and AUC-PR. This may be attributed to the fixed set of answers (e.g., yes or no) in these tasks, which inherently exhibit higher consistency, making verification based solely on answer agreement challenging. In contrast, PDS can relieve this issue by evaluating reasoning processes.

PDS demonstrates a stronger capability to verify the accuracy of the predictions for free-form questions than ADS, with notable average improvements of 7.00% in HotpotQA and 9.80% in 2WikiMultihop. This superior performance can be attributed to the process-oriented verification approach employed by PDS. By addressing the challenges associated with indirectly assessing answer consistency in free-form questions, PDS proves to be more effective in ensuring accuracy. It is important to note that the average improvement metrics provided in Table 1 serve primarily as an indicator of the extent to which PDS enhances performance over ADS. However, a direct comparison of F1 scores and AUC-PR between different subsets within R2PE is not recommended. This is because the efficacy of the F1 score as a comparative metric is limited by variations in the ratio of TRUE and FALSE instances across different subsets.

To meticulously evaluate the efficacy of the PDS, we delve into the (HotpotQA from GPT-4) subset, partitioning it into two sets: one where answers fail to reach a consensus greater than half, and its counterpart. We then label the examples with an ADS less than the threshold H as ˆL = F, and all the others as ˆL = T. As shown in Figure 3 (left), while the ADS proficiently segregates the examples with L = T, its discernment for the "False" category is less acute. On the contrary, the PDS (Figure 3 middle) strikes a better balance between precision and recall. Eliminating the implicit answer verification component in Equation (1) results in diminished precision (Figure 3 right, discussed in Section 4.2).

4.2 Ablation Study

To understand the contribution of PDS' components, we have carried out an extensive ablation study involving the following variants (See Appendix C.4):

PDS-ADS. we remove the ADS in our Equation (1), and the alternative of PDS becomes PDS - ADS = PSS.

PDS w/o ans. The original PSS considers all PPSS(Ri,Rj), where each response Ri equals a reasoning path ri and an answer ai. As the information from the answers ai' is already used to obtain the final answer a, we can discard them and only compute the entailment score among rationale pairs PPSS(ri, rj), that is, PDSw/o ans = (ADS + meani̸=j P P SS(ri, rj ))/2.

PDS-avg. We can change the aggregation operation min of PPSS(·, ·) (Equation (2)) by taking the average. Then P P SS(A, B) = avgj maxi(ent(ai, bj ) - con(ai, bj )).

PDS-HaloCheck;PDS-SelfCheckNLI. HaloCheck [Elaraby et al., 2023] and SelfCheckNLI [Manakul et al., 2023] can be alternatives to PSS, denoted by PDS-halocheck and PDS-selfcheckNLI, respectively (see Appendix C.5).

As shown in Table 3 and 4, our PDS implementation gains the best results.

4.3 PDS for Improving DownstreamPerformance

VE [Zhao et al., 2023] post-edits reasoning process using external knowledge (see Appendix D). VE first checks the answer agreement and then edits the reasoning chains based on a retriever if ADS < H. That is to say, VE selects potential incorrect samples by ADS. Our PDS can be incorporated into the VE framework, dubbed "VE + PDS".

Experiments are carried out on (HotpotQA from GPT-4) and (2WikiMultihop from GPT-4), utilizing ground-truth supporting contexts along with distractor paragraphs as the retrieved documents to negate the influence of the retrieval system. Our baseline is VE, and the standard prompting and CoT-SC are also included as references.

Figure 4 illustrates that integrating our method with the VE framework leads to enhanced accuracy for both datasets.

Specifically, VE+PDS leads to a notable 2.9% absolute increase in accuracy for HotpotQA, and 2.8% for 2WikiMultihop. Given that the retrieval system in our experiments is assumed to be near-perfection, the observed improvements in performance are attributed to the more effective answer verification through the substitution of ADS with PDS. This success is likely a consequence of the PDS's capability to supervise processes across multiple reasoning chains, leading to better selection of all potential incorrect predictions.

4.4 Error Analysis

We also conduct an error analysis on HotpotQA using a GPT4 subset of R2PE. Specifically, we find 68 cases where PDS correctly identifies the falsehood of answers that ADS does not. In these cases, all the labels are "False" and the answers are consistent, meaning ADS could not verify the answers due to its reliance on answer consistency alone. In contrast, PDS is able to identify potential inconsistencies among different reasoning chains, even when the answers are consistent. However, we also find 53 cases where the labels are "True" and the answers are consistent, where ADS succeeds but PDS fails. After a careful manual inspection, we classify these cases into the following categories: I. Conflicting information among chains despite consistent and correct answers. II. Excessive information that is correct in some chains, leads to inconsistency among chains. III. Imperfections of the NLI models even when there is no conflicting information. IV. Different solutions with diverging chains lead to the same correct answers. We present the number of cases in each category in Table 5. See discussion in Appendix E.

5 RelatedWork

Extensions of CoT. CoT [Wei et al., 2022] improves LLM performance by decomposing reasoning into steps via fewshot examples. Refinements include sampling-based decoding [Wang et al., 2022], automated exemplar selection [Kojima et al., 2022; Zhang et al., 2022], and hybrid strategies [Zou et al., 2023]. Task-specific adaptations like PHP [Zheng et al., 2023] for mathematical reasoning and verify-and-edit (VE) [Zhao et al., 2023] focus on improving reasoning accuracy. While much work emphasizes end-task metrics, our study explores reasoning quality-outcome linkage, verifying predictions through chain analysis. We will also extend PDS to Long-CoT LLMs [Jaech et al., 2024], which benefit from sampling-based verification [Wen et al., 2025], and propose potential extensions to test-time scaling methods [Wu et al., 2025] via chain consistency analysis.

Reasoning Chain Quality Evaluation. Rationale analysis is key for understanding AI performance and limitations[Golovneva et al., 2022; Prasad et al., 2023; He et al., 2023]. Rationale evaluation methods are either referencebased, comparing against a gold standard [Clinciu et al., 2021; Saparov and He, 2022], or reference-free, using metrics like ROSCOE [Golovneva et al., 2022]. While they link high-quality reasoning to better task performance, the direct impact of reasoning quality on prediction accuracy is underresearched. Our study fills this gap with a detailed quantitative analysis of this relationship. There is also research focused on evaluating not only the final predictions but also the reasoning steps in CoT [Xia et al., 2024; Huang et al., 2024; Xu et al., 2025].

Uncertainty Estimation. A range of methods measure LLM uncertainty by token probabilities [Kuhn et al., 2023; Malinin and Gales, 2020; Kadavath et al., 2022]; however, they are inapplicable to R2PE, which does not furnish probabilities for responses. Moreover, they concentrate on short answer sequences, aligning more closely with the standard prompting. In contrast, our work verifies the veracity of CoT answers, which encompasses both the reasoning chains and answer sequences. Different from [Kuhn et al., 2023], the entailment model in our work is used to measure the informational similarities between two responses.

6 Conclusion

To shed more light on the connection between the quality of the reasoning steps and the final outcome, we present R2PE, the first benchmark that quantitatively analyzes whether we can validate the answer veracity by evaluating the reasoning chains on a variety of reasoning tasks across five different domains. Six different LLMs are used in the creation of R2PE to guarantee the generalizability of our findings. We develop the PDS framework by integrating multiple rationales' information to predict the falsehood of the final predictions, which significantly boosts the performance of answer checking on our R2PE benchmark. Furthermore, our approach can be easily combined with the verify-and-edit framework to improve end-task performance.

Acknowledgements

The work was partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (AoE/E-601/24-N)