MathCritique: Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision

1 Introduction

With the rapid development of large language models (LLMs) [OpenAI, 2023; LlamaTeam, 2024], providing supervision that is both scalable and effective for them has become a key challenge (i.e., the scalable oversight problem), especially for complex tasks such as mathematical reasoning [Kenton et al., 2024; Sun et al., 2024]. Training critique models is a promising approach, as they can provide reliable and useful feedback to help actor models identify issues and refine their responses [Welleck et al., 2023; Wang et al., 2024;

† Corresponding authors.

Yao et al., 2024]. However, the open-source community currently lacks a suitable dataset for effectively training critique models and for integrating them into actor models in a principled way during both training and inference.

To bridge this gap, we first propose an automated and scalable framework AutoMathCritique to collect diverse and high-quality step-level critique data without substantial human annotation (Section 3). It consists of three stages: (1) flawed reasoning construction, (2) critique generation, and (3) data filtering. First, we synthesize diverse erroneous reasoning paths using controlled error injection (e.g., specifying error location or content), which also yields hints for critique. Next, annotator models label step-level correctness and provide constructive feedback given the reasoning path and optional hints. Finally, the reasoning model revises its response based on the critiques, and we apply Monte Carlo sampling [Wang et al., 2024] to improve critique quality. A case is illustrated in Figure 1 of Appendix A.

Next, using AutoMathCritique, we create MathCritique76k, a critique dataset containing 76, 321 samples, which is subsequently used to fine-tune a critique model. With the critique model's assistance, the actor model improves its exploration efficiency and reasoning quality at test time, leading to a significant enhancement in final performance (Section 4). Through in-depth analysis, we find that the critique models are particularly effective in difficult queries, and notably, these performance gains scale favorably with increased test-time compute [Snell et al., 2024; Brown et al., 2024].

Motivated by test-time insights, we further propose a critique-in-the-loop self-improvement method (Section 5), which integrates a critique model into the actor's exploration and learning. With critique supervision and increased exploration compute on hard queries, our approach improves exploration efficiency and solution diversity, mitigating tail narrowing [Ding et al., 2024] during iterative training. We validate its effectiveness through extensive experiments and provide further analysis (Section 6).

In summary, our main contributions are:

• We propose AutoMathCritique, a scalable framework for collecting step-level critique data, and use it to build MathCritique-76k.

Stage1: Flawed Reasoning Paths Construction

Stage2: Critique Generation

Figure 1: The overview of AutoMathCritique framework.

• We fine-tune critique models and show that their feedback improves actor reasoning at test time, especially on hard queries and with more inference compute.
• Inspired by test-time insights, we propose critique-in-theloop self-improvement method, which integrates critique models into actor self-training.
• We conduct extensive experiments and analyses to validate our method. The results demonstrate its effectiveness in enhancing exploration efficiency and solution diversity.

2 Preliminaries

In the two-player setting studied in this paper, there are two roles: the actor model and the critique model. Also, there are three tasks [Saunders et al., 2022]: reasoning, critique, and refinement.

In the reasoning task, the actor model πθ is given a problem x and is prompted to generate a response y = πθ(x).

Next, the critique model πϕ performs the critique task, where, given the problem and response, it generates critical feedback c = πϕ(x, y). Notably, if the oracle reward of the response is not given, the critique task consists of two subtasks: the discriminative task and the feedback generation task. The former determines whether the response contains flaws, while the latter generates constructive natural language feedback.

Finally, for the refinement task, given the problem, response, and critique, the actor generates a new response y′ = πθ(x, y, c). Alternatively, we can define direct refinement y′ = πθ(x, y), where the actor provides an improved answer only based on an existing answer, which is also referred to as "self-correction" [Welleck et al., 2023].

3 AutoMathCritique: A Framework for Critique Data Collection

To train critique models that offer step-level supervision and constructive feedback for reasoning, we introduce AutoMathCritique-an automated and scalable framework for collecting critique data. As shown in Figure 1, this framework consists of three main stages: flawed reasoning path construction, critique generation, and data filtering.

Using AutoMathCritique, we create a dataset containing 76, 321 samples named MathCritique-76k. The construction queries are drawn from the training sets of two widely used datasets in the field of mathmatical reasoning: GSM8K [Cobbe et al., 2021] and MATH [Hendrycks et al., 2021b]. The statistics are listed in Table 1, and further details are provided in Appendix C.

3.1 Construction of Flawed Reasoning Paths

To construct a dataset of flawed reasoning paths, we leverage several distinct response generation (RG) approaches that induce diverse errors (i.e., location or specific details), with Llama3-8B [LlamaTeam, 2024] serving as the primary sampling model.

RG1: sampling from scratch. The actor generates a response solely from the query. We use repeated sampling to obtain flawed responses.

RG2: generating error-location-aware response. Starting from a specific step of a correct response, we modify the model's hyperparameters (e.g., increasing temperature) to sample a flawed response. This ensures all preceding steps are exactly the same as the original correct response, while subsequent steps may contain errors.

RG3: introducing detailed mistakes. The model is explicitly prompted to implant mistakes into a correct solution. Inspired by previous work [Lanham et al., 2023; Li et al., 2024], our prompt enumerates common reasoning error types and provides few-shot examples. Once an error is inserted, the model continues reasoning from the erroneous step for up to 16 attempts, until it reaches a flawed answer. Finally, we balance the number of data among different error types in the dataset.

3.2 Generation of Critiques

Step-level critique generation. We enhance quality by checking each step to identify the first error in the solution. Specifically, we employ two methods to generate step-level critique data: (1) We instruct the critique annotator (GPT-4o) to directly identify the location of the first error and provide corresponding feedback. (2) We instruct the annotator model to label step by step, stopping the process once the first error is detected, at which point they provide the corresponding feedback.

Critique generation based on varying information about errors. Our construction strategy varies the hint information provided, depending on the correctness of the response and its generation method. For correct responses, no hints are given, and the critique is collected only if every step is correctly labeled by the annotator. For flawed responses, the prompts are designed accordingly: RG1 prompts include a correct reference; RG2 prompts are augmented with the likely starting point of the errors; and RG3 prompts further incorporate detailed information about the mistakes.

3.3 Data Filtering

To enhance data quality, we apply a filtering process through Monte Carlo sampling: each (query, response, critique) tuple is refined by the actor model for 10 trials, and the critique data is retained only if its accuracy exceeds a predefined threshold τ = 0.3. To further assess the quality, we conduct a human evaluation on 500 randomly sampled critiques from the dataset, repeated 5 times. The rate of low-quality data is approximately 1.2%, suggesting the high quality of our dataset.

4 Critique Models Improves Reasoning through Test-Time Supervision

Using MathCritique-76k, we train critique models to provide step-level supervision and useful feedback on reasoning paths, along with the actor models with reasoning and refinement ability, with details provided in Appendix D.

Then, we investigate the impact of trained critique models in supporting the reasoning model at test-time (see top of Figure 2). Detailed setups can be found in Appendix E.

Critique models are highly effective at identifying the correctness of reasoning, offering constructive feedback for erroneous responses, and improving the overall accuracy of the actor. First, we compare our critique models with SOTA models used as critics. Results in Table 2 show that our 8B critique model significantly outperforms GPT-3.5, while our 70B model achieves performance on par with the GPT-4 series. Specifically, our 8B model surpasses GPT-3.5-Turbo in reasoning path judgment accuracy, and demonstrates a helpfulness improvement of 17.70% on GSM8K and 1.93% on MATH. The 70Bmodel advances further: its discriminability exceeds GPT-4-Turbo and GPT-4o on GSM8K and closely matches them on MATH; its correction accuracy on both datasets approaches that of GPT-4 series models, ultimately guiding the actor to comparable levels of accuracy.

Critique models assist the actor in better handling challenging queries. Next, we investigate the distribution of performance gains brought by the critique model across different difficulty levels. The results are illustrated in Figure 3, where the queries are categorized into 5 difficulty levels based on the number of correct responses among 100 samples [Hendrycks et al., 2021b]. It is evident that on both training and testing sets, critique models provide minimal benefit for simpler queries-where the actor already performs well-but significantly improve performance on more challenging problems. This phenomenon is more pronounced on the training set, offering valuable insights for incorporating critique supervision during training (Section 5) [Tong et al., 2024].

Scaling up test-time compute consistently improves reasoning performance. Moreover, we investigate whether incorporating critique models can further elevate the performance ceiling as test-time compute scales. A widely used metric is majority voting (MV@K) [Wang et al., 2023], which measures whether the most frequent answer among K parallel samples is correct, reflecting the model's consistency in generating high-quality responses. As shown in Figure 4, without critique models, MV@K performance improves with increased compute but quickly plateaus, even at higher levels like MV@2K and MV@3K. In contrast, with critique models, performance surpasses the baseline significantly under the same computing budget. These findings indicate that critique models effectively improve the exploration efficiency and quality of the actor, extending the performance ceiling when allocated more inference-time compute.

5 Critique-in-the-loop Self-Improvement for Better Reasoning Models

In Section 4, we observe that critique models significantly aid in raising the reasoning performance when scaling up inference compute. To further leverage their strengths, we present a critique-in-the-loop self-improvement method, which scales up exploration compute on challenging queries and leads to the development of stronger reasoning models (see Figure 2).

5.1 Critique-in-the-loop Self-improvement

Vanilla self-improvement. Self-improvement is an iterative method that leverages the actor model's correct responses to progressively enhance its performance [Huang et al., 2023; Tian et al., 2024]; each of the T iterations consists of two steps: exploration and learning.

In the exploration step of iteration t, we sample N responses for each query xj ∈ Dreason from the previous model πt-1 θ , i.e., ˆ yj = πt-1 θ (xj). Each data point is then filtered using the reward function r(x, y), where only correct solutions are retained to form a new dataset Dt = {(x, y)}|Dt | j=1 .

In the learning step of iteration t, the new dataset Dt is used to fine-tune the actor reasoning model πθ. The training loss is as in Equation 2 and we also include the original reasoning set Dreason and refinement set Drefine [Gülçehre et al., 2023].

A key challenge in self-improvement is obtaining diverse, correct answers per query during the exploration step [Huang et al., 2023; Tian et al., 2024]. However, models often suffer from "tail narrowing": over-sample easy queries while undersample hard ones. This bias accumulates across iterations, resulting in a long-tail distribution that ultimately leads to performance plateau or even degradation [Ding et al., 2024].

Introduce critique models for high-coverage exploration.

Our method enhances vanilla self-improvement by integrating critique models πϕ during exploration. At iteration t, πϕ provides feedback on incorrect responses from actor πtθ , prompting πtθ to refine them. The resulting correct solutions are then added to the next training set. This process broadens solution coverage for harder queries, effectively mitigating the tail-narrowing problem [Ding et al., 2024].

Difficulty-aware compute allocation for exploration. Building on our findings in Section 4, we allocate more compute to harder problems. This enables additional response generation, critique, and refinement to obtain high-quality and diverse solutions. Concretely, we adopt a simple difficultybased strategy: for incorrect initial responses, we performL times of critique and refinement; for correct ones, we add them directly to the training set. This mitigates the long-tail issue of self-improvement, enhances sampling quality, and improves overall performance [Ding et al., 2024].

We summarize the critique-in-the-loop self-improvement in Algorithm 1 of Appendix B.

5.2 Experiments

Implementation Details

To mitigate overfitting, we follow [Zelikman et al., 2022] and always fine-tune the original model π0 θ instead of the previous model πt-1 θ . After the learning step, a new dataset of better quality samples can be created once again [Xi et al., 2024]. More details are discussed in Appendix F.

Empirical Results and Findings

Critique-in-the-loop self-improvement consistently improves reasoning performance. The evaluation results in Figure 5 reveal that: (1) Performance improves with increasing sampling number N during exploration, demonstrating the benefit of enhanced compute. (2) Our method consistently outperforms vanilla self-improvement, especially with larger N. (3) The vanilla method quickly suffers from the tail narrowing issue [Ding et al., 2024] and reaches a bottleneck, while our method maintains steady improvement.

Critique-in-the-loop self-improvement balances the solution distribution across difficulty levels, and enhances performance on challenging queries in the test set. Figure 6 show that our method samples a higher proportion of solutions for challenging queries during exploration. This significantly balances the data distribution, mitigates the tail-narrowing issue, and consequently results in stronger test-time performance, with our approach significantly outperforms the vanilla baseline on harder problems (see Figure 7).

Combining test-time supervision with training-time supervisions yields more performance gains. We evaluate the combined use of critique models at both training- and test-time. We include two baselines: (1) self-correction, where the actor model directly refines its own reasoning (an SFT-based version of SCoRe [Kumar et al., 2024]); and (2) LMSI [Huang et al., 2023] as the vanilla self-improvement baseline. The training data consist of original datasets (GSM8K and MATH) and correction data derived from our refinement data by removing critiques, and reformatted into (query, original response, new response) triplets.

Results in Table 3 reveal that: (1) Integrating critique models during test-time consistently enhances performance, especially without training-time critique supervision. (2) With training-time critique, the additional benefit of test-time critique supervision becomes marginal, suggesting successful "distillation" from the critic into the actor during training. (3) Using separate critique models outperforms self-correction, consistent with prior work that highlights models' limited selfrefinement capability without external feedback [Huang et al., 2024; Xu et al., 2024] and potential objective conflicts [Huang et al., 2024]. (4) Compared to vanilla self-improvement + response-only, the approach of supervised fine-tuning + testtime critique supervision achieves better performance with less training compute (though more test-time compute), particularly on the more challenging MATH dataset.

6 Discussion and Analysis

Ablation study. Table 5 presents an ablation on two design aspects: (1) difficulty-aware compute allocation and (2) critique v.s. refinement generation compute allocation. We find both are essential: performance drops significantly without difficulty-aware allocation, highlighting its role in countering tail narrowing [Ding et al., 2024]; and also drops when allocating more compute to refinement generation, showing that critique variety, not refinement depth, is critical for final gains.

Generalization of actor models and critique models. We evaluate the generalization of our method on out-of-domain reasoning tasks in university-level mathematics and physics, using three evaluation tasks: AQUA-RAT [Ling et al., 2017], MMLU (Math), and MMLU (Physics) [Hendrycks et al., 2021a]. Results of accuracy in Table 4 demonstrate strong outof-domain generalization. Our actor model consistently outperforms baselines, and integrating the critique model yields additional gains across all training-time methods. Notably, these gains are slightly larger than those observed on in-domain tasks (Table 3), highlighting the critique model's critical role in generalization. Meanwhile, the self-correction method degrades performance on two datasets, further illustrating models' limitations in self-evaluation and self-refinement.

Scaling properties of critique models. Furthermore, we explore the potential of our critique models when supervising actors with different scales. Results in Figure 3 in Appendix H show that the 3B critique model successfully supervises actor models with various scales, with performance further enhanced with oracle reward functions. This indicates that smaller critique models can effectively guide larger actors, and oracle reward models provide additional supervisory signal. More details are provided in Appendix H.

How do critique models improve correct frequency and majority voting? Following [Brown et al., 2024], we investigate the relationship between the correct frequency in multiple samples and the performance of majority voting, particularly under the influence of critique models. Figure 8 reveals a critical failure mode without critique models: a high-frequency incorrect answer (e.g., 45%) may outweigh a relatively frequent correct answer (e.g., 40%), leading majority voting to fail. In contrast, critique models address this through their discriminative ability and helpful feedback to suppress incorrect paths and boost the relative frequency of correct answers, thereby improving both the overall correct frequency and the performance of majority voting.

Should test-time compute be scaled sequentially or in parallel? We compare two implementations of test-time compute: parallel sampling of (response, critique, refinement) triplets and sequential generation. Figure 9 indicates that the parallel strategy achieves higher Pass@K, likely due to the greater sampling diversity of the actors. Furthermore, majority voting analysis in Figure 10 demonstrates a performance trade-off [Snell et al., 2024]: parallel implementations perform better in lower compute budget, while sequential implementations scale better with higher computation. This inspires us in the development of the critique-in-the-loop self-improvement method. More details are discussed in Appendix I.

Moreover, for detailed computational cost analysis of our method, please refer to Appendix G.

Number of samples Number of samples

Figure 10: Evaluation results of different answer selection techniques. "Parallel MV" refers to generating K triplets in parallel and selecting the most frequent answer; "Sequential MV" refers to selecting the most frequent answer from linearly generated responses and refinements; "Sequential Final" selects the K-th response.

7 RelatedWork

Developing models for critique, reflection and correction. Building models that can critique, reflect, and correct is a promising route to scalable supervision across diverse domains [Welleck et al., 2023; Xi et al., 2023; Paul et al., 2024]. Early efforts primarily rely on prompting techniques to elicit self-correction; however, these methods often underperform without oracle rewards, as models struggle to reliably indentify their own mistakes [Huang et al., 2024; Xu et al., 2024]. To address this, recent studies shift toward fine-tuning strategies to instill critique and correction capabilities. Yet, these methods are constrained by the need for substantial human annotation, while reinforcement learning depending on carefully engineered rewards [Kumar et al., 2024;

Gao et al., 2024; Ankner et al., 2024]. Moreover, suitable reasoning datasets for training critique models remain scarce. In this paper, we introduce AutoMathCritique, a scalable pipeline for synthesizing high-quality critique data. The trained critique models provide effective supervision that yields stable gains for the actor, both at training- and test-time.

Self-improvement for LLMs. Prior work improves LLM reasoning by training on expert-labeled step-by-step trajectories, but large-scale annotation is costly and difficult to scale [Song et al., 2023]. While self-improvement methods leveraging self-generated data offer a more scalable alternative, they often suffer from tail narrowing, leading to diminishing returns [Shumailov et al., 2023; Alemohammad et al., 2024; Ding et al., 2024]. Some works train correction models to allocate more compute toward sequential self-corrections during test-time [Ye et al., 2023; Paul et al., 2024]. In this paper, we fine-tune critique models to provide reliable step-level supervision and actionable feedback during both training and inference. This improves sampling efficiency and quality, ultimately strengthening the actor's reasoning performance.

8 Conclusion

In this work, we study scalable oversight through training critique models. We first propose AutoMathCritique, an automated and scalable framework for constructing critique data, and use it to build the MathCritique-76k dataset. We then train critique models capable of providing step-level supervision and effective feedback. By incorporating critique-based supervision at both test time and training time, we show that critique models can significantly improve actor model performance. Finally, we conduct extensive experiments and analyses, and hope our findings offer valuable insights to the research community.

Acknowledgements

The authors wish to thank all anonymous reviewers. This work was partially funded by National Natural Science Foundation of China (No.62576106, 62476061, 62376061).

Contribution Statement

Zhiheng Xi and Dingwen Yang share co-first authorship.