RM-Distiller: Exploiting Generative LLM for Reward Model Distillation

1 Introduction

Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains, and their effective deployment relies on alignment with human values through Reinforcement Learning from Human Feedback (RLHF) [Bai et al., 2022a]. This paradigm primarily depends on reward modeling, which provides a proxy for human preferences to supervise policy optimization. However, effective reward models (RMs) require to train on large-scale, high-quality human preference data. As task complexity scales, obtaining high-quality human annotations becomes prohibitively expensive and difficult [Touvron et al., 2023].

∗Corresponding author.

To mitigate reliance on human feedback, some works proposed to directly use powerful generative LLMs as RMs [Zheng et al., 2023; Huang et al., 2026]. However, despite their strong preference modeling capabilities, their practical use is limited by high API costs and inference latency. Consequently, other works have proposed to distill discriminative RMs from generative LLMs [Bai et al., 2022b; Malik et al., 2026]. These methods typically employ the LLM as an annotator to assign binary preference labels to response pairs. The resulting preference data with triplets of (instruction, responses, label) are then used to train RMs. Distilling RMs from powerful generative LLMs offers a cost-effective and scalable alternative to human annotation. As a result, it has been adopted widely in recent RL practice [Liu et al., 2024; Liu et al., 2025a].

Despite its popularity, this paradigm predominantly utilizes the teacher model's capability to assign binary preference labels, often treating it as a mere binary classifier [Huang et al., 2025], failing to fully leverage the rich capabilities embedded in LLMs. In practice, generative LLMs possess a wide range of capabilities and internal knowledge that are highly beneficial for preference modeling. The insufficient utilization of these capabilities limits the upper bound of RM performance and subsequent RL-based alignment.

We therefore investigate a crucial yet unexplored question: How can we more effectively distill generative LLMs

into reward models?

In this paper, we propose RM-Distiller, a framework that exploits generative LLMs to distill RMs, by incorporating their multifaceted capabilities across three dimensions:

1. Refinement capability: We leverage the refinement capability of generative LLMs to construct highly contrastive preference pairs for RM training. By performing targeted revisions on rejected responses, the teacher synthesizes highly correlated response pairs that differ only in the attributes determining preference. This compels the RM to discern subtle quality nuances instead of spurious correlations [Jiang et al., 2025].
2. Scoring capability: We utilize the scoring capability of generative LLMs to assess response quality and model preference strength. Instead of relying on discrete binary annotation, the teacher provides continuous preference scores that better reflect varying degrees of quality differences between response pairs. This allows the RM to be optimized with a margin-aware objective, capturing more nuanced preference strengths.
3. Generation capability: We incorporate the generation capability of generative LLMs to safeguard the foundational linguistic proficiency of the student model. By distilling the teacher's output distribution into the student backbone, we enforce a constraint that prevents the RM from undergoing catastrophic forgetting. This ensures the model retains robustness during regressive RM training [Yang et al., 2024b].

We evaluate RM-Distiller on both RM benchmarks and downstream RL alignment tasks. The results demonstrate that RM-Distiller significantly outperforms existing distillation methods with superior RM accuracy and RL alignment efficacy. Despite the prevailing practice of distilling generative LLMs into RMs, this is the first work that systematically explores the best practice for RM distillation.

Our contributions are as follows:

1. We conduct the first systematic investigation into RM distillation, by introducing a multifaceted distillation strategy RM-Distiller.
2. We propose to exploit three capabilities of the generative LLMs, refinement, scoring, and generation, to provide comprehensive supervision for training the RMs.
3. Extensive experiments demonstrate that RM-Distiller significantly improves RM performance both on RM benchmarks and downstream alignment scenarios.

2 Background

2.1 Preliminary: RewardModeling

RMs traditionally relied on human preference datasets, typically trained using the Bradley-Terry (BT) model to predict pairwise rankings. The standard objective of RM training is to learn a scalar function rϕ(x, y) that reflects human preference. Given a preference dataset D = {(x, yw, yl)}, where yw and yl denote the chosen and rejected responses for instruction x, respectively, the RM is optimized by minimizing the negative log-likelihood:

where σ is the sigmoid function. While effective for ranking, this loss function only considers the binary order of responses, ignoring the preference strength between them.

2.2 RMDistillation fromGenerative LLM

Due to the high cost and limited scalability of human oversight [Touvron et al., 2023], the community has increasingly shifted toward getting preference annotation from generative LLM [Liu et al., 2024], where teacher models are employed to evaluate paired responses and identify the preferred ones, thereby constructing preference annotations for training BT models [Liu et al., 2025a]. However, this paradigm often treats the teacher as a black-box classifier, failing to exploit its generative potential or the nuanced preference intensities necessary for reliable alignment.

To address this, recent studies have explored ways to enrich the distillation signal by incorporating more diverse information from teacher models. For instance, SynRM [Ye et al., 2025] integrates natural language critiques into RMs by appending teacher-generated justifications to the responses during both training and inference. Further extending this idea, CLoud [Ankner et al., 2024] explicitly distills the capability of generating critiques into discriminative RMs. Another line of research focuses on data augmentation, utilizing LLMs to synthesize a larger volume of preference data through response perturbation to cover a broader distribution of scenarios [Park et al., 2024; Shen et al., 2025]. While these methodologies enhance supervision or data diversity, they often only exploit a fraction of the teacher model's multifaceted capabilities, failing to fully harness its potential for RM distillation.

3 RM-Distiller

RM-Distiller begins by Initial Preference Pair Construction, which serves as the foundation for RM distillation. Subsequently, it fully exploits the teacher model's multifaceted capabilities through: (1) Refinement capability for Contrastive Refinement, (2) Scoring capability for Strength Guidance, (3) Generation capability for Generative Regularization.

3.1 Initial Preference Pair Construction

Initial Candidate Generation

Given an instruction x from an instruction set Dx, we first choose a set of candidate models M = {m1,m2, . . . ,mK}. After that, we collect a response pool Yx = {y1, y2, . . . , yK} such that each yj ∼ mj(x). The candidate models are chosen to differ in model families and sizes, ensuring responses to each instruction are with varying styles and quality levels.

Initial Preference Annotation

To construct the initial preference data, for each instruction x, we randomly sample two distinct responses (ya, yb) from Yx. We then employ the generative teacher LLM fT to determine the relative quality between (ya, yb). This results in an annotated preference pair (yw, yl), where yw and yl denote the chosen and rejected responses, respectively.

At this stage, we obtain preference triplets (x, yw, yl), which can be directly used to train RMs under the BT objective, as in standard RM distillation. However, this will result in the insufficient utilization of teacher models' capabilities. Therefore, we introduce the following methods to further leverage the teacher's supervision signal.

3.2 Contrastive Refinement

The efficacy of RMs depends heavily on the contrastive quality of the preference pairs. In standard distillation practices, pairs are often sampled independently, therefore exhibit significant discrepancies in length, formatting, or linguistic style [Jiang et al., 2025], which results in the RMs to rely on superficial shortcut instead of intrinsic quality attributes [Zhou et al., 2024]. To eliminate these spurious correlations and force the model to identify fine-grained preference boundaries, we utilize the refinement capability of the teacher model to synthesize highly correlated contrasts, as shown in Figure 3.

The refinement process begins by analyzing the gap between the rejected response yl and the teacher-generated response yT , where yT serves as a high-quality reference. We first prompt the teacher model fT to perform a contrastive diagnostic, where it articulates specific reasons for the rejection and outlines a path for improvement. Following this diagnostic, fT is instructed to minimally modify yl into a refined chosen version y∗w that matches the quality of yT while strictly

Contrastive Refinement Example

Instruction

How to get the last element of a list L in Python?

Rejected Response (yl)

You can use L[0] to access the last element of a list.

Refined Chosen Response (y∗w)

You can use L[-1] to access the last element of a list.

Teacher Response (yT )

In Python, negative indexing allows you to access elements from the end, with -1 representing the last item.

Figure 3: A concrete example of Contrastive Refinement.

preserving the original structure and phrasing. By maintaining a minimal edit distance, we ensure that y∗w and yl share a nearly identical semantic backbone, differing only in the attributes that define superiority.

3.3 Preference Modeling with Strength Guidance Margin-Aware Regression

In the traditional BT framework, the RM is trained as a binary classifier to maximize the probability P(yw ≻ yl). This objective implicitly assuming a uniform preference strength across all response pairs. However, in reality, the quality difference between responses varies significantly [Wang et al., 2024]. Therefore, we leverage the scoring capability of LLMs to explicitly capture and represent the strength of preferences to provide fine-grained supervision signals.

First, we prompt the teacher model fT to assign scalar quality scores sT (x, y) for chosen and rejected responses yw and yl, respectively. After that, we propose Margin-Aware Regression, to guide the student model to mimic the teacher's evaluative sensitivity by aligning the predicted reward difference with the teacher-generated score margin. Specifically, we define margin-aware loss as follows:

With this objective, the student model explicitly learns to align not only the relative order but also the exact magnitude of preferences, thereby being able to capture finegrained variations in preference strength, distinguishing between subtle improvements and absolute errors. While some recent studies have attempted to introduce a margin as a lower bound [Touvron et al., 2023], they still fail to capture the nuanced intensity of preference as in this work.

Self-Calibrated Scoring

As established in Section 3.2, we obtained the refined response y∗w through Contrastive Refinement as the chosen response. To assign more precise quality scores, we introduce Self-Calibrated Scoring. Specifically, we explicitly provide the teacher with the previously assigned score of the rejected response sT (x, yl) as a fixed anchor, and task it with scoring the refined response y∗w on the same scale. As a result, the score sT (x, y∗w ) reflects a calibrated increase proportional to the specific corrections applied during refinement. This relative evaluation protocol yields more stable and learnable preference margins for downstream reward modeling.

Moreover, during this process, we reuse the refinement dialogue history as the context for scoring, exposing the scorer to the teacher-produced contrastive diagnosis and its refinement outcome. By conditioning the scoring on the explicit refinement history, Self-Calibrated Scoring transforms absolute quality assessment into a relative adjustment.

Preference Pair Filtering

To ensure high data quality, we filter the synthesized preference pairs to retain sufficiently strong contrasts based on two criteria: (1) a substantial score margin and (2) a minimum edit distance between y∗w and yl:

where τs and τe denote the minimum thresholds for the score margin and the edit distance, respectively. Filtering for edit distance ensures that the refinement has introduced meaningful changes, while score margin ensures that the teacher recognizes a sufficient quality improvement. This ensures effective contrast between preference pairs for RM training.

3.4 Regularization with Generative Distribution Generative Regularization

Solely optimizing for discriminative signals can lead to reward over-optimization, where the model gradually loses its foundational understanding of language, which is essential for generalizability [Yang et al., 2024b]. Therefore, we propose Generative Regularization to preserve the generative knowledge of the backbone model during distillation.

Specifically, we retain the student's language model (LM) head and use the teacher fT as a linguistic anchor. For each teacher response yT , we supervise the student's LM head using a combination of negative log-likelihood (NLL) and Kullback-Leibler (KL) divergence distillation:

where yT = {y1T , y2T , . . . , yL T } denotes the teacher response of length L, and V denotes the vocabulary. Pϕ and PT represent the next-token probability distributions of the student and the teacher, respectively. α and β are coefficients that control the strength of the NLL loss and the KL divergence. Notably, if the teacher model is closed-source or if the teacher and student do not share a common vocabulary, we set β = 0.

Total Objective Function

Combining our proposed Margin-Aware Regression and the Generative Regularization loss, the total optimization objective of RM-Distiller is formulated as a weighted sum:

This multi-task objective synergizes precise preference modeling with linguistic anchoring. Specifically, Lmargin enables the model to capture precise quality differences, while Lreg prevents representation collapse by grounding the shared backbone in the teacher's generative distribution. This dual supervision ensures semantic robustness, leading to better generalization and stability during downstream RLHF.

4 Experiments

In this section, we evaluate RM-Distiller through a series of experiments to assess its reward modeling performance.

4.1 Experimental Setup

Data Settings

For training RMs, we utilize a 10,000-sample subset randomly drawn from the Skywork-Preference-v0.2 dataset [Liu et al., 2024]. Our method is evaluated across two scenarios:

• Unlabeled Scenario where no prior preference annotations are available, and we only utilize instructions from the dataset and employ candidate models to generate responses. This setup mirrors practical distillation settings with no access to high-quality human annotations, making LLM-based distillation a more practical alternative.
• Labeled Scenario where response pairs are associated with high-quality ground-truth preference annotations, and we retain the original response pairs from the dataset and keep only those samples for which the teacher's preference judgments are consistent with the original labels.

We employ RewardBench [Lambert et al., 2025] and RMBench [Liu et al., 2025b] as our primary evaluation benchmarks, which cover diverse subsets.

Models

We employ Qwen2.5-3B-Instruct [Yang et al., 2024a] as the backbone for the student RM. To facilitate distillation, we use GPT-4o [Hurst et al., 2024] and Qwen3-14B [Yang et al., 2025] in its non-thinking mode as the teacher models. For constructing the response pool in the unlabeled scenario, we leverage a set of candidate models including Llama-3.1-8BInstruct [Grattafiori et al., 2024], Mistral-7B-Instruct-v0.3 [Jiang et al., 2023], Gemma-2-9b-it [Team et al., 2024] and Vicuna-7b-v1.5 [Zheng et al., 2023].

Baselines

We compare RM-Distiller against several RM distillation methods including the vanilla BT Classifier, alongside its extensions Margin BT [Touvron et al., 2023] and LSAM [Wang et al., 2024]. We also evaluate against methods that leverage teacher-generated critiques, revisions or attributes, including SynRM [Ye et al., 2025], CLoud [Ankner et al., 2024], RMBoost [Shen et al., 2025], and SteerLM [Dong et al., 2023].

We also include a range of external RMs as reference points. These comprise discriminative models such as EurusRM-7B [Yuan et al., 2025], as well as off-the-shelf generative models including Claude-3.5-Sonnet and Llama-3.170B-Instruct, along with fine-tuned generative models including GRAM-LLaMA3.2-3B-RewardModel [Wang et al.,

2025] and JudgeLRM-7B [Chen et al., 2025].

4.2 Main Results

As shown in Table 1, RM-Distiller consistently and significantly outperforms traditional distillation methods across both unlabeled and labeled settings，whether using a closedsource or open-source model as a teacher. Notably, RMDistiller also surpasses several powerful external baselines, including both frontier generative models and open-source RMs built on significantly larger foundation models or trained with more extensive datasets.

These performance gains are driven by the effective synergy of three key capabilities leveraged from the teacher model. Specifically, the teacher's refinement and scoring abilities enable the RM to identify fine-grained differences in response pairs, thereby enhancing the perceptual capability of nuanced quality aspects. Furthermore, the teacher's generation capability helps the RM maintain a foundational understanding of language, allowing it to better comprehend what constitutes a good response.

4.3 Ablation Study

To further verify the effectiveness of RM-Distiller, we conduct an ablation study by incrementally adding each module, corresponds to each capability of the teacher. As shown in Table 3, each proposed module contributes significantly to the overall performance gains. Across both RewardBench and RM-Bench, we observe a steady increase in average accuracy as we incrementally integrate Margin-Aware Regression, Contrastive Refinement, and Generative Regularization. This confirms that leveraging the teacher's multifaceted capabilities is essential for high-fidelity RM distillation on both evaluation benchmarks.

4.4 Performance on RLHF

To evaluate the effectiveness of the distilled RMs in downstream alignment, we conduct RLHF experiments using PPO [Schulman et al., 2017], GRPO [Shao et al., 2024], and DAPO [Yu et al., 2025]. For all RL experiments, we adopt Llama-3-8B [Grattafiori et al., 2024] as the base policy model. We first perform SFT on the base model using the UltraChat dataset [Ding et al., 2023] as a cold-start. We then use prompts from the HH-RLHF [Bai et al., 2022a] and ShareGPT [Zheng et al., 2023] datasets as training instructions during RL. Alignment effectiveness are evaluated on both general and complex instruction following benchmarks, including AlpacaEval 2.0 [Dubois et al., 2024], FollowBench [Jiang et al., 2024] and CFBench [Zhang et al., 2025] 1 .

As shown in Table 2, RM-Distiller consistently outperforms the BT Classifier across all datasets and RL algorithms, indicating more effective reward signal for RLHF. Our distilled RM significantly enhances both general alignment and complex instruction following, demonstrating its ability to discern nuanced quality differences. Furthermore, the consistent gains under length-controlled evaluations confirm that these improvements reflect genuine enhancements in response quality rather than a mere verbosity bias.

1 Since these benchmarks rely on LLM-based evaluation, we adopt GPT-4o-0806 as the judge model.

4.5 Fast Adaptation Capability

Fast Adaptation refers to the ability to rapidly provide reliable reward signals for novel, domain-specific tasks, which is a vital determinant of RM's efficacy. RM-Distiller offers an efficient and practical solution, enabling the rapid distillation of specialized RMs with only a small amount of targeted data. To evaluate this capability, we choose the Arabic language domain, where we utilize an Arabic dataset 2 , which has significant difference with general preference modeling 3 . We utilize Qwen2.5-3B-Instruct as the student model and Qwen3-14B as the teacher model.

As shown in Table 4, RM-Distiller not only demonstrates a substantial margin over the BT model but also surpasses currently top-performing generalist RMs. This empirical evidence reveals that generalist RMs struggle to achieve true universality across diverse specific domains. In contrast, RMDistiller provides an efficient mechanism to adapt and distill reliable reward signals into specialized scenarios with only a small group of instruction data, enabling fast adaptation of reward modeling to a new domain.

4.6 Generalizability

A robust RM should generalize to challenging scenarios without exploiting spurious shortcuts. Poor generalizability often leads to reward hacking, where the model exploits unintended biases to achieve deceptively high scores without fully following the instructions [Zhou et al., 2026]. In this section, we employ EvalBiasBench [Park et al., 2024] and IFBench [Peng et al., 2025] as our evaluation benchmarks.

2 https://huggingface.co/datasets/FreedomIntelligence/

Arabic-preference-data-RLHF

3 We randomly sample 10,000 instructions as the start point for distillation. For evaluation, we extract a test set of 1,000 samples, which are filtered by GPT-5 to ensure a reliable ground truth.

As shown in Table 5 and 6, RM-Distiller demonstrates a superior capacity to generalize to out-of-distribution tasks. In EvalBiasBench, while the BT Classifier collapses by relying on superficial patterns, RM-Distiller exhibits significantly higher robustness. This indicates the student model learns to discern semantic boundaries rather than spurious correlations. In IFBench, RM-Distiller not only outperforms the BT Classifier baseline but even surpasses the direct prompting performance of the teacher. This demonstrates that RM-Distiller effectively captures fine-grained constraints within instructions, verifying its generalizability to unseen tasks.

4.7 Data Efficiency

Data efficiency is critical in RM distillation, as high-quality instruction data is often scarce when dealing with specialized domains or rapid model iteration. To evaluate this, we compare RM-Distiller against the BT baseline using a reduced training set of only 1,000 instructions.

As illustrated in Table 7, when the training set is reduced to only 1,000 instructions, the performance of the student model trained with BT model deteriorates sharply. In contrast, RMDistiller exhibits exceptional data efficiency, maintaining a high level of discriminative accuracy. By extracting significantly more supervisory information from each sample, RMDistiller effectively distills the teacher's underlying preferences. This high-density training signal enables the development of reliable RMs with minimal data, offering a practical solution for real-world alignment tasks.

5 Conclusion

In this work, we present the first systematic research about how to distill generative LLMs into discriminative RMs. By exploiting the multifaceted capabilities of generative LLMs, our proposed RM-Distiller provides the student model with rich, reliable supervision signals, enabling high RM performance without additional inference overhead.

Acknowledgments

This work was supported by the National Key R&D Program of China (Grant No. 2024YFC3809100), the National Natural Science Foundation of China (Grant No. 62276077, 62376075, 62376076), the Department of Science and Technology of Heilongjiang Province (Grant No. ZL2025F004), and the MOE Key Laboratory of Cognitive Intelligence and Content Security (Grant No. 10120251103, Harbin Institute of Technology).

Contribution Statement

Hongli Zhou, Hui Huang and Wei Liu contributed equally.