S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning

1 Introduction

Reinforcement Learning (RL) has made significant progress across a variety of fields, including gameplay [Mnih et al., 2013; Silver et al., 2016], robotics [Chen et al., 2022] and autonomous systems [Bellemare et al., 2020; Mu et al., 2024; Luan et al., 2025]. Yet, the success of RL often relies on the careful construction of reward functions, a process that can be both labor-intensive and costly. To solve this issue, Preference-based Reinforcement Learning (PbRL) emerges as a compelling alternative [Christiano et al., 2017; Lee et al., 2021b]. PbRL uses human-provided preferences among various agent behaviors to serve as the reward signal, thereby eliminating the need for hand-crafted reward functions.

Existing PbRL methods [Lee et al., 2021b; Park et al., 2022a; Shin et al., 2023; Kim et al., 2023] focus on enhancing feedback efficiency, by maximizing the expected return with minimal feedback queries. However, these methods rely on high-quality or even ideal expert feedback, overlooking an important issue in preference labeling: indistinguishability of segments. For example, asking humans to specify preferences between two similar trajectories can be challenging, as it is difficult for human observers to discern which is superior. Consequently, the resulting preference labels are often incorrect, further degrading the performance of the algorithm [Lee et al., 2021a]. Therefore, the segment indistinguishability issue limits the broader applicability of PbRL.

In the field of unsupervised reinforcement learning, information-theoretic methods have been shown to discover useful and diverse skills, without the need for rewards [Eysenbach et al., 2019; Hansen et al., 2019]. In contrast to the unlabeled agent behaviors in PbRL, these discovered skills possess a higher degree of distinguishability, allowing humans to easily express preferences between different skills. However, the skill-driven approach has not been fully explored in PbRL, and how to apply the discovered skills to preference learning remains unclear. This naturally leads to the following question:

How can we integrate the skill mechanism with PbRL to overcome the indistinguishability of segments?

In this work, we aim to provide an effective solution to the important and practical issue in PbRL: indistinguishability of segments, by incorporating the skill mechanism. First, we conduct skill-based unsupervised pretraining to learn useful and diverse skills. Next, we introduce a novel query selection mechanism in the learned skill space, which effectively balances the information gain with the distinguishability of the query. We name our method as Skill-Enhanced Preference Optimization Algorithm (S-EPOA). Our experiments demonstrate the necessity of the above two techniques, showing that S-EPOA significantly outperforms conventional PbRL methods in terms of both robustness and learning efficiency.

In summary, our contributions are threefold:

• First, we highlight the critical issue of segment indistinguishability, validate its practical significance through human experiments, and theoretically analyze the limitations of current mainstream query selection methods, such as disagreement.
• Second, we propose S-EPOA, a skill-driven reward learning framework, which selects queries in a highly

distinguishable skill space to address the segment indistinguishability issue.

• Lastly, we conduct extensive experiments to show that S-EPOA significantly outperforms conventional PbRL methods under non-ideal feedback conditions. The results indicate that by introducing the skill mechanism, we can effectively mitigate the segment indistinguishability issue, thereby broadening the application of PbRL.

2 Preliminaries

Reinforcement Learning. A Markov Decision Problem (MDP) could be characterized by the tuple (S,A, P, r, γ), where S is the state space, A is the action space, P : S ×A → ∆(S) is the transition function, r : S ×A → R is the reward function, and γ ∈ [0, 1) is the discount factor balancing instant and future rewards. A policy π interacts with the environment by sampling action a from distribution π(s, a) when observing state s. The goal of RL agent is to learn a policy π : S → ∆(A), which maximizes the expectation of a discounted cumulative reward: L(π) = Eµ0,π [∑∞t=0 γ t r(st, at)].

For any policy π, the corresponding state-action value function is Qπ (s, a) = E[∑∞k=0 γ k rt+k|St = s, At = a, π]. The state value function is V π (s) = E[∑∞k=0 γ k rt+k|St = s, π]. It follows from the Bellman equation that V π (s) = ∑a∈A π(a|s)Qπ (s, a).

Preference-based Reinforcement Learning. In PbRL, the reward function r is replaced by human-provided preferences over segment pairs, denoted as σ0, σ1. A segment σ is a continuous sequence in a fixed length H of states and actions, i.e. {sk, ak, . . . , sk+H-1, ak+H-1}. Preferences are expressed as one-hot labels y ∈ {(0, 1), (1, 0)}, indicating the preferred segment. All preference triples (σ0, σ1, y) are stored in the dataset D. The algorithm first estimates a reward function ˆ rψ : S ×A → R, parameterized by ψ, using provided preferences, This learned reward function ˆ rψ is then used to train the policy with standard RL algorithms. To construct ˆ rψ, we employ the Bradley-Terry model [Bradley and Terry, 1952;

Christiano et al., 2017] as follows:

where σ1 ≻ σ0 indicates the human prefer σ1 than σ0. ˆ rψ can be trained by minimizing the cross-entropy loss:

3 Indistinguishability of Segments

In an ideal PbRL scenario, human labelers can give the true preference between two distinct behaviors. In practice, however, humans often have to label similar trajectories, which can lead to mistakes. These errors could reduce the precision of the trained reward function, and consequently degrade the performance. We name this labeling issue as the Indistinguishability of Segments. The issue significantly limits the broad application of PbRL, particularly in safety control fields [Fulton and Platzer, 2018] where the precision of the reward function is strictly required.

Human experiments. To validate this issue, we conduct human experiments, where human labelers provide preferences between segments with various return differences. Then, we calculate the match ratio between human labels and ground truth. Specifically, the human labelers watch a video rendering each segment and select the one that better achieves the objective based on the instruction. For example, the instruction given to human teachers in the Cheetah_run task is to run as fast as possible. The results in Figure 2 shows that as the return differences decrease, the match rate between human-labeled and ground truth preferences diminishes, indicating an increase in labeling errors. This confirms the practical significance of the indistinguishability issue. Please refer to Appendix C for experimental details.

Furthermore, the segment indistinguishability issue can be particularly severe when the query selection method prioritizes "informative" queries. A representative example is the

Algorithm 1 S-EPOA Framework

Require: frequency of feedback K, number of queries M per feedback session, total feedback number Ntotal

• 1: Initialize π(a|s, z) ←UNSUPERVISED PRETRAIN
• 2: for each iteration do
• 3: if iteration % K == 0 and total feedback < Ntotal then
• 4: Update trajectory estimator Rθ(z) based on Eq. 6
• 5: for m in 1 · · · M do
• 6: (σ0, σ1) ∼ QUERY SELECTION
• 7: Query the teacher for preference label y
• 8: Store preference D ← D ∪ {(σ0, σ1, y)}
• 9: end for
• 10: Update the reward model ˆ rψ using D
• 11: Relabel replay buffer B using ˆ rψ
• 12: end if
• 13: if the current episode ends then
• 14: Update ztask based on Eq. 9
• 15: end if
• 16: Obtain action at ∼ π(a|st, ztask) and next state st+1
• 17: Store transitions B ←B ∪ {(st, at, st+1, ˆ rψ(st, at)}
• 18: Sample transitions (s, a, s′ , ˆ rψ) from B
• 19: Minimize Lcritic and Lactor with ˆ rψ
• 20: end for

disagreement mechanism in PEBBLE [Lee et al., 2021b], which selects queries based on the highest variance in predictions from an ensemble of reward models. To illustrate why this occurs, we conduct the following theoretical analysis. As shown in Proposition 1, the disagreement mechanism tends to select queries with similar return values, resulting in segments with similar and often indistinguishable behaviors.

Proposition 1. Let {ˆ ri } be an ensemble of i.i.d. reward estimators, and (σ1, σ2) be a segment pair with ground-truth cumulative discounted reward r1 ≥ r2. Suppose ˆ ri estimates the cumulative discounted reward of σj as ˆ rij ∼ N(rj, c) (c is a constant), and induces preference

Then the disagreement of induced preference across {ˆ ri }, i.e. Var[ ˆ P[σ1 ≻ σ2]], approximately and monotonically increases as the dissimilarity of segment pair ∆ = r1 - r2 decreases.

4 Skill-Driven PbRL

To address the issue of segment indistinguishability, we hope to choose segments with different behaviors. Skill discovery methods can discover diverse skills, which align with this requirement. In this section, we propose the Skill-Enhanced Preference Optimization Algorithm (S-EPOA), which leverages skill discovery techniques to enhance PbRL's reward learning by selecting distinguishable queries. S-EPOA can be integrated with any skill discovery method, by introducing the following two key components:

• Skill-based unsupervised pretraining, where the agent explores the environment and learns useful skills without supervision (see Section 4.1).
• Skill-based query selection, which can select more distinguishable queries based on the learned skill space (see Section 4.2).

We show the overall framework of S-EPOA in Figure 1 and Algorithm 1.

4.1 Skill-based Unsupervised Pretraining

Previous unsupervised pretraining methods [Lee et al., 2021b; Park et al., 2022a] aim to help the agent explore the state space by maximizing the state entropy H(s). However, these methods often fail to produce clearly distinguishable behaviors, as the learned policies primarily focus on exploration, rather than diversity. As a result, the queries may be difficult for human teachers to distinguish. Skill-based unsupervised pretraining addresses this issue by guiding the agent to discover diverse skills, resulting in more distinct behaviors, and enabling the selection of distinguishable queries in the early training process.

In skill discovery, the policy is in the form of π(a|s, z), where z ∈ Z denotes the skill and Z represents the skill space. To ensure that the policies generated by each skill have distinct behaviors, we aim to maximize the mutual information between the policy's behavior and its skill. For tractability, we optimize a variational lower bound, as shown in Eq. 4, where p is the underlying distribution, and qϕ is learned via maximum likelihood on data sampled from p.

Skill discovery methods use intrinsic rewards to encourage agents to explore different behaviors. A typical form of the intrinsic reward is shown in Eq. 5:

In this way, after pre-training in this stage, we obtained a policy in the form of π(a|s, z). By selecting different skills z, we can generate segments with diverse behaviors.

4.2 Skill-based Query Selection

In this subsection, we focus on selecting distinguishable queries to enhance reward learning. To achieve this, the two segments being compared should come from different skills with distinct behaviors and performances. We introduce a trajectory estimator Rθ(z) to estimate the expected return of

Algorithm 2 QUERY SELECTION

• 1: Randomly sample N queries (σ0, σ1), where σ0, σ1 are segments generated by skills z0, z1
• 2: Calculate the difference |Rθ(z0) -Rθ(z1)|
• 3: Calculate the uncertainty Var(Pψ[σ0 ≻ σ1])
• 4: Normalize |Rθ(z0) -Rθ(z1)| and Var(Pψ[σ0 ≻ σ1])
• 5: Calculate I(σ0, σ1) for each query
• 6: Select the query with maximum I(σ0, σ1)

have clearly distinguishable skill explanations. The specific method is illustrated in Algorithm 2.

4.3 Implementation Details

To convert the unsupervised pretraining policy π(a|s, z) in Section 4.1 to PbRL's policy π(a|s) in Section 4.2, we attempt to obtain the skill nearest to the current task, i.e., performs the best under the current estimated reward function. This skill is denoted as ztask. Intuitively, ztask can be derived by solving the optimization problem in Eq. 8:

Using the learned trajectory estimator Rθ(z), we sample Nz skills zi from Z uniformly, and approximate ztask with

trajectories generated by skill z, which helps us identify skills with significant performance differences. The training objective of Rθ(z) is:

where τ is the trajectory generated by z, and ˆ rψ is the learned reward model. In practice, we normalize the targets of Rθ(z) to the range of [0, 1] for training stability. Based on Rθ(z), we define the skill-based selection criteria I(σ0, σ1) for query (σ0, σ1) with underlying skills (z0, z1):

where Pψ is the probability that reward model prefers σ1 than σ0, as defined in Eq. 1. The first term assesses the difference between the skills, while the second term measures the reward model uncertainty, which is commonly used in prior works [Lee et al., 2021b; Liang et al., 2022]. For training stability, we normalize both terms to the [0, 1] range and add 1 to balance the two values.

Thus, we propose the skill-based query selection method: For each query (σ0, σ1), we calculate the skill-based selection criteria I(σ0, σ1), and select queries with the highest value. Based on Eq. 7, this approach not only considers the query's uncertainty to maximize the information gain, but also considers the differences between skills, ensuring the segments Based on the above discussion, we obtain the overall process of S-EPOA, as shown in Algorithm 1. Firstly, we initialize the policy with skill-based unsupervised pretraining. Then, for each feedback session, we update the trajectory estimator as in Eq. 6, and select queries based on the skill-based selection criteria in Eq. 7. The PbRL reward model ˆ rψ is trained using the selected queries based on Eq. 2. Finally, we train the critic and actor using the learned reward function ˆ rψ .

Besides the two key components in Section 4.1 and 4.2, we also adopt the semi-supervised data augmentation technique for reward learning [Park et al., 2022a]. To elaborate, we randomly sub-sample several shorter pairs of (ˆσ0, ˆσ1) from the queried segments (σ0, σ1, y), and use these (ˆσ0, ˆσ1, y) to optimize the cross-entropy loss in Eq. 2. Moreover, we sample a batch of unlabeled segments (σ0, σ1), generate the artificial labels ˆ y, if Pψ[σ0 ≻ σ1] or Pψ[σ1 ≻ σ0] reaches a predefined confidence threshold.

5 Experiments

We design our experiments to answer the following questions: Q1: How does S-EPOA compare to other state-ofthe-art methods under non-ideal teachers? Q2: Can S-EPOA select queries with higher distinguishability? Q3: Can SEPOA be integrated with various skill discovery methods? Q4: What is the contribution of each of the proposed techniques in S-EPOA?

5.1 Setup

Domains. We evaluate S-EPOA on several complex robotic manipulation and locomotion tasks from DMControl [Tassa et al., 2018] and Metaworld [Yu et al., 2020]. Specifically, We choose 4 complex tasks in DMControl: Cheetah_run, Walker_run, Quadruped_walk, Quadruped_run, and 3 complex tasks in Metaworld: Door_open, Button_press, Window_open. The details of experimental tasks are shown in Appendix B.1.

Baselines and Implementation. We compare SEPOA with several state-of-the-art methods, including PEBBLE [Lee et al., 2021b], SURF [Park et al., 2022a], RUNE [Liang et al., 2022], and RIME [Cheng et al., 2024], a robust PbRL method. We also train SAC with ground truth rewards as a performance upper bound. For PEBBLE, SURF, and RUNE, we employ the disagreement query selection, which performs the best among all the query selection methods. In our experiment, we use APS [Liu and Abbeel, 2021a] for unsupervised skill discovery. The impact of skill discovery methods is discussed in the ablation study. More implementation details are provided in Appendix B.2 and D.

Noisy scripted teacher imitating humans. Following prior works [Lee et al., 2021b; Liang et al., 2022], we use a scripted teacher for systematic evaluation, which provides preferences between segments based on the sum of groundtruth rewards. To better mimic human decision-making uncertainty, we introduce a noisy scripted teacher. When the performance difference between two policies is marginal, humans often struggle to distinguish between them, as Section 3 shows. To imitate this, we implement an error mechanism: if the ground truth returns of two trajectories are nearly identical, we randomly assign a preference label of 0 or 1. The core idea is to evaluate policy performance by comparing the overall returns of entire trajectories, which aligns with how humans assess policies based on their overall effectiveness. Specifically, for a query (σ0, σ1) with underlying trajectories (τ0, τ1) and ground truth reward function rgt, if

a random label is assigned. Here, Ravg is the average return of the most recent ten trajectories. We refer to ϵ ∈ (0, 1) as the error rate. For fairness, we constrain that the two segments in a query come from different trajectories. It is important to note that our noisy teacher differs from the "mistake" teacher in B-Pref [Lee et al., 2021a], which randomly flips correct labels. Our teacher only introduces errors in too-close queries.

5.2 Results on Benchmark Tasks

Locomotion tasks in DMControl. Figure 3 shows the learning curves of S-EPOA and baselines on the four DMControl tasks with error rates ϵ ∈ {0.1, 0.2, 0.3}. S-EPOA outperforms baselines in most environments and remains robust under non-ideal conditions, while other PbRL methods are unstable and even fail.

Robotic manipulation tasks in Metaworld. Figure 4 shows the learning curves for S-EPOA and baselines on the three Metaworld tasks with ϵ = 0.2. These results further demonstrate that S-EPOA improves robustness against nonideal feedback across diverse tasks.

Enhanced query distinguishability. To evaluate that S-EPOA selects more distinguishable queries, we compare the ratios of queries that can be distinguished by the noisy teacher, for both S-EPOA and disagreement query selection methods. As shown in Table 1, S-EPOA achieves higher distinguishability ratios across all environments, demonstrating its effectiveness in selecting easily differentiable queries. To further assess the distinguishability of queries, we conduct human experiments. We compare S-EPOA with three stateof-the-art PbRL methods in DMControl tasks, where labelers provide 20 preference labels for each run, across 3 random seeds. As shown in Figure 6, humans find the queries selected by S-EPOA easier to distinguish. Please refer to Appendix C for experimental details.

Query visualizations. We visualize the segment pairs selected by both S-EPOA and the disagreement mechanism. As shown in Figure 7, the pair chosen by the disagreement mechanism has similar behaviors, while the pair selected by S-EPOA clearly differs, with σ1 being preferred. These results confirm that S-EPOA can select more distinguishable queries, enhancing both the labeling accuracy of noisy teachers and the quality of human preference labeling.

5.3 Ablation Study

Component analysis. To evaluate the effect of each technique in S-EPOA individually, we incrementally apply skillbased unsupervised pretraining and skill-based query selection to our backbone algorithm. Figure 5(a) shows the learning curves on the Quadruped_run task with error rate ϵ = 0.3. First, skill-based unsupervised pretraining significantly boosts performance, for both skill-based (red vs. green) and disagreement-based query selection (orange vs. blue). This improvement is because the pretrained policy generates diverse behaviors, which leads to a better reward function. Additionally, skill-based query selection enhances results (red vs. orange), as it selects more distinguishable queries, enabling the agent to obtain more accurate labels. In summary, both components of S-EPOA are effective, and their combination is crucial to the method's success.

Enhanced learning efficiency under the ideal teacher.

Under the ideal scripted teacher (ϵ = 0), S-EPOA can also significantly enhance learning efficiency. As shown in Figure 5(b), S-EPOA converges faster and achieves better final performance. This advantage results from selecting skills with high distinguishability, highlighting the robustness and effectiveness of our approach.

Integrate S-EPOA with other skill discovery methods.

To show that S-EPOA is integratable with various skill learning methods, we replace APS with DIAYN [Eysenbach et al., 2019] and CIC [Laskin et al., 2022], which are typical and commonly used skill discovery methods. The details of these methods are shown in Appendix D. As is shown in Figure 5(c), the performance is consistent and similar across all skill discovery methods, which demonstrates that S-EPOA is compatible with various skill discovery methods.

Enhanced query efficiency. We compare the performance of S-EPOA and SURF using different numbers of queries. As is shown in Table 2, S-EPOA outperforms SURF consistently, even if only 500 queries are provided, which demonstrates the ability of S-EPOA to make better use of the limited queries.

Effect of data augmentation in S-EPOA. We conduct an ablation study to evaluate the impact of data augmentation in S-EPOA. Figure 5(d) shows that without data augmentation, both S-EPOA and the baselines (PEBBLE as the backbone) show similar performance deficiencies. This highlights the importance of data augmentation for achieving optimal performance, while it is not the sole factor in our method's success and does not undermine the innovation of our approach.

6 RelatedWork

Preference-based reinforcement learning. PbRL enables humans (or supervisors in other forms, like script teachers) to guide the RL agent toward desired behaviors by providing preferences on segment pairs, where the feedback efficiency is a primary concern [Lee et al., 2021b; Park et al., 2022a]. Prior works improve the feedback efficiency from various perspectives. Some works focus on the query selection scheme, trying to improve the information quality of queries [Ibarz et al., 2018; Biyik et al., 2020; Hejna III and Sadigh, 2023]. Some works integrate unsupervised pretraining to avoid the waste on initial nonsense queries [Lee et al., 2021b]. Some works augment queries from humans to better utilize limited human feedback [Park et al., 2022a; Liu et al., 2023]. These methods depend on reliable feedback. However, humans could make mistakes, especially when the segment pair for comparison is slightly different, which restricts and even harms the performance in practice [Lee et al., 2021a; Cheng et al., 2024].

Unsupervised pretraining for RL. Unsupervised pretraining has been well studied in RL [Xie et al., 2022], where unlabeled data (i.e., transitions without task-specific rewards)

are used to learn a policy or set of policies that effectively explore the state space through intrinsic rewards. The method to calculate the intrinsic reward varies in different unsupervised pretraining works, including uncertainty measures like prediction errors [Pathak et al., 2017; Pathak et al., 2019; Burda et al., 2019], state entropy [Hazan et al., 2019; Liu and Abbeel, 2021b], pseudo-counts [Bellemare et al., 2016; Ostrovski et al., 2017] and empowerment measures like mutual information [Eysenbach et al., 2019; Sharma et al., 2020; Liu and Abbeel, 2021a; Park et al., 2022b; Park et al., 2023]. The learned policy could serve as a strong initialization for downstream tasks, enhancing the sample efficiency in multitask and few-shot RL.

Unsupervised skill discovery methods. Unsupervised skill discovery methods are a subset of unsupervised pretraining methods, which use empowerment measures as the intrinsic reward, trying to discover a set of distinguishable primitives. Mutual information I(s, z) is a common choice for the empowerment measure, where s is a state, and z is a latent variable indicating the skill. Some studies consider the reverse form I(s, z) = H(z)-H(z|s) [Eysenbach et al., 2019; Park et al., 2022b], which train a parameterized skill discriminator q(z|s) together with the policy. On the other hand, the forward form I(s, z) = H(s)-H(s|z) [Sharma et al., 2020; Liu and Abbeel, 2021a; Laskin et al., 2022] can be integrated with model-based RL and state entropy-based unsupervised pretraining algorithms. Additionally, some studies design the skill latent space for unique properties by parameterizing the distribution q(z|s) or q(s|z). VISR [Hansen et al., 2019] and APS [Liu and Abbeel, 2021a] let the latent z be the successor feature to enable fast task inference. LSD [Park et al., 2022b] and METRA [Park et al., 2023] bind the distance in state space and latent space to force a significant travel distance in a trajectory, thereby capturing dynamic skills.

7 Conclusion

This paper presents S-EPOA, a robust and efficient PbRL algorithm designed to address the segment indistinguishability issue. By leveraging skill mechanisms, S-EPOA learns diverse behaviors through unsupervised learning and generates distinguishable queries through skill-based query selection. Experiments show that S-EPOA outperforms PbRL baselines in robustness and efficiency, with ablation studies confirming the effectiveness of skill-based query selection. In future work, we aim to extend S-EPOA to broader applications.

Acknowledgments

This work is supported by the NSFC (No. 62125304, 62192751, and 62073182), the Beijing Natural Science Foundation (L233005), BNRist project (BNR2024TD03003), the 111 International Collaboration Project (B25027) and the Strategic Priority Research Program of the Chinese Academy of Sciences (No.XDA27040200).

Contribution Statement

• ∗N. Mu and Y. Luan contributed equally.
• †Y. Yang and Q-S. Jia are corresponding authors.