State Revisit and Re-explore: Bridging Sim-to-Real Gaps in Offline-and-Online Reinforcement Learning with An Imperfect Simulator

1 Introduction

Robot skill learning based on Reinforcement Learning (RL) [Mnih, 2013; Ibarz et al., 2021; Johannink et al., 2019; Kober et al., 2013] has shown promising prospects to solve complex tasks in real-world scenarios such as dexterous manipulation, locomotion, and drone flight control [Celemin et al., 2019; Xiang and Su, 2019; Andrychowicz et al., 2020; Rudin et al., 2022; Song et al., 2023; Lee et al., 2020b]. As directly training RL policy on real robots is extremely expensive and time-consuming, a simulator is usually indispensable for learning robot skills. In most cases, a high-fidelity simulator is unattainable since the real environment dynamics are too complex or even unobservable to model. An alternative way is to simplify the environment dynamics and build a less accurate simulator to train the RL policies. However, directly deploying policies trained in the simulator to the real world is usually unfeasible because of the inconsistent environment dynamics, known as the sim-to-real gap problem.

∗Corresponding authors.

The most straightforward approach is to train RL policy in a simulator and then fine-tune it with real data [Peng et al., 2020; Kalashnikov et al., 2018; Lee et al., 2020a; James et al., 2019], which heavily rely on the quality of the simulator and also need high-cost real robot training. To make the simulation dynamics closer to the real-world dynamics, domain randomization approaches optimize relevant parameters for several randomized simulated dynamics while training RL policies, which could achieve more adaptable policies in the real world [Chebotar et al., 2019; Muratore et al., 2019]. However, these methods usually need manually specified randomized parameters and nuanced randomization distributions, leading to unstable training or conservative policies.

Despite different methods, the cost will increase quickly when an approach involves online RL training on real robots. Therefore, offline RL approaches that train policies exclusively with pre-collected datasets seem to be more affordable. As the performance of offline RL policies strongly depends on the size and state-action space coverage of the offline dataset, it is natural to combine offline RL on the realworld dataset with online RL in simulators, that is, offlineand-online RL methods. Previous methods such as [Niu et al., 2022; Niu et al., 2023; Xue et al., 2024] borrow ideas from Conservative Q-Learning (CQL) [Kumar et al., 2020], which designs a dynamics-aware policy evaluation scheme to penalize the Q-function learning on simulated state-action pairs with large dynamic gaps, which adaptively adjust the weights of the simulation policy and the real policy. However, due to the unrestricted exploration in the imperfect simulator, these methods inevitably suffer from low sample efficiency and insufficient state-action space coverage during training.

To solve this problem, we propose a State Revisit and Reexplore (SR2 ) hybrid offline-and-online RL framework. The basis of our approach is to assume both real and simulation environments share the same state space, and there is only a difference between the transitional dynamics. It is natural that the accumulated errors grow with the exploration horizon increases. As shown in Figure 1, the key idea of our approach is to introduce a mechanism that ensures access to better quality and diversity of state-action data, which effectively improves the data coverage and sample efficiency as well as alleviates the sim-to-real gaps problem. Specifically, we design a hierarchical RL framework that consists of a meta-policy and a sub-policy. The meta-policy guides to trace back to the explored high-value state-action samples and generates more high-quality samples in the imperfect simulator. The subpolicy simultaneously learns the robot skills with offline realworld datasets and simulation rollouts. For the meta-policy, the sub-policy training procedure could be viewed as an interactive environment that provides feedback.

Through extensive simulation and real-world experiments, we demonstrate the superior performance of our approach against other state-of-the-art methods. In addition, we prove that the proposed algorithm guarantees a suboptimality with the polynomial sample complexity in the sim-to-real robot skill learning task.

2 RelatedWork

2.1 Sim-to-Real Policy Learning

The dynamic gaps between simulators and real environments, known as the sim-to-real gaps problem, have long been recognized as a main challenge in RL-based robot skill learning [Peng et al., 2018; Zhao et al., 2020]. To solve this problem, existing methods could be grouped into three lines. The methods such as [Peng et al., 2020; Kalashnikov et al., 2018; Lee et al., 2020b; James et al., 2019; Kaufmann et al., 2023] adopt the most straightforward solution, which trains the policy in a simulator and then fine-tunes it in real environments. In these methods, a high-fidelity simulator is usually indispensable to guarantee a reliable policy transfer. Another line of works, such as [Chebotar et al., 2019; Muratore et al., 2019; Tobin et al., 2017], are domain randomization methods, which train a policy variety of simulated environments with randomized properties. They close the sim-to-real gap by optimizing the simulation's dynamic parameters. There is also a line of works such as [Eysenbach et al., 2020; Lyu et al., 2024; Liu et al., 2022; Liu et al., 2024a; Niu et al., 2022; Xu et al., 2023; Xue et al., 2024] which design a dynamics adaption mechanism to penalize the high dynamics-gap samples in the online exploration.

2.2 Hybrid Sim-to-Real Learning

Considering that directly learning transferable policy in simulators is extremely difficult, and directly training policies on real robots is unaffordable. The recent hybrid offline and online RL methods [Niu et al., 2022; Niu et al., 2023; Song et al., 2022; Hou et al., 2024; Song et al., 2023] combine the advantages of offline RL [Kumar et al., 2020; Agarwal et al., 2020] and online RL, which provide a prospecting solution to learning policies from offline realworld data and an imperfect simulator. The key to these methods is to design a dynamics-aware policy evaluation scheme to penalize the Q-function learning on simulated state-action pairs with large dynamics gaps, which adaptively adjust the weights of the simulation policy and the real policy. However, due to the unrestricted exploration in the imperfect simulator, these methods inevitably suffer from low sample efficiency and insufficient state-action space coverage during training. Inspired by the efficient exploration methods such as [Uchendu et al., 2023; Ecoffet et al., 2021; Feng et al., 2020; Liu et al., 2024b], there are also a number of works [Wagenmaker et al., 2024; Qu et al., 2024] that aim to improve the state-action space coverage and the sample efficiency in the sim-to-real RL. Different from previous methods, our approach aims to enlarge the state-action space coverage in a meta-learning manner, while we also give an insight to analyze our approach in a theoretical way.

3 Background

3.1 Hybrid offline-and-online RL

Hybrid offline-and-online (H2O) RL consists of two MDPs {Ms, ˜ Ms} := (S,A, r, {PMs , P˜ Ms }, ρ, γ), where S is the state space, A is the action space, r is the reward function, PMs , P ˜ Ms are the transitional dynamics corresponding to the real and the simulated environments, ρ represent the initial state distribution, and γ ∈ [0, 1) is the discount factor. Given the offline dataset D, which is generated by a behavior policy πb in the real environment, the goal of H2O is to learn a transferable policy πs using RL that maximizes the expected discounted rewards.

The basic idea is built upon the Conservative Q-Learning (CQL). It pushes down the dynamics-gap weighted Q-values and pulls up Q-values on trustworthy real offline data. The learning objective is designed as:

where dϕ (s, a) is a particular state-action sampling distribution that is associated with high dynamics-gap samples, R(dϕ ) is a regularization term for dϕ to control the behavior of dϕ (s, a). ˜ E(Q, ˆ Bπ ˆQ) represents the modified Bellman error of the mixed data from dataset D and the simulation rollout samples in online replay buffer B, which are generated by the real MDP M and the simulated MDP ˜ M.

4 Limitation of H2O

In this section, we theoretically analyze the bottlenecks affecting the sample efficiency of previous H2O RL methods.

Definition 1 (Dynamics Residual). Let Ms and ˜ Ms represent the real and simulated MDPs. There is only a difference between the transitional dynamics PMs and P˜ Ms . The dynamics residual is defined as:

As H2O involves the offline and online MDPs, its occupancy distribution could be defined as:

where the dMs πb,h (s, a), d ˜ Ms πs,h (s, a) are the occupancy distributions correspond to the real offline data and simulated data.

Denote the occupancy distribution of the optimal policy π∗ by dMs π∗ ,h (s, a; ρ). Considering that in the real environment, there is an optimal trajectory {s0, s1, ..., sh} that π∗ can visit, the probability of visiting sh in ˜ M could be defined as:

Theorem 1. Within the inaccurate transitional dynamics, there exists an MDP instance such that one has to suffer from an exponential sample complexity in total horizon H in order to explore a state that guarantees the policy to be suboptimal.

5 Method

Aiming to solve the low sample efficiency and insufficient state-action space exploration problems in the hybrid offline-and-online MDPs, we propose a state revisit and re-explore RL algorithm. Specifically, we introduce a meta MDP into the H2O framework, denoted as Mm := (Sm,Am, rm, PMm , ρm, γm), where Sm is the state space, Am is the action space, rm is the reward function, PMm is the transitional dynamics, ρm is the initial state distribution, and γm ∈ [0, 1) is the discount factors. Therefore, the proposed framework consists of a meta-policy πm and a sub-policy πs, where the πm guides the generalization of high-quality samples in the imperfect simulator to boost the training of πs, and πs adaptively learns the robot skill using mixed offline and online generated data.

5.1 State Revisit and Re-explore

State revisit by meta-policy. Regarding the training procedure πs as an interactive environment, πm is expected to find some valuable state-action samples in D and then re-explore them in the imperfect simulator. Considering that directly finding the valuable states according to the training procedure of πs is intractable, we simply decompose this process into two steps. First, we sort the state-action samples in D according to their rewards. Then we randomly choose some samples according to their position in the sorted dataset. We construct the state s(i) m,t = (η(i) , z(i) t , z(i) t-1 , a(i) m,t-1 ), where η = Qs,t-Qs,t-1 Qs,t-1 is the change in Q-value of the sub-policy πs at iteration t which makes the meta-policy promotes the learning of sub-policy. zt and zt-1 are sampled from a uniform distribution over (0, 1) which represent the normalized position index at current and previous iterations. am,t-1 ∈ {0, 1} is the action of previous iteration. As all state-action samples are sorted, the position implicitly encodes the reward information. Taking a sm,t as input, πm decides whether to conduct the exploration. Therefore, the learning objective of meta-policy is defined as:

where ˆ Bπm is the Bellman operator and Bm is the replay buffer.

State re-explore by Sub-policy. As πm determines to reexplore a pre-visited state or explore from an initial state in the imperfect simulator, the replay buffer Bs,πm used for subpolicy learning contains two kinds of samples, including the samples in dataset D, and the samples generated in the simulated environments under the control of πm. Therefore, we adopt the objective in Eq.(1) to learn πs where the only difference is the modified Bellman error, defined as:

where PM P ˜M = PM(st+1|st,at) P ˜M(st+1|st,at) , PM(st+1|st, at) and P˜ M(st+1|st, at) are the transitional dynamics of the real and simulation environments.

5.2 Algorithm

We summarize the training procedure of SR2 in Algorithm 1. To start, we sort the state-action samples according to the rewards and get a sorted dataset Dsort . To train the meta policy, we first sample N trajectories. For each of them, based on a random scalar z(i) t which represents the normalized position in Dsort , we adopt a positional function that selects an

Algorithm 1: State Revisit and Re-explore(SR2 )

• 1 Data: an offline dataset D from real environment, an imperfect simulator with biased dynamics Ms
• 2 Initialize: Q function Qm,Qs, actor network πm,πs, sub-policy mixed replay buffer Bs = ∅, meta-policy replay buffer Bm = ∅, A reward-based reordered dataset Dsort = Sort(D, r)
• 3 for step t = 1, · · · , T do

initial state-action pair. By constructing the state sm,t, the meta-policy πm takes it as input and decides whether to conduct the re-exploration in the imperfect simulator. Guided by the meta-policy, the sub-policy collects state-action samples in a replay buffer and mixes the samples with offline real data. Then the sub-policy could be optimized in the hybrid offline-and-online learning. According to the feedback of the sub-policy training procedure, the meta-policy could be optimized by Eq.(5) and Eq.(6).

Specifically, in this work, the sub-policy is implemented using H2O, chosen for its advantageous scalability. It is vital to note that employing other algorithms as sub-policy is also feasible due to the portability of SR2 . The meta-policy consists of a random policy and the Deep Q-Network (DQN) algorithm. The random policy function is employed to select state-action pairs from the real environment dataset, whereas the DQN algorithm, given the current sub-policy's training information and the corresponding state-action pair information, decides whether to reset the simulator from the selected state to roll out a new trajectory.

5.3 Theoretical Comparison

In this section, we provide the theoretical analysis, showing that our approach could effectively enlarge the state-action space coverage while yielding a polynomial sample complexity. We focus on comparing SR2 with finite-horizon pessimistic offline MDPs e.g. CQL, and the hybrid method H2O.

As our approach also involves offline and online MDPs, the occupancy distribution also consists of two parts, like Eq.(3). In our approach, we design a state revisit and reexplore mechanism that automatically finds potential highvalue states in the offline dataset by the meta policy πm and uses them to re-initialize the imperfect simulator and start from them for further exploration. Different from H2O, the probability of visiting sh in ˜ M could be defined as:

Definition 2 (Policy concentrability of finite-horizon MDPs). The policy concerntrability coefficient of π is defined as:

where d∗h (s, a) and dπh (s, a) are the state visitation distributions of optimal policy π∗ and the learned policy π at time step h, respectively.

According to Theorem 1, for any states sh that π∗ could visit, the unrestricted exploration in the imperfect simulator suffers from an exponential sample complexity. However, in Eq.(8), guided by the meta policy πm, there is a probability that πs finds an explored state sk and further re-explore it in the imperfect simulator for h-k steps. According to Definition 2, it is easy to conclude that C ∗CQL > C ∗H2O > C ∗SR2. Then we can get Theorem 2.

Theorem 2. With an appropriate choice of sample strategy and the meta-policy, Algorithm 1 guarantees a suboptimality bound up to a polynomial sample complexity that is lower than CQL and H2O.

6 Experiments

In this section, we conduct experiments to validate the effectiveness of the proposed algorithm (SR2 ) and compare it with other state-of-the-art methods in the field of cross-domain online and offline RL. We begin with a detailed description of the experimental setups, including the environment setting and the implementation of SR2 and baselines, followed by presenting the results of the benchmark experiments performed within the MuJoCo simulation environment and real robot experiments.

6.1 Experimental Setups

Simulation-based Experiments

To empirically validate the capacity of policy adaptation, we create two types of sim-to-real gaps. In this experiment, the real environment is configured as the standard OpenAI Gym MuJoCo. We select the standard HalfCheetah and Walker2d. Due to the non-interactive properties of the real environment, we choose the widely used offline RL benchmark D4RL [Fu et al., 2020]. For the simulated environment, as shown in Figure 2, the dynamics gaps are introduced by modifying the corresponding parameters in the configuration file or adding noise in robot actions. Following [Niu et al., 2023], we modify the gravity coefficient (×2, Gravity), friction coefficient (×0.3, Friction), adding Gaussian noise to joint actuators (N(0, action range), Joint Noise), increasing thigh size (×2, Big Thigh), increasing thigh joint motion range(×2, Flexible Thigh), stretching torso length (×4, Long Torso).

Figure 2: Illustrations of modified dynamics in MuJoCo environments

Real-world Experiments

We utilize an 8kg wheel-legged robot in Figure 3 as the platform for our real-world experiments. The robot employs a VMC controller[Pratt et al., 2001] to fix the relative position between the waist joint and the wheeled feet, allowing the RL model to control only the hub motors of a pair of wheels. To facilitate the experiment, we manually collect a dataset of 1M steps from the real world. Subsequently, we deploy H2O, PAR and SR2 in Isaac Sim for training.

We define the task as standing still, where the real wheellegged robot maintains balance in place by relying on two wheels under RL control. The state space is defined as (θ, ˙ θ, ϕ, ˙ϕ, x, ˙x), where θ is the pitch angle, ˙ θ is the pitch angular velocity, ϕ is the yaw angle, ˙ϕ is the yaw angular velocity, x is the linear displacement, and ˙x is the linear velocity. In Isaac Sim, we can directly read the aforementioned states, while in the real world, we deploy a Visual-Inertial Odometry based on the onboard stereo camera and IMU data to obtain these states. The actions are defined as the torques τl and τr of the two wheels. We also define a set of hyperparameters (c1, . . . , c8) to adjust the weights of the penalties for each offset during the trajectory collection procedure. Accordingly, the reward is calculated as:

We aim to maximize the reward when the robot is stationary. To achieve this, we impose penalties on pitch angle offset and displacement offset, assigning relatively higher penalty weights to θ and x. Additionally, to ensure that the output torques do not become excessively large, we also increase the penalty weights on τl and τr.

Baselines

Our baselines include the following algorithms: the crossdomain online RL algorithm DARC [Eysenbach et al., 2020], the dynamics-aware hybrid offline-and-online RL algorithm H2O [Niu et al., 2022], its improved version H2O+ [Niu et al., 2023], and the recently proposed Policy Adaptation by Representation mismatch algorithm PAR [Lyu et al., 2024]. In our experiments, we use the same set of five seeds. All experiments involve a simulated environment that allows online interaction but has differences in dynamics or morphology, while the real environment is non-interactive but comes with pre-collected datasets, such as HalfCheetah-medium replay, HalfCheetah-medium, HalfCheetah-medium expert, and Walker2d-medium replay. During the execution of the algorithms, evaluations are conducted in a fully interactive real environment. We adopt the experimental setups of H2O and H2O+, involving 1M interactions with the simulator and 1M training steps, which aligns better with sim-to-real research logic, rather than the offline design in PAR[Lyu et al., 2024], where the source domain is non-interactive and the target domain is interactive. Given that the advantages of the metapolicy may not be fully manifested under fewer iterations and training steps, we also conduct a set of SR2 experiments with 3M training interactions.

6.2 Comparative Experiment

Mujoco Benchmark Experiments

Our comparative results shown in Table 1 demonstrate that SR2 achieves superior or highly competitive performance on 20 out of 23 tasks. Notably, the results of DARC and H2O+ are absent in the HalfCheetah Medium Expert experiment, as we were unable to find a Pytorch implementation of DARC, and H2O+ has not been open-sourced yet. The corresponding data in table 1 are sourced from H2O [Niu et al., 2022] and H2O+ [Niu et al., 2023]. The results of PAR are obtained using its open-source code with only the source and target domain adjusted to align with the experimental settings, but its performance was slightly below expectations, potentially due to default parameter settings. While SR2 can integrate PAR as a sub-policy, further development will proceed after confirming the suitability of PAR's parameter configurations.

It is important to note that the -m and -mr datasets are considered to have higher data diversity, covering a large number of samples. Consequently, learning from these datasets presents a lower difficulty. However, SR2 outperforms other algorithms on these simple tasks, primarily due to the role of the meta-policy in reducing sample complexity and improving data utilization efficiency. In the HalfCheetah-me tasks, most baselines fail to perform well, while SR2 maintains strong competitiveness in this scenario. Moreover, when interactions and training steps are increased to 3M, SR2 's performance shows a significant improvement. This could be attributed to the characteristics of the HalfCheetah-me dataset, which is considered intermediate or expert-level with lower data diversity, limited sample coverage, and higher sample complexity. Baseline models struggle to learn sufficient patterns or features from this data, leading to poor performance. In contrast, SR2 effectively reduces sample complexity, allowing it to gain a substantial advantage in the more complex HalfCheetah-me tasks.

Real Robot Transfer Experiments

The experimental results on the real robot are shown in Figure 4, where SR2 demonstrates significantly better control performance compared to H2O and PAR. From the pitch angle information in Figure 4(a), it can be observed that PAR loses balance and falls after approximately 5 seconds, while H2O maintains balance for about 15 seconds but gradually becomes unstable and ultimately fails to remain balanced. In contrast, SR2 exhibits excellent balance throughout the entire experiment. Figure 4(b) further reveals that PAR initially moves backward and subsequently attempts to move forward again, but eventually loses stability. H2O, on the other hand, gradually deviates from the starting point and fails to force the robot to the starting point and stabilize it despite briefly maintaining balance. SR2 successfully maintains high stability by making fine adjustments near the starting point. Figure 4(c) illustrates the changes in pitch angular velocity, showing that SR2 achieves more stable performance, while H2O and PAR exhibit significant fluctuations and instability. Considering that the sim-to-real task is more challenging than Mujoco benchmark experiments due to the difficulty in quantifying dynamic gaps and morphological gaps, the overall task complexity is significantly higher. In such a complex task, SR2 outperforms other baselines primarily due to the role of its meta-policy in improving sample efficiency.

6.3 Analytical Experiments

Ablation Study

In the ablation study, we aim to evaluate the specific contributions of each component in the SR2 algorithm. The primary components of SR2 include the meta-policy and sub-policy, with the sub-policy being selected as the H2O algorithm in this study. In the comparison experiments, the meta-policy is composed of a random policy and a DQN policy. The random policy is employed to select state-action pairs from the real dataset, while the DQN policy, based on the information from the sub-policy's training and the state-action pairs selected by the random policy, decides whether to reset the simulator from the chosen state to roll out a new trajectory. Through this experiment, we seek to investigate the roles and effects of the random policy and DQN policy within SR2 .

The experiments are divided into three groups for evaluation: the H2O group, the H2O with random policy group(SR2 -random), and the H2O with random policy and DQN policy group (SR2 -random dqn). The H2O group utilizes the open-source original version of the H2O algorithm. In the H2O with random policy group (SR2 -random), the subpolicy uses H2O, and the meta-policy consists of a random policy, which selects any state-action pair and resets the simulator from the selected state to roll out a new trajectory. In the H2O with random policy and DQN policy group(SR2 - random dqn), the sub-policy still uses H2O, while the metapolicy is composed of both the random policy and the DQN policy, aligning with the SR2 in the comparison experiment.

The ablation study results are shown in Table 2. We observe that using only a random policy as the meta-policy (SR2 -random) leads to a significant improvement in model performance. Furthermore, when combining a random policy with a DQN policy as the meta-policy(SR2 -random dqn), the performance of the model improves even further. This demonstrates that our meta-policy approach effectively enhances model performance by reducing sample complexity and improving data utilization.

Research on Data Validity

Based on our theoretical analysis, when the optimal strategy in the real world accesses the real data state-action pair (s, a), the probability of SR2 accessing this state-action pair is higher than that of H2O. In other words, SR2 accesses more valid data compared to H2O. From a broader perspective, we can use H2O's discriminator to validate the authenticity of the data. Therefore, we conduct an experiment. For the data sampled during training, we record the number of data triplets

(s, a, s′ ) in every training step that the H2O's discriminator classified as having a higher real probability than sim probability, and we calculate the proportion of these data triplets (s, a, s′ ) among all the sampled triplets during the training process. If the real probability is judged to be higher than the sim probability, the data will be referred to as valid data.

As shown in Figure 5, in the HalfCheetah-me Long Torso task, the valid data of SR2 , including data triplets (s, a, s′ ), is generally more abundant than that of H2O. This indicates that from the perspective of the discriminator the data explored by SR2 are closer to the real distribution.

7 Conclusion

In Reinforcement learning (RL) based robot skill acquisition, a high-fidelity simulator is usually indispensable but unattainable since the real environment dynamics are difficult to model, which leads to severe sim-to-real gaps. To deal with this problem, we propose a State Revisit and Re-explore (SR2 ) hybrid offline-and-online RL algorithm in this paper. The proposed algorithm employs a meta-policy and a subpolicy, where the meta-policy aims to find high-quality states in the offline trajectories for online exploration, and the subpolicy learns the robot skill using mixed offline and online data. Through extensive simulation and real-world experiments, we demonstrate the superior performance of our approach against other state-of-the-art methods. In addition, we prove that the proposed algorithm guarantees a suboptimality with the polynomial sample complexity in the sim-to-real robot skill learning task.

Acknowledgments

This work was supported in part by the National Key R&D Program of China under grant No.2024YFB4707600, NSFC under grant No.62125305, No.U23A20339, No.62088102, No.62203348, No.52435010, Shaanxi Natural Science Basic Research Program 2025SYS-SYSZD-083.