On the Reuse Bias in Off-Policy Reinforcement Learning

1 Introduction

Off-policy reinforcement learning algorithms [Mnih et al., 2015; Lillicrap et al., 2016; Haarnoja et al., 2018; Fujimoto et al., 2018] typically have high sample efficiency and are suitable for many real-world applications. The key idea of off-policy algorithms is to reuse historical trajectories in the replay buffer. Though promising, it also causes an issue that these trajectories are not sampled from the current policy. When estimating the expected cumulative return of the current policy with trajectories generated by historical policies, offpolicy evaluation, as the core of off-policy algorithms, yields the distribution shift in consequence [Hanna et al., 2019]. To alleviate this issue, the standard method in off-policy evaluation is importance sampling [Kahn and Marshall, 1953; Precup et al., 2000], which is a widely used Monte Carlo technique to evaluate the expected return of the target policy when the training data is generated by a different behavior policy. In practice, it usually requires computing the degree of deviation between the target policy and the behavior policy.

∗Corresponding author.

Previous off-policy methods, which are usually combined with importance sampling, have suffered from the problem of unstable training [Li et al., 2015]. To address this problem, previous work mainly focuses on the high variance of importance sampling [Li et al., 2015; Liu et al., 2018; Xie et al., 2019; Fujimoto et al., 2021]. However, we find that reusing trajectories in the replay buffer to optimize and evaluate may introduce a bias to off-policy evaluation (See Fig. 1(1)), which may also lead to unstable training and can not be ignored.

In this work, we present a new analysis beyond variance by introducing the concept of Reuse Bias, which formally characterizes the bias in off-policy evaluation caused by the reuse of the replay buffer for evaluation and optimization (See Definition 1). Under some mild assumptions (See Theorem 1, 2), we theoretically show that off-policy evaluation is an overestimation of the true expected cumulative return. To explain how severe the Reuse Bias is, we first prove that the expectation of off-policy evaluation for the optimal policy over the replay buffer may be arbitrarily large when its true return is arbitrarily small (See Theorem 3). Moreover, we experiment on MiniGrid to show that Reuse Bias will cause an erroneous policy evaluation and even gets more significant as the environments become more complex (See Fig. 1(2), and more details are in Sec. 6.2). Consequently, the Reuse Bias can be significantly severe and the inaccurate off-policy evaluation may mislead the direction of the gradient. This becomes particularly problematic for further policy improvement because sub-optimal actions might be reinforced in the next policy update, and further induces unstable training.

To reveal how to optimize the policy for controlling the Reuse Bias, we present a high-probability upper bound of Reuse Bias for any possible policies by extending PAC Bayes techniques [McAllester, 1998; McAllester, 2003] (See Theorem 4). Compared with classical bounds in statistical learning theory, our result holds for any policy hypothesis and is related to the optimized policy for guiding further optimization to con-

1. Reuse Bias due to resample τ3 and our BIRIS

1

trol the Reuse Bias. Moreover, we present that controlling one term, which only depends on the replay buffer and the policy, of our upper bound can effectively control the Reuse Bias by introducing the concept of stability for off-policy algorithms (See Definition 2, Theorem 5 and 6). Based on these analyses, we propose a novel Bias-Regularized Importance Sampling (BIRIS) framework which can improve the sample efficiency of off-policy algorithms by controlling Reuse Bias. We then experiment on MiniGrid to show that our BIRIS can conspicuously reduce the Reuse Bias (See Fig. 1(2), and more details are in Sec. 6.2). Since our method is orthogonal to existing off-policy algorithms and variance-reduction methods, like weighted importance sampling (WIS) [Mahmood et al., 2014], BIRIS can be easily combined with them. Extensive experimental results show that our BIRIS can significantly reduce the Reuse Bias in MiniGrid compared with PG+IS and PG+WIS. Since practical off-policy algorithms for handling complicated continuous control tasks always consider using actor-critic methods rather than directly using trajectories, we also extend our Reuse Bias to AC-based methods and show that these methods also suffer from the negative impact of the Reuse Bias. Moreover, we extend our BIRIS to this more practical setting and provide two practical algorithms (SAC+BIRIS and TD3+BIRIS). Extensive experiments show that our BIRIS-based methods significantly improve the performance and sample efficiency compared with baselines on a series of continuous control tasks in MuJoCo. In summary, our contributions are:

• To the best of our knowledge, this is the first attempt to discuss the bias of off-policy evaluation due to reusing the replay buffer. We show that the off-policy evaluation via importance sampling is an overestimation when optimized by the same replay buffer, which is recognized as the Reuse Bias in this paper.
• Moreover, we derive a high-probability bound of the Reuse Bias, which indicates that it cannot be eliminated only by increasing the number of trajectory samples.

Then we introduce the concept of stability and provide an upper bound for the Reuse Bias via stability, which indicates that controlling one term of the high-probability bound can effectively control the Reuse Bias.

• Based on these analyses, we propose BIRIS and show that our BIRIS can conspicuously reduce the Reuse Bias through experiments in MiniGrid. Moreover, we experimentally show that our method can significantly improve the performance and sample efficiency for different offpolicy methods in different MuJoCo tasks.

2 RelatedWork

Off-policy Evaluation. Designing safe, stable, and effective methods are significant for reinforcement learning (RL) [Garcıa and Fern´andez, 2015; Ying et al., 2022], especially for off-policy methods, which hopes to reuse historical trajectories for improving the sample efficiency [Precup et al., 2000; Precup et al., 2001]. It is well known that, when the current policy is independent of the samples, importance sampling is unbiased but has a high variance that is exponentially related to trajectory length [Li et al., 2015], which may cause the result unstable. Consequently, a lot of research focuses on reducing the variance in off-policy evaluation, like weighted importance sampling [Mahmood et al., 2014], doubly robust estimator methods [Jiang and Li, 2016; Thomas and Brunskill, 2016; Su et al., 2020], marginalized importance sampling methods [Liu et al., 2018; Xie et al., 2019; Fujimoto et al., 2021], and so on [Metelli et al., 2018; Hanna et al., 2019; Kumar et al., 2020a; Liu et al., 2021; Zhang et al., 2021; Metelli et al., 2021]. Some researchers also pay attention to providing a high confidence estimation for off-policy evaluation. [Thomas et al., 2015] first applied concentration inequality to provide a lower bound of off-policy evaluation. Follow-up work considered how to provide confidence intervals [Hanna et al., 2017; Shi et al., 2021] as well as a high-probability lower bound for self-normalized importance weighting [Kuzborskij et al., 2021]. Some recent work also provided a high-confidence estimation of the variance [Chandak et al., 2021b] and other parameters of the return distribution [Metelli et al., 2018; Metelli et al., 2021; Chandak et al., 2021a]. However, little work has been done on systematically examining the bias of off-policy evaluation.

Bias in Off-policy and Offline RL. Since bias is significant to analyze and develop algorithms in RL, there are some widely discussed biases in off-policy and offline settings like Q-function overestimation in DQN [Van Hasselt et al., 2016], extrapolation error in offline RL [Fujimoto et al., 2019], and sampling error in TD learning [Pavse et al., 2020]. These bias is mainly caused by out-of-distribution data [Fujimoto et al., 2019] and about Q function estimation in actor-critic methods [Van Hasselt et al., 2016; Fujimoto et al., 2019]. Besides them, in this work, we first discuss the bias resulting from reusing the trajectories in the replay buffer, which can regard as the bias in the policy optimization in actor-critic methods (more details are in Sec. 5.3).

3 Preliminary

We consider the Markov Decision Process (MDP) of M = (S, A, P, R, γ) in which S and A represent the state and action spaces, respectively. For any state-action pair (s, a) ∈ S ×A, P(·|s, a) is a distribution over S, representing its transition dynamic. Moreover, R(s, a) is the reward function and γ ∈ (0, 1) is the discount factor. The policy of any agent is modeled as a mapping π : S → D(A), i.e., π(·|s) is a distribution over A given state s ∈ S.

In RL, the agent interacts with the environment by making its decision at every timestep. At the beginning, we assume that the agent is in the state s0 ∼ µ(·), where µ(·) is the initial state distribution. At timestep t, the agent chooses its action at ∼ π(·|st), arrives at the next state st+1 ∼ P(·|st, at), and receive a corresponding reward rt = R(st, at). The performance of policy π is defined as the expectation of the discounted return of the trajectory:

where R(τ) represents the discounted return of the trajectory τ ≜ (s0, a0, r0, s1, a1, r1, ...) ∈ Γ.

As the trajectory distribution τ ∼ π is unknown, in practice, we estimate J(π) by Monte Carlo sampling. In order to improve sample efficiency, off-policy algorithms utilize trajectories sampled from other policies via importance sampling. We denote that there are m trajectories in the replay buffer B and the trajectory τi is sampled from policy ˆπi, i.e. B = {τi}mi=1 ∈ Γm, τi ∼ ˆπi.

For notational clarity, we denote ˆΠ = (ˆπ1, ˆπ2, ..., ˆπm) and B ∼ ˆΠ = τ1 ∼ ˆπ1, τ2 ∼ ˆπ2, ..., τm ∼ ˆπm. In off-policy evaluation, to estimate J(π) of a different policy π, we need to modify the weight of the return for each trajectory in B via importance sampling [Kahn and Marshall, 1953; Precup et al., 2000]. The resulting importance sampling estimate of the return for π is:ˆ

where pπ (τ) represents the probability of generating the trajectory τ with policy π, i.e.,

4 Reuse Bias

In this section, we first introduce a novel concept of Reuse Bias to measure the bias in current off-policy evaluation methods. Then, we show that the reuse of the replay buffer causes overestimation and provide a high probability upper bound of the Reuse Bias with some insights into this result.

4.1 Reuse Bias

Assuming that the hypothesis set of policies is H, for any offpolicy algorithm, we formalize it as a mapping O : H×Γm → H that takes an initialized policy π0 ∈ H and a replay buffer B ∈ Γm as inputs, and outputs a policy O(π0, B) ∈ H. In offpolicy evaluation, we use the importance sampling estimate ˆ JˆΠ,B (O(π0, B)) defined in Eq. (2) as an approximation of the expected return J(O(π0, B)). We define their difference as the Reuse Error of an off-policy algorithm O on π0 as well as B, and the expectation of the Reuse Error as the Reuse Bias:

Definition 1 (Reuse Bias). For any off-policy algorithm O, initialized policy π0 and replay buffer B ∼ ˆΠ, we define the Reuse Error of O on π0 and B as

Moreover, we define is expectation as the Reuse Bias:

It is well-known that importance sampling is an unbiased estimation when the estimated distribution is independent of the samples. In other words, the off-policy evaluation is unbiased, i.e., ϵRB(O, π0) = EB [ϵRE(O, π0, B)] = 0, when O(π0, B) is independent of B. However, off-policy algorithms [Haarnoja et al., 2018; Fujimoto et al., 2018] utilize trajectories in the replay buffer B to optimize the policy. In this case, the policy O(π0, B) does depend on B which makes the off-policy evaluation no longer unbiased. And we will provide a first systematic investigation of the Reuse Bias, as detailed below.

We first assume that the off-policy algorithmO∗ satisfies the condition that O∗ (π0, B) has the highest estimated performance with B and consider the Reuse Bias under this assumption. This assumption is mild and practical since off-policy algorithms always hope to maximize the expected performance of the policy w.r.t. the replay buffer in practice. Under this assumption, the estimation ˆ JˆΠ,B (O∗ (π0, B)) is an overestimation of J(O∗ (π0, B)) as summarized below:

Theorem 1 (Overestimation for Off-Policy Evaluation, Proof in Appendix A.1). Assume that O∗ (π0, B) is the optimal policy of H over the replay buffer B, i.e.,

We can show that ˆ JˆΠ,B (O∗ (π0, B)) is an overestimation of J(O∗ (π0, B)), i.e., ϵRB(O∗ , π0) = EB∼ˆΠ [ϵRE(O∗ , π0, B)] ≥ 0. If the equality holds, then for any B, B′ ∼ ˆΠ, we have

However, in practice, we always optimize our policy through several steps of gradient descent rather than directly getting the optimal policy O∗ (π0, B) over the replay buffer B. To analyze the Reuse Bias in more practical cases, we consider the one-step policy gradient optimization and prove that the estimation ˆ JˆΠ,B is strictly an overestimation of J in this case only under some mild assumptions as:

Theorem 2 (Overestimation for One-Step PG, Proof in Appendix A.2). Given a parameterized policy πθ which is independent with the replay buffer B and is differentiable to the parameter θ, we consider the one-step policy gradient

where α is the learning rate. If ∇θ ˆ JˆΠ,B (πθ), as the function of B, is not constant, and α > 0 is sufficiently small, then the Reuse Bias is strictly larger than 0, i.e.,

Furthermore, to explain how severe the Reuse Bias will be, we provide Theorem 3 to show that the estimated return of O∗ (π0, B) might be arbitrarily large when its true return is arbitrarily small, which indicates that the Reuse Bias will result in the degeneration of the policy optimization and the final performance.

Theorem 3 (Proof in Appendix A.3). For any fixed replay buffer size n, assume that Π is the set of random policies, and the reward of any state-action pair (s, a) is not negative, i.e., R(s, a) ≥ 0, then there exists an environment satisfying that EB∼ˆΠ [J(O∗ (π0, B))] is arbitrarily small when EB∼ ˆΠ [ ˆ JˆΠ,B (O∗ (π0, B))] is arbitrarily large. In other words, for ∀n, M ≥ 1, ϵ > 0, ∃ an environment satisfying that

here π∗ is the optimal policy of this environment.

In Theorem 3, we point out that the Reuse Bias may cause an erroneous off-policy policy evaluation, which might be sufficiently severe, and affect further policy improvement, which is also shown in experiments of MiniGrid (See Sec. 6.2).

4.2 High Probability Bound for Reuse Error

Since the Reuse Bias may mislead the optimization direction and further affect the performance, which cannot be ignored, it is important to analyze the connection of the replay buffer B and the Reuse Error to better control it. Without losing generality, we mainly consider the case that all trajectories are sampled from the same original policy ˆπ.

When the hypothesis set has bounded statistical complexity, for example, H is finite or its VC-dimension is finite, we can naturally provide a high-probability upper bound of the Reuse Error via using some classical results of statistical learning theory [Shalev-Shwartz and Ben-David, 2014] (We provide the result when H is finite in Appendix A.4 as an example). However, these results are based on the statistic complexity of the hypothesis set and maxπ,τ [ pπ (τ ) p ˆπ (τ ) ]. In practice, the statistic complexity of the hypothesis set is extremely huge while the sample number m is relatively small, and maxπ,τ [ pπ (τ ) p ˆπ (τ ) ] is the term related on the environment as well as the hypothesis set, which can not be optimized in the training stage.

Here, we provide another high-probability upper bound for the Reuse Error as below. Compared with prior results, our bound holds for all H even with unbounded statistical complexity, and is related with the optimized policy O(π0, B), which guides us to optimize O(π0, B) by controlling Reuse Error.

Theorem 4 (High-Probability Bound for Reuse Error, Proof in Appendix A.5). Assume that, for any trajectory τ, we can bound its return as 0 ≤ R(τ) ≤ 1. Then, for any off-policy algorithm O and initialized policy π0 ∈ H, with a probability of at least 1 - δ over the choice of an i.i.d. training set B = {τi}mi=1 sampled by the same original policy ˆπ, the following inequality holds:

where ϵ1 and ϵ2 are defined as:

In Theorem 4, ϵ1, as the KL divergence of the trajectory distribution under O(π0, B) and ˆπ, indicates their similarity over the trajectory space Γ. Moreover, ϵ2 only focuses on the similarity in the replay buffer B of these two policies. Then we will introduce some insights into this theorem.

First, when the replay buffer size m tends to infinity, our high-probability bound will converge to √ϵ1+ϵ2 rather than 0. This is reasonable because the policy O(π0, B) depends on B and the hypothesis space in our results may be arbitrary large since the statistical complexity of practical policies might be very huge or even difficult to analyze. Also, for better explaining that the Reuse Bias may still exist even when the sample number m tends to infinity, we conduct a concrete example satisfying that |ϵRE(O, ˆπ, B)| = 1 holds for any finite m and any replay buffer B in Appendix C.

Moreover, the high-probability upper bound depends on the KL divergence of these two policies as well as on their probability over B. Consequently, controlling their divergence, especially over B, guides us on how to optimize the policy for controlling the Reuse Bias.

5 Methodology

In this section, introducing the concept of stability for offpolicy algorithms, we further analyze the connection between controlling ϵ2 in Theorem 4 and controlling the Reuse Bias. Moreover, we propose a general framework for controlling Reuse Bias with practical algorithms by extending actor-critic methods for complicated continuous control tasks.

5.1 Theoretical Analysis on Controlling Reuse Bias

Inaccurate off-policy evaluation caused by Reuse Bias may lead the policy optimization to the wrong direction and further affect both the performance and sample utilization. Therefore, it is necessary to develop methods to control the Reuse Bias.

Based on Theorem 4, we can control ϵ1 and ϵ2 to control the Reuse Bias. However, directly using the trajectories in the replay buffer to estimate ϵ1, which is the KL divergence of pO(π0,B) and pˆπ , may also be biased because O(π0, B) is dependent on the replay buffer. Thus we only consider constraining ϵ2 and use L(π, B) to denote it, i.e.,

Moreover, we hope to analyze the Reuse Bias of policies with bounded L(π, B). Borrowing ideas of stability [Hardt et al., 2016], we similarly define the concept of stability for offpolicy algorithms and further provide an upper bound of stable off-policy algorithms as shown below

Definition 2 (Stability for Off-Policy Algorithm). A randomized off-policy algorithm O is β-uniformly stable if for all Replay Buffer B, B′ , such that B, B′ differ in at most one trajectory, we have

Theorem 5 (Bound for the Reuse Error of Stable Algorithm, Proof in Appendix A.6). Suppose a randomized off-policy algorithm O is β-uniformly stable, then we can prove that

Theorem 5 controls Reuse Bias for stable off-policy algorithms Furthermore, we provide Theorem 6 to point out that, under some mild assumptions, Reuse Bias of off-policy stochastic policy gradient can be controlled just by L(π, B).

Theorem 6 (Details and Proof are in Appendix A.7). We assume that the policy πθ is parameterized with θ, and |∇θ log pπθ (τ)| ≤ L1 holds for any θ, τ, and pπθ (τ) is L2-Lipsticz to θ for any τ. If we constrain the policy by L(π, B) ≤ M, then off-policy stochastic policy gradient algorithm (detailed in Appendix A.7) is β-uniformly stable where β is positively correlated with M, L1 and L2.

5.2 Bias-Regularized Importance Sampling Framework

Based on above analyses, for any given off-policy algorithm O, we focus on controlling L(O(π0, B), B). To simplify the notation, we denote π ≜ O(π0, B) as the policy trained by the off-policy algorithm. For any trajectory τ , we have

Thus, we can derive L(π, B) as

Consequently, we propose a novel off-policy reinforcement learning framework named Bias-Regularized Importance Sampling (BIRIS) for controlling the Reuse Bias as below

here LRL denotes the nominal loss function of the basic offpolicy algorithm, and α ≥ 0 is a hyperparameter to control the trade-off between standard RL loss and regularized loss.

5.3 Connection with Actor-Critic Methods

However, since directly using the return of trajectories to optimize the policy suffers from instability, many off-policy practical methods for complicated environments consider using TD-based actor-critic methods, like Soft Actor Critic (SAC) [Haarnoja et al., 2018] and Twin Delayed Deep Deterministic Policy Gradient (TD3) [Fujimoto et al., 2018]. These methods always alternately optimize the critic and the actor as below

• Optimizing the Q function via TD-learning, i.e., minimizing ∑(s,a)∈B (Qθ(s, a) - Qtarget(s, a))2
• Optimize the policy via the Q function, i.e., maximizing ∑s∈B Qθ(s, πϕ(s))

Current work mainly considers the error in Q function optimization caused by out-of-distribution data. For example, Extrapolation Error [Fujimoto et al., 2019] is because erroneously estimating target Q values of unseen state-action pairs.

However, similar to our analyses above, in the policy optimization stage, reusing the state-action pairs in the replay buffer will also cause the Reuse Error, and we can also implement our BIRIS to reduce the negative impact. Moreover, for complicated environments, trajectories are so long that their probabilities are difficult to calculate and numerically unstable.

If we assume ∣∣∣ π(a|s) ˆπ(a|s) ∣∣∣ ≤ ϵ for all (s, a) ∈ B and the length of the trajectory is T , we can show that

Consequently, for handling complicated environments, we use the state-level ratio LBR as a surrogate

Our LBR has some similar points with pessimistic offline reinforcement learning methods [Kumar et al., 2020b; Liu et al., 2020; Fujimoto and Gu, 2021] and is a supplement to the effectiveness of pessimistic methods based on our theoretical analyses on the view of the Reuse Bias. Furthermore, for handling complicated continuous control tasks, we concretize our framework to specific algorithms, like SAC and TD3, of which the core is to calculate π(a|s) ˆπ(a|s) for any state-action pair (s, a) ∈ B (More details are in Appendix D).

6 Experiments

We now present empirical results to answer the questions:

• How severe is Reuse Bias in the practical experiments and can our BIRIS effectively reduce Reuse Bias?
• What is the empirical performance of our BIRIS for actorcritic methods in complicated continuous control tasks?

6.1 Experiment Setup

We experiment in two different domains to answer the questions above. Before that, we first provide a short description of environments, including algorithms as well as metrics.

Gridworld. For the first question, we experiment in MiniGrid 1 , which includes different shapes of grids with discrete state space and action space, and is simple for optimizing and evaluating our policy. We will calculate and compare the Reuse Bias of PG+IS, PG+WIS [Mahmood et al., 2014], PG+IS+BIRIS, and PG+WIS+BIRIS.

Simulated Robotics. To evaluate BIRIS in complicated environments, we choose several continuous control tasks from MuJoCo and use two popular off-policy algorithms: SAC [Haarnoja et al., 2018] and TD3 [Fujimoto et al., 2018], as baselines, with uniform sampling and prioritization experience replay (PER) [Schaul et al., 2016]. Our implementation is based on Tianshou [Weng et al., 2022].

6.2 Results for Reuse Bias

We first present our experimental results in MiniGrid to answer the first question, i.e., how severe the Reuse Bias is for practical off-policy evaluation and whether our BIRIS can significantly control the Reuse Bias. For different grids (5×5, 5 × 5-random, 6 × 6, 6 × 6-random, 8 × 8, 16 × 16), we initialize our policy parameterized with a simple three-layer convolutional neural network. Moreover, we choose 30, 40, and 50 as the replay buffer size and sample trajectories to fill the replay buffer, to test its impact. Then we use PG [Sutton et al., 2000] and PG+BIRIS, respectively, to optimize the policy by maximizing the estimated return via replay buffer by IS and weighted importance sampling (WIS) [Mahmood et al., 2014]. In the test stage, we compute the return of our policy estimated by the replay buffer as well as the true expected return of our policy. To mitigate the impact of randomness caused by the replay buffer and sampling, we repeat our experiments for each task and each algorithm with 50 different random seeds.

1 https://github.com/mit-acl/gym-minigrid

We report part of our results (8×8 with the size of replay buffer 30) in Fig. 1(2) in the form of box diagrams (All figures are in Appendix B.1). In each subgraph, the distribution of the true return J of the optimized policy is shown on the left (filled in with blue) while the distribution of the estimated return ˆ J is on the right (filled in with green). In each box diagram, the horizontal lines at the top and bottom indicate the largest and smallest value respectively, while the horizontal lines at the top and bottom of the box indicate the first quartile and third quartile respectively. Moreover, the orange dotted line and the green plus sign represent the median and the expectation separately. In Table 1, we also report the Relative Reuse Bias, i.e., ϵRB(O, π0, B)/J(O(π0, B)) of all algorithms.

As shown in Fig. 1(2) and Table 1, the estimated return is obviously higher than the true return in all environments for PG+IS, which matches our theoretical analyses, i.e., using the same replay buffer to optimize and evaluate the policy will lead to overestimation. Furthermore, as shown in Table 1, the Relative Reuse Bias shown in this experiment is greater when the grid is larger and more complicated, which means that the Reuse Bias will be more serious in more complicated environments. It is probably because the policy is more likely to converge to local optimization in complicated environments.

As shown in Table 1, when the replay buffer size increases, the relative Reuse Bias of PG+IS will reduce to some degree but is still large even if the size is 50. Moreover, the relative Reuse Bias of PG+WIS is lower than it of PG+IS but is still relatively large, which shows that variance-reduced methods like WIS can somehow reduce the impact of the Reuse Bias. Finally, in almost all tasks and all replay buffer sizes, the Relative Reuse Bias of PG+IS+BIRIS and PG+WIS+BIRIS are conspicuously smaller than PG+IS and PG+WIS, which means that our BIRIS can significantly reduce the Reuse Bias.

6.3 Results for Simulated Robotics

We now report results to evaluate BIRIS-based methods in more complicated environments against actor-critic methods. We compare the performance of our methods against common off-policy algorithms like SAC and TD3 in MuJoCo environments, including Ant, Halfcheetah, Humanoid, Walker2d, and InvertedPendulum. To mitigate the effects of randomness caused by stochastic environments and policies, we train 10 policies with different random seeds for each algorithm in each task and plot the mean with standard deviations of these 10 policies as a function of timestep in the training stage. Moreover, we use the solid line and the part with a lighter color to represent the average reward and standard deviations of 10 strategies, respectively. We also report the optimal mean and standard deviations of the cumulative return of 10 policies trained by each algorithm in each environment in Table 2. As shown in this table, PER does not outperform uniform sampling, which is also reported in [Novati and Koumoutsakos, 2019] for DDPG [Lillicrap et al., 2016] in MuJoCo.

In Fig. 2 and Table 2, we compare SAC, SAC+PER, and SAC+BIRIS. As we can see, SAC+BIRIS learns a better policy compared with SAC, i.e., SAC+BIRIS can effectively improve the performance and the sample efficiency compared with SAC. Overall, the performance of SAC+BIRIS increases more smoothly in the training stage, especially in HalfCheetah and Humanoid. This is consistent with our theoretical results, in that BIRIS can better estimate off-policy performance and provide a more precise optimization direction.

The comparison of TD3, TD3+PER, and TD3+BIRIS is reported in Fig. 2 and Table 2. Similarly, for all tasks, TD3+BIRIS performs better than TD3, i.e., TD3+BIRIS can improve the performance as well as the sample efficiency compared with TD3, especially in Ant and HalfCheetah. We also notice that TD3+BIRIS can conspicuously reduce the standard deviations of the policies compared with TD3 in Ant and HalfCheetah. This is also because BIRIS can better reduce the Reuse Bias and provide a more stable optimization direction. Moreover, we also do some ablation studies in Appendix B.2.

7 Conclusion

In this paper, we first show that the bias in off-policy evaluation is problematic and introduce the concept of Reuse Bias to describe it. We theoretically prove the overestimation bias in off-policy evaluation, which is because of reusing historical trajectories in the same replay buffer. Moreover, we provide a high-probability upper bound of Reuse Bias as well as an expectation upper bound of Reuse Bias for stable off-policy algorithms. Based on these analyses, we provide a framework of Bias-Regularized Importance Sampling (BIRIS) with practical algorithms for controlling Reuse Bias. Experimental results demonstrate that BIRIS can significantly reduce Reuse Bias and mitigating Reuse Bias via BIRIS can significantly improve the sample efficiency for off-policy methods in MuJoCo. As our modifications are independent of existing variance reduction methods, they can be easily integrated.

Ethical Statement

Off-policy methods are important to improve sample efficiency and apply RL in larger scenarios. This paper studies the Reuse Bias of off-policy evaluation and is beneficial for further studying more efficient and stable off-policy methods. There are no serious ethical issues as this is basic research.

Acknowledgments

This work was supported by the National Key Research and Development Program of China (2020AAA0106302, 2020AAA0104304), NSFC Projects (Nos. 62061136001, 62106123, 62076147, U19B2034, U1811461, U19A2081, 61972224), BNRist (BNR2023RC01004), Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J.Z was also supported by the XPlorer Prize.