MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator

1 Introduction

Reinforcement learning (RL) has achieved state-of-the-art performance on numerous control decision-making tasks [Singh et al., 2022; Yu et al., 2021a]. However, the trial-and-error procedure in which the agent interacts with the environment directly can be costly and unsafe in some scenarios. Offline RL effectively overcomes this problem by learning policy from static dataset [Levine et al., 2020].

Offline RL suffers from the distribution shift between the learned policy and the behavior policy in the offline dataset due to limited coverage of state-action pairs. To tackle this issue, model-free offline RL algorithms either penalize value function for out-of-distribution (OOD) state-action pairs [Kumar et al., 2020; Kostrikov et al., 2021] or constrain the policy closed to the behavior policy directly [Kumar et al., 2019; Fujimoto et al., 2019]. However, inhibition of exploration for OOD region hinders the improvement of agent performance.

∗Corresponding Authors.

Model-based offline RL (MBORL) trains the environment model using offline dataset and optimizes the policy based on the constructed environment model. Compared with modelfree methods, MBORL can perform better to state not occur in offline dataset via generating more model data [Yu et al., 2020]. However, due to the gap between the estimated and true environment, the generated model data is unreliable, which degrades agent performance. Therefore, conservatism is incorporated into the algorithm through different means.

One type is applying reward penalties for OOD data by uncertainty quantification directly [Yu et al., 2020; Kim and Oh, 2023; Sun et al., 2023] or value function regularization indirectly [Yu et al., 2021b; Liu et al., 2023]. Another type employs a robust adversarial training framework to enforce conservatism by training an adversarial dynamics model [Rigter et al., 2022; Bhardwaj et al., 2023]. However, the dynamics model of the second type is constantly updated during the policy learning process, which demands more computation costs.

Due to the gap between simulation to reality and disturbances in the real world, a robust policy is a necessity in many real-world RL applications. However, the current MBORL algorithms rarely consider agent robustness when improving agent performance. To tackle the gap between the simulation (source) and actual (target) environment, the robust RL based on Robust Markov Decision Process (RMDP) [Nilim and El Ghaoui, 2005] optimizes policy under the worst possible environment [Pinto et al., 2017; Kamalaruban et al., 2020; Panaganti et al., 2022]. Notice that, this is consistent with the core issue of MBORL: the gap between the estimated and the actual environment. Therefore, is it possible to introduce robust RL into MBORL as a way to improve agent performance and guarantee robustness?

Motivated by the above question, a new MBORL algorithm is proposed, named MICRO, where a new conservative Bellman operator is designed to incorporate conservatism and guarantee robustness. Compared with [Rigter et al., 2022; Bhardwaj et al., 2023], MICRO only needs to choose the minimal Q value in the state uncertainty set, which significantly reduces computation cost. Fig. 1 illustrates the framework of MICRO conceptually. The main contributions are summarized as follows:

• MICRO, a novel and theoretically grounded method, is the first MBORL algorithm that trades off performance and robustness by introducing robust Bellman operator.
• A conservative Bellman operator, which combines standard and robust Bellman operators, is designed to impose conservatism and guarantee agent robustness.
• Compared with previous MBORL algorithms, MICRO can guarantee agent robustness with less computation cost while improving performance.

In this paper, the theoretical and experimental results show that MICRO has the guarantee of a robust policy improvement and outperforms current state-of-the-art algorithms on the D4RL dataset benchmark. Furthermore, MICRO achieves better robustness on different adversarial attacks.

2 RelatedWork

Model-free offline RL: Online RL algorithms perform poorly in offline RL due to value overestimation and distribution shift. Existing model-free algorithms typically adopt two methods: policy constraint and value regularization. The goal of policy constraint is to restrict the learned policy close to the behavior policy [Kumar et al., 2019; Fujimoto et al., 2019]. In contrast, value regularization methods underestimate the value for OOD data by conservatively estimating Q values [Kumar et al., 2020; Cheng et al., 2022] or assigning penalization based on the Q function uncertainty [An et al., 2021; Bai et al., 2021]. However, the root of challenge for offline RL stems from limited data coverage. Model-free offline RL algorithms learn policy only from offline dataset, which hinders agent exploration.

Model-based offline RL: Model-based offline RL algorithms train a dynamics model from the offline dataset and utilize the model to broaden data coverage. However, there is a gap between the learned and actual dynamics model due to the inadequate offline dataset. The conservatism should be incorporated into algorithms to prevent agent from areas where the prediction of the learned dynamic model is unreliable. One method is to penalize the reward by uncertainty quantification [Yu et al., 2020; Kim and Oh, 2023; Sun et al., 2023]. Another method is to enforce small Q-value for imprecise model data generated by the dynamics model [Yu et al., 2021b; Li et al., 2023; Liu et al., 2023]. Different from the above methods, RAMBO [Rigter et al., 2022] and ARMOR [Bhardwaj et al., 2023] incorporated conservatism by training the robust adversarial model. However, the above two methods demand high computation cost and perform poorly compared with current SOTA methods.

Robust RL: The research line of robust RL mainly includes two parts. One line of works focus on the adversarial learning. They applied adversarial attacks to states or actions during policy learning [Pinto et al., 2017; Pattanaik et al., 2018; Tessler et al., 2019]. Another line of works are based on RMDP, which optimize policy under the worst-case environment. The related studies are centered around how to solve minimization problem under uncertainty set [Wang and Zou, 2021; Badrinath and Kalathil, 2021; Panaganti et al., 2022]. Similar to [Kuang et al., 2022], the robust RL in MICRO ultimately only needs to consider the state uncertainty set, which reduces computation complexity.

This work: The works that are closest to our work MICRO are by RAMBO [Rigter et al., 2022] and ARMOR [Bhardwaj et al., 2023] that introduces the robust adversarial idea into algorithm. MICRO is significantly different from these in the following ways: 1) MICRO incorporates conservatism into policy by introducing robust Bellman operator, instead of training the adversarial model like RAMBO and ARMOR. 2) MICRO directly selects minimal Q value in state uncertainty set rather than constantly updating environment model during policy learning, which consumes less computation time. 3) MICRO learns the robust policy against environment perturbations and adversarial attacks, which RAMBO and ARMOR fail to consider. To the best of our knowledge, MICRO is the first work that incorporates the robust Bellman operator into model-based offline RL, and guarantees the performance and robustness of the learned policy with less computation.

3 Preliminaries

Markov Decision Process and Offline RL: An Markov Decision Process (MDP) is defined by the tuple, M = (S, A, r, TM, µ0, γ), where S is the state space, A is the action space, r : S × A → R is the reward function in M, TM : S × A → ∆(S) is the transition dynamics under M, ∆(S) denotes the set of probability distribution over S, µ0 is the initial state distribution, and γ ∈ (0, 1] is the discount factor. A policy π : S → ∆(A) is a conditional distribution over action given a state. The state-action value function QπM(s, a) (i.e., Q-function) represents the expected discounted return from taking action a at state s and then executing π thereafter in M: QπM(s, a) := Eπ,TM [∑∞t=0 γ t r(st, at) | s0 = s, a0 = a]. The value function is defined as V πM(s) := ∑a∈A π(a | s)QπM(s, a).

The goal of RL is to optimize the policy π that maximizes the expected discounted return J(M, π) := ∑s∈S µ0(s)V πM(s). Such policy π can be derived from Qlearning, which learns the Q-function that satisfies the following Bellman optimal operator:

where the operator TM is a contraction on R|S×A| and Q⋆M is the fixed point of the above operator. Therefore, the Q-iteration can be defined as Qk+1 = TMQk, and it satisfies that Qk → Q⋆M. The Q-iteration uses a function class Q, defined as Q = {Q : S × A → [0, 1/(1 - γ)]}, to approximate Q⋆M. The sample-based counterpart of Bellman operator TM can be expressed as ̂ TMQ(s, a) := r(s, a) + γ[maxa′∈AQ(s′ , a′ )]. The Q-function is updated by Qk+1 = arg minQ∈Q ∑Ni=1 [ ̂ TMQk(s, a)-Q(s, a)]2 . The dataset {(si, ai, ri, s′i )}Ni=1 can be a fixed offline dataset or replay buffer generated by the current policy interacting with the environment.

The goal of offline RL is to learn a policy which maximizes J(M, π) from the fixed dataset Doffline = {(si, ai, ri, s′i )}Ni=1 collected by a behavior policy µ. The discounted state visitation distribution obtained by rolling out policy π on M, denoted as dπM(s) := (1-γ)Eπ,TM [∑∞t=0 γ tP(st = s|π,M)], where P(st = s|π,M) is the probability of reaching state s at time t. The state-action visitation distribution is dπM(s, a) = dπM(s)π(a|s). We denote the ground truth MDP as M⋆ , so the state-action visitation distribution of a fixed dataset can be written as dµM⋆(s, a). However, since some (s, a) barely occurs in Doffline, implying dµM⋆(s, a) approaches zero, offline RL tends to suffer from extrapolation error.

Robust MDP and Robust RL: The Robust Markov Decision Process (RMDP) considers the discrepancy between simulation and real environments, and gives an uncertainty set Mε of possible environments. Following [Abdullah et al., 2019; Kuang et al., 2022], Wasserstein Distance of order p is used to describe uncertainty set Mε, which is defined as

where T̂ M(· | s, a) and TM⋆(· | s, a) are the transition dynamics under simulation and real environment, respectively. If the reward function is unknown, since the analysis is similar to the above, we can concatenate the reward onto the state. Wp is pth -order Wasserstein distance given by

where K(T̂ M, TM⋆) denotes the set of couplings between T ̂ M and TM⋆ , d(·, ·) : S × S → R+ is a distance metric for S [Hou et al., 2020].

The objective of robust RL is to find the optimal robust policy that maximizes the expected discounted return against the worst possible environment in uncertainty set Mε. The robust value function V π and the robust Q-function Qπ are defined as V π (s) = infM∈Mε V πM(s),Qπ (s, a) = infM∈Mε QπM(s, a), respectively [Nilim and El Ghaoui, 2005]. By incorporating uncertainty set into Bellman operator, the robust Bellman operator can be denoted as [Iyengar, 2005]

Since the above formulation involves a minimization over the uncertainty set Mε, RMDP is much more computationally intensive compared to standard MDP, which poses a great computational challenge.

Model-Based Offline RL: MBORL uses the offline dataset Doffline to estimate transition dynamics model T̂ M(s′ | s, a), which is typically trained via maximum likelihood estimation (MLE) : minT ̂M L̂ M := E(s,a,s′ )∼Doffline [-logT̂ M(s′ | s, a)]. In the following part, we assume that the reward function r is known, although the r can be learned together with dynamics model if unknown. Then, the estimated MDP ̂ M = (S,A, r, T̂ M, µ0, γ) can be constructed to approximates the true MDP M⋆ . The MBORL methods use the estimated MDP ̂ Mto generate model data Dmodel through hstep rollouts, which broadens the data coverage and mitigates the problem of extrapolation errors. The current MBORL algorithms use an augmented dataset Doffline ∪Dmodel to train the policy, where each data is sampled from the model data Dmodel with the probability f, f ∈ [0, 1], and from the offline data Doffline with probability 1 - f . However, due to the gap between the estimated and true MDP, some generated model data are inaccurate, leading to poor performance of agent. Therefore, how to strike a balance between accurate offline data and inaccurate model data becomes the key to improving the performance of MBORL algorithms.

4 ProblemFormulation

To address the MBORL problem proposed in Section 3, conservative policy optimization is performed within that model [Kim and Oh, 2023; Yu et al., 2021b; Yu et al., 2020]. Different from the above methods, [Rigter et al., 2022] and [Bhardwaj et al., 2023] introduced the robust adversarial idea into MBORL. They both incorporate the policy learning information into the dynamics model learning to train the robust dynamics model, which effectively improves agent performance.

However, there are the following drawbacks for the above two methods: 1) The dynamics model is constantly updated during the policy learning process, which needs much more computation cost. 2) They only consider the agent policy improvement and ignore the agent robustness. Therefore, we want to design an MBORL algorithm with good performance, strong robustness and low computation cost. Before this, we first make Assumption 1 that is widely used in the offline RL literature [Chen and Jiang, 2019; Xie et al., 2021; Wang et al., 2021]. This assumption states that there must exist a function Q ∈ Q which can well-approximate Q⋆M under the operator TM.

Assumption 1 (Approximate realizability). Let dπ ∈ ∆(S × A) be the arbitrary data distribution. Then, for any policy π ∈ Π, infQ∈Q E(s,a)∼dπM [(Q - TMQ)2 ] < εr holds.

Inspired by the goal of robust RL to train a robust policy that can be against the gap between the simulation and the actual environment, we introduce robust RL to MBORL as a way to cope with the gap between the trained and the actual environment in MBORL. [Rigter et al., 2022] gave the following proposition, proven by [Uehara and Sun, 2021], which shows that the performance of any policy in the true MDP is at least as good as in the worst-case MDP with high probability.

Proposition 1 (Pessimistic value function). Let M⋆ be the true MDP and Mε be the uncertainty set of MDP. Then with high probability, for any policy π, infM∈Mε V πM ≤ V πM⋆ holds.

Proposition 1 states that the value under the worst MDP is a lower bound on that in the real MDP, which guarantees the conservatism of value function in uncertainty set Mε. Therefore, we combine robust RL and MBORL directly and propose a new conservative Bellman operator T Q(s, a).

Definition 1. The conservative Bellman operator T is defined as

The above operator expresses: 1) When the data are accurate offline data collected from the true MDPM⋆ , the standard Bellman operator is applied for the computation. 2) When the data are inaccurate model data collected from the estimated MDP ̂ M, the robust Bellman operator is used to perform conservative computation. The conservative Bellman operator incorporates conservatism into the algorithm, which reduces training risk frommodel errors. Meanwhile, this operator can guarantee robustness by optimizing policy under worst-case MDP. Compared with [Rigter et al., 2022] and [Bhardwaj et al., 2023], this method only needs to consider the worst MDP and does not require updating the dynamics model constantly, which significantly reduces computation cost. Next, we theoretically analyze dynamic programming properties of T in the tabular MDP setting and give the following proposition.

Proposition 2 (γ-contraction mapping operator). Let ∥ · ∥∞ be the L∞ norm and (R|S×A| , ∥ · ∥∞) is complete space. Then, the conservative Bellman operator T is a γ-contraction mapping operator, i.e. ∥T Q1 - T Q2∥∞ ≤ γ∥Q1 - Q2∥∞.

The proof of Proposition 2 is deferred to Appendix A.1. According to the fixed point theorem, any initial Q function can converge to a unique fixed point by repeatedly applying the conservative Bellman operator T . Note that the ground truth MDP M⋆ is unknown in Eq. (2). Here, considering M⋆ ∈ Mε holds with high probability, the MDP ̂ MMLE trained by MLE given offline dataset Doffline is employed to approximate the true MDPM⋆ .

However, the computation of operator T also faces the following crucial challenge. It is infeasible to consider all MDPs in the uncertainty set Mε for minimization in Eq. (4). How to solve the optimization over uncertainty set Mε? For this challenge, [Panaganti et al., 2022] gave the dual reformulation of RMDP and solved the dual problem using empirical risk minimization. However, they introduced a new network that is updated constantly in the solving process, which also demands much computation cost. Therefore, the following section gives a solution for this challenge and derives how to further incorporate robust RL into MBORL.

5 Methodology

5.1 Reformulation for Conservative Bellman Operator

Following the previous works [Kuang et al., 2022; Blanchet and Murthy, 2019], the dual reformulation of optimization, infM∈Mε Es′∼TM(s′ |s,a)[maxa′∈AQ(s′ , a′ )], is given below.

Proposition 3. Let Mε be the uncertainty set defined by (2). Then, for any Q : S × A → [0, 1/(1 - γ)], the term infM∈Mε Es′∼TM(s′ |s,a)[maxa′∈AQ(s′ , a′ )] can be equivalently written as

The proof can be found in Appendix A.2. Note that, the expection only relies on the true MDP M⋆ . This avoids the demand to consider all MDP in Mε. However, the above equation involves minimum and maximum optimization, which is also difficult to solve. The above equation can be reduced to the following optimization problem: maxλ infx∈S{q(x) + λ(d(x) - ε)}, λ ≥ 0. The above optimization is essentially a Lagrange dual problem for the following optimization: minx q(x), d(x) ≤ ε. Therefore, the above equation is equivalent to the following form

Let Uε(s′ ) be the uncertainty set of s′ , denoted as Uε(s′ ) = {¯ s ∈ S : d(¯ s, s′ )p ≤ ε}. Then, the Eq. (5) can be written as Es′∼TM⋆ {inf ¯ s∈Uε(s′ )[maxa′∈A Q(¯ s, a′ )]}. Since the ̂ MMLE is used to approximate the trueM⋆ , the conservative Bellman operator T can be expressed as

The operator T performs conservative estimation for model data by choosing the worst-case state ¯ s from Uε(s′ ), which also improves the agent robustness. Although the above operator still faces the optimization dilemma, it is easier to analyze the state compared with MDP. Next, the best function mapping f(s, a) is chosen from the function class F = {f : S ×A→R} that satisfies the following form

where f(s, a) is the penalty term, β is a tuning coefficient. Since f (s, a) = 0 when M = M⋆ , we only need to consider the solution for f ∈ F under the estimated MDP ̂ MMLE. Theoretically, the empirical risk minimization approach is applied to find f , denoted as

where Dmodel is the dataset collected from ̂ MMLE. Due to uncertainty in the model parameters, the ensemble of N dynamics models, {T iφ = N(µiφ, σiφ)}Ni=1 , is trained similar to [Yu et al., 2020; Janner et al., 2019], each of which is a neural network that outputs a Gaussian distribution. Notice that there is a bias in the states predicted by each dynamics model, so the set X (s, a) = {s′ |s′ ∼ T iφ(s′ |s, a), i = 1, 2, ..., N} is applied to approximate Uε(s′ ). Further, the following equation is adopted to approximate f(s, a):

where s′ ∼ ∑Ni=1 T iφ(s′ |s, a)/N. Since X(s, a) still fails to cover all region of the state uncertain set, the coefficient β is introduced in Eq. (6) to adjust the value. From this, the f(s, a) also reflects the gap of Q-function value between the worst-case and average level. Obviously, Eq. (6) also satisfies γ-contraction mapping, which guarantees the iteration convergence.

Note that f(s, a) can be regarded as a type of uncertainty quantification like MOPO [Yu et al., 2020]. However, f(s, a) contains more robustness information. This term trade off performance and robustness. For example, if ε1 ≤ ε2, then Uε1 (s′ ) ⊆ Uε1 (s′ ) holds, thereby inf¯ s∈Uε1 (s′ )[F (¯ s)] ≥ inf ¯ s∈Uε2 (s′ )[F (¯ s)] holds. Therefore, as ε increases, the penalty f(s, a) for model data increases, which means robustness increases while performance is difficult to improve. The ε for different data (s, a) is adjusted dynamically by model uncertainty indirectly in Eq. (8), instead of the fixed value. This is beneficial to trading off the performance and robustness.

5.2 Theoretical Analysis

The theoretical guarantees on Robust Policy Improvement property of MICRO are given in this section. First, the policy concentrability coefficient is introduced below [Bhardwaj et al., 2023; Uehara and Sun, 2021].

Definition 2 (Policy concentrability coefficient). We define CMε (π) to measure how well model errors transfer between the visitation distribution dπ̂ M under estimated MDP ̂ M and policy π and offline data distribution dµM⋆ ,

where TV is total variation distance, and the CMε (π) is upper bounded by ∥dπ̂ M/dµM⋆∥∞[Uehara and Sun, 2021]. It represents the discrepancy between the offline data distribution compared to the visitation distribution under policy π.

The optimal policy π⋆ of standard RL can be obtained by maximizing J(M, π) = E(s,a)∼dπM(s,a)[r(s, a)]/(1 - γ). Note that in Eq. (6), the conservative Bellman opeator T is equivalent to penalize the reward r(s, a), i.e. ̂ r(s, a) = r(s, a) - γβf(s, a). Therefore, the output policy π† of MICRO is expressed as

Algorithm 1 Model-based offline reinforcement learning with a conservative Bellman operator (MICRO)

Input: offline dataset Doffline, critic {Qωi }Ki=1 , policy πθ, environment model Tφ, and rollout length h.

Output: the final policy π† learned by the algorithm MICRO.

• 1: Initialize policy πθ, target networkQ¯ωi ←Qωi , andmodel dataset Dmodel = ∅.
• 2: Train an ensemble dynamics model {T iφ}Ni=1 on Doffline.
• 3: for t = 1, 2, · · · , niter do
• 4: Generate synthetic h-step rollouts by T iφ. Add transition data into Dmodel.
• 5: Draw samples B = (s, a, s′ , r) fromDoffline ∪Dmodel.
• 6: Compute f(s, a) according to Eq. (8).
• 7: Update policy πθ and critic network Qωi using B according to Eqs. (11) and (12).
• 8: if t % update period = 0 then
• 9: Soft update periodically: ¯ωi ← τωi + (1 - τ )¯ωi.
• 10: end if
• 11: end for

where theMf = f ̂ M+(1-f )M denotes f -interpolant MDP, which combines sampling error and model error. The M is empirical MDP that is based on sample data. The dπMf = f dπ̂ M + (1 - f)dπM is the mixed distribution of offline and model dataset. Since f(s, a) = 0 whenM=M⋆ in Eq. (9), the final learned policy π† can be denoted as

where η(π) = Edπ̂M(s,a)[f(s, a)] denotes the expectation of the discrepancy of Q value between the worst-case and average level under estimated MDP ̂ M. The Bellman operator on empirical MDPM suffers from sampling error but is unbiased in expectation, whereas Bellman operator on estimated MDP ̂ M induces bias but no sampling error [Yu et al., 2021b; Liu et al., 2023]. The performance between the true MDPM⋆ and the estimated MDP ̂ M and the empirical MDP M under policy π can be found in Lemma 2 and Lemma 3, respectively.

Theorem 1 (Robust policy improvement). Under Assumption 1, for any policy π ∈ Π, the policy π† learned by MICRO in Eq.(10), with high probability 1-δ, satisfies that J(M⋆ , π)- J(M⋆ , π† ) is upper bound by

where ω(π) = EdπM(s,a) [(Cr,δ + RmaxCT,δ)/√|D(s, a)|] reflects the sampling error under policy π. The proof of Theorem 1 can be found in Appendix A.3. Theorem 1 proves that the policy π† learned by MICRO can compete with any arbitrary policy π with a large enough dataset [Cheng et al., 2022]. The term∆1 captures the penalty discrepancy of model uncertainty between the policy π and π† . The terms ∆2 and ∆3 reflect the sampling error and model error, respectively. When the policy π is the behavior policy µ, the generated model data are more accurate under ̂ M and µ, i.e. f(s, a) approaching zero, which leads to η(µ) -η(π† ) < 0. Therefor, the upper bound of J(M⋆ , µ) - J(M⋆ , π† ) is more tight.

5.3 Practical Implementation

In practical implementation, the value function Q and the policy π are approximated via neural networks, denoted as Qω and πθ, respectively. The algorithmMICRO, presented in algorithm 1, is built on model-based policy optimization (MBPO) [Janner et al., 2019] and soft actor-critic (SAC) [Haarnoja et al., 2018]. The MLE method is used to train N ensemble dynamics models {T iφ = N(µiφ, σiφ)}Ni=1 . Before agent updating, the model data is collected through h-step rollouts and is added into model dataset Dmodel. Then, higher coverage dataset, D = Doffline ∪ Dmodel, are utilized to update agent. The principle of agent updates is described below

where ̂ T Qωi (s, a) is the target value, α is the entropy regularization coefficient, Qωi denotes the i-th Q-function, and K is the number of critic. The entropy regularization term H(s, a) := log πθ(a | s) is applied to prevent policy degradation [Haarnoja et al., 2018]. The minimization operation in above equation is to prevent overestimation of agent training [Van Hasselt et al., 2016]. For any pairs (s, a, r, s′ ) ∈ D, the target value can be defined as

where Q¯ωi represents the i-th target Q-function, and ̂ T denotes the practical conservative Bellman operator.

6 Experiments

This section focuses on the following questions. Q1: How does MICRO compare with state-of-the-art RL algorithms in standard offline RL benchmarks? Q2: How does MICRO of computation cost compare with RAMBO and ARMOR? Q3: Does the method perform better robustness than other RL algorithms? We answer the above questions using D4RL benchmark [Fu et al., 2020] with several control tasks and datasets. More hyperparameters and implementation details are provided in Appendix B. The code for MICRO is available at github.com/xiaoyinliu0714/MICRO.

6.1 Benchmark Results (Q1 and Q2)

This section compares recent model-free offline RL algorithms, such as ATAC [Cheng et al., 2022], EDAC [An et al., 2021], IQL [Kostrikov et al., 2021], and PBRL [Bai et al., 2021], as well as model-based offline RL algorithms, including ARMOR [Bhardwaj et al., 2023], RAMBO [Rigter et al., 2022], MOBILE [Sun et al., 2023], and Count-MORL [Kim and Oh, 2023] (denoted as COUNT here). These methods are evaluated on Gym domain involving three environments (HalfCheetah, Hopper, and Walker2d) with three types of datasets (medium, medium-replay, medium-expert) for each environment.

Table 1 gives the results of the average normalized score with standard deviation. The results for the above methods are obtained from the original papers. The table shows that MICRO outperforms prior RL methods on 4 of total 9 tasks and achieves comparable results in the remaining settings. Overall, MICRO achieves best average performance compared to previous RL algorithms. Table 2 compares computation time between MICRO and RAMBO under three environments (ARMOR is not compared, as code is not open-source). It shows MICRO reduces computation cost by nearly three times.

6.2 Environment Parameter Perturbations (Q3)

In most cases, it is difficult to achieve high-fidelity simulation environment, which inevitably results in sim-to-real gaps in policy learning. The simulation environment of D4RL tasks is based on the MuJoCo physics simulator. To test the performance of method in high dynamics-gap environment, we modify the parameters of MuJoCo environment, including friction and gravity, to construct high sim-to-real gap environment following [Niu et al., 2022].

Fig. 2 gives the results under various test conditions, where different gravity and friction coefficients are set for the three environments and darker colour means higher performance. The robustness of MICRO is significantly superior to MOBILE and RAMBO in sim-to-real gaps, and MICRO achieves the best generalization in Walker2d environment.

6.3 External Adversarial Attacks (Q3)

Due to the observation errors and external disturbances within real-world scenarios, three noise perturbation methods, namely RA, AD, MQ following [Yang et al., 2022; Zhang et al., 2020], are used to test the robustness of policy. The observation states are attacked through the above perturbation in evaluation phase. Details about three perturbation types are as follows.

• RA randomly samples perturbed states from the ball Uε(s) = {s′ ∈ S : d(s, s′ )p ≤ ε}, where ε denotes the scale of perturbation.
• AD is an effective perturbation based on agent policy. It selects the state s′ from Uε(s) that maximizes the gap between action distribution πθ(·|s) and πθ(·|s′ ), i.e. mins′∈Uε(s) -DJ (πθ (·|s)||πθ (·|s′ )).
• MQ performs the perturbation based policy and Qfunction. It finds the state s′ fromUε(s) that minimizes the Q-function, i.e. mins′∈Uε(s) Q(s, πθ(s′ )).

Fig. 3 gives the result of different methods with varying perturbation scale ε under three perturbation types in Walker2d environment. This figure shows that MICRO achieves better robustness than other methods under RA and AD attacks. However, the performance of MICRO is inferior to RAMBO in large MQ attack. This is mainly because this attack selects the minimum Q-value, which is more conservative for MICRO. The results in Halfcheetah and Hopper environments can be found in Appendix C. Overall, MICRO greatly outperforms MOBILE in external attacks and achieves superior robustness with less time compared with RAMBO.

7 Conclusion

MICRO is a promising novel model-based offline RL algorithm that introduces robust Bellman operator into method, where a conservative Bellman operator is proposed to improve performance and guarantee robustness with less computation. Extensive experiments demonstrate that MICRO can achieve state-of-the-art performance on standard benchmarks and better robustness under different adversarial attacks and perturbations. The main limitation of this work is the robustness slightly performs poorly under strong attacks, which should be improved in future works. MICRO is cutting-edge work in how to trade off performance and robustness in model-based offline RL field. We hope this can provide a promising direction for future research.

Acknowledgements

This work was supported in part by the National Key Research and Development Program of China under 2023YFC2415100; in part by the National Natural Science Foundation of China under Grant 62222316, Grant 62373351; in part by the Youth Innovation Promotion Association of Chinese Academy of Sciences (CAS) under Grant 2020140; and in part by the CIETencent Robotics X Rhino-Bird Focused Research Program.