Model-Based Offline Planning with Trajectory Pruning

1 Introduction

Recent advances in offline reinforcement learning (RL) have taken an important step toward applying RL to real-world tasks. Although online RL algorithms have achieved great success in solving complex tasks such as games [Silver et al., 2017] and robotic control [Levine et al., 2016], they often require extensive interaction with the environment. This becomes a major obstacle for real-world applications, as collecting data with an unmatured policy via environment interaction can be expensive (e.g., robotics and healthcare) or dangerous (e.g., industrial control, autonomous driving). Fortunately, many real-world systems are designed to log or have sufficient precollected historical states and control sequences data. Offline RL tackles this challenge by training the agents offline using the logged dataset without interacting with the environment. The key insight of recent offline RL algorithms [Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Yu et al., 2020] is to restrict policy learning stay "close" to the data distribution, which avoids the potential extrapolation error when evaluating on unknown out-of-distribution (OOD) samples.

However, implementing offline RL algorithms on real-world robotics and industrial control problems still faces some practical challenges. For example, many control agents have limited computational resources for policy learning, which require a light-weighted policy improvement procedure. Moreover, industrial control tasks often require extra control flexibility, such as occasionally changing reward signals due to altering system settings or certain devices, and involvement of statebased constraints due to safety considerations (e.g., restrict policy to avoid some unsafe states). Most existing offline RL algorithms need computationally extensive offline policy learning on a fixed task and do not offer any control flexibility.

Model-based planning framework provides an attractive solution to address the above challenges. The system dynamics can be learned offline based on the prior knowledge in the offline dataset. The policy optimization can be realized by leveraging model-predictive control (MPC) combined with a computationally efficient gradient-free trajectory optimizer such as the cross-entropy method (CEM) [Botev et al., 2013] or model-predictive path integral (MPPI) control [Williams et al., 2017]. The planning process also allows easy integration with the change of reward signals or external state-based constraints during operation, without requiring re-training agents as needed in typical RL algorithms.

Most model-based planning methods are designed for online settings. Recent studies [Wang and Ba, 2020; Argenson and Dulac-Arnold, 2021] have borrowed several ingredients of offline RL by learning a behavior cloning (BC) policy from the data to restrain trajectory rollouts during planning. This relieves OOD error during offline learning but unavoidably leads to over-restrictive planning. Limited by insufficient expressive power, behavior policies learned using BC often fit poorly on datasets generated by relatively random or multiple mixed policies. Moreover, restricting trajectory rollouts by sampling near behavior policies also impacts the performance of trajectory optimizers (e.g., CEM, MPPI require reasonable state-action space coverage or diversity in order to find good actions), and hinders the full utilization of the generalizability of the dynamics model. Dynamic models may learn and generalize reasonably well in some low-density regions if the data pattern is simple and easy to learn. Strictly avoiding OOD samples may lead to over conservative planning.

We propose the Model-Based Offline Planning with Trajectory Prunning (MOPP) framework, which allows sufficient yet safe trajectory rollouts and have superior performance compared with existing approaches. MOPP uses ensembles of expressive autoregressive dynamics models (ADM) [Germain et al., 2015] to learn the behavior and dynamics from data to capture better prior knowledge about the system. To enforce better planning performance, MOPP encourages stronger exploration by allowing sampling from behavior policy with large deviation, as well as performing the greedy max-Q operation to select potentially high reward actions according to the Q-value function evaluated from the offline dataset. At the same time, to avoid undesirable OOD samples in trajectory rollouts, MOPP prunes out problematic trajectories with unknown state-action pairs detected by evaluating the uncertainty of the dynamics model. These strategies jointly result in an light-weighted and flexible algorithm that consistently outperforms the state-of-the-art model-based offline planning algorithmMBOP [Argenson and Dulac-Arnold, 2021]. It also provides competitive performance and much better control flexibility compared with existing offline RL approaches.

2 RelatedWork

2.1 Offline Reinforcement Learning

Offline RL focuses on the setting that no interactive data collection is allowed during policy learning. The main difficulty of offline RL is the distributional shift [Kumar et al., 2019], which occurs when the distribution induced by the learned policy deviates largely from the data distribution. Policies could make counterfactual queries on unknown OOD actions, causing overestimation of values that leads to non-rectifiable exploitation error during training.

Existing offline RL methods address this issue by following three main directions. Most model-free offline RL algorithms constrain the learned policy to stay close to a behavior policy through deviation clipping [Fujimoto et al., 2019] or introducing additional divergence penalties (e.g., KL divergence, MMD or BC regularizer) [Wu et al., 2019; Kumar et al., 2019; Fujimoto and Gu, 2021; Xu et al., 2021]. Other model-free offline RL algorithms instead learn a conservative, underestimated value function by modifying standard Bellman operator to avoid overly optimistic value estimates on OOD samples [Kumar et al., 2020; Liu et al., 2020; Kostrikov et al., 2021; Buckman et al., 2020; Xu et al., 2022]. Model-based offline RL methods [Yu et al., 2020; Kidambi et al., 2020; Zhan et al., 2022], on the other hand, incorporate reward penalty based on the uncertainty of the dynamics model to handle the distributional shift issue. The underlying assumption is that the model will become increasingly inaccurate further from the behavior distribution, thus exhibits larger uncertainty. All these algorithms require a relatively intensive policy learning process as well as re-training for novel tasks, which make them less flexible for real-world control systems.

2.2 Model-Based Planning

The model-based planning framework provides a more flexible alternative for many real-world control scenarios. It does not need to learn an explicit policy, but instead, learns an approximated dynamics model of the environment and use a planning algorithm to find high return trajectories through this model. Online planning methods such as PETS [Chua et al., 2018], POLO [Lowrey et al., 2019], POPLIN [Wang and Ba, 2020], and PDDM [Nagabandi et al., 2020] have shown good results using full state information in simulation and on real robotic tasks. These algorithms are generally built upon an MPC framework and use sample efficient random shooting algorithms such as CEM or MPPI for trajectory optimization. The recent MBOP [Argenson and Dulac-Arnold, 2021] further extends model-based planning to offline setting. MBOP is an extension of PDDM but learns a behavior policy as a prior for action sampling, and uses a value function to the extend planning horizon. The problem of MBOP is that its performance is strongly dependent on the learned behavior policy, which leads to over-restrictive planning and obstructs the full potential of the trajectory optimizer and the generalizability of the dynamics model. In this work, we propose MOPP to address the limitations of MBOP, which provides superior planning while avoids undesirable OOD samples in trajectory rollouts.

3 Preliminaries

We consider the Markov decision process (MDP) represented by a tuple as (S, A, P, r, γ), where S, A denote the state and action space, P(st+1|st, at) the transition dynamics, r(st, at) the reward function and γ ∈ [0, 1] the discounting factor. A policy π(s) is a mapping from states to actions. We represent R = ∑∞t=1 γ t r(st, at) as the cumulative reward over an episode, which can be truncated to a specific horizon H as RH. Under offline setting, the algorithm only has access to a static dataset B generated by arbitrary unknown behavior policies πb, and cannot interact further with the environment. One can use parameterized function approximators (e.g., neural networks) to learn the approximated environment dynamics fm(st, at) and behavior policy fb(st) from the data. Our objective is to find an optimal policy π∗ (st) = arg maxa∈A ∑Ht=1 γ t r(st, at) given only dataset B that maximizes the finite-horizon cumulative reward with γ = 1.

4 The MOPP Framework

MOPP tackles the fundamental dilemma between the restrictions of offline learning and high-performance planning. Planning by sampling strictly from behavior policy avoids potential OOD samples. However, this also leads to over-restrictive planning, which forbids sufficient exploitation of the generalizability of the learned dynamics model. MOPP provides a novel solution to address this problem. It allows more aggressive sampling from behavior policy fb with boosted variance, and performs max-Q operation on sampled actions based on a Q-value function Qb evaluated based on behavioral data. This treatment can lead to potential OOD samples, so we simultaneously evaluate the uncertainty of the dynamics models to prune out problematic trajectory rollouts. To further enhance the performance, MOPP also uses highly expressive autoregressive dynamics model to learn the dynamics model fm and behavior policy fb, as well as uses the value function to extend planning horizon and accelerate trajectory optimization.

4.1 Dynamics and Behavior Policy Learning

We use autoregressive dynamics model (ADM) [Germain et al., 2015] to learn the probabilistic dynamics model (rt, st+1) = fm(st, at) and behavior policy at = fb(st). ADM is shown to have good performance in several offline RL problems due to its expressiveness and ability to capture nonunimodal dependencies in data [Ghasemipour et al., 2021].

The ADM architecture used in our work is composed of several fully connected layers. Given the input x (e.g., a state for fb or a state-action pair for fm), an MLP first produces an embedding for the input, separate MLPs are then used to predict the mean and standard deviation of every dimension of the output. Let oi denote the i-th index of the predicted output o and o[<i] represent a slice first up to and not including the i-th index following a given ordering. ADM decomposes the probability distribution of o into a product of nested conditionals: p(o) = ∏Ii p(oi|x, o[<i]). The parameters θ of the model p(o) can be learned by maximizing the log-likelihood on dataset B: L(θ|B) = ∑x∈B [ ∑|o| i=1 log p(oi|x, o[<i])].

ADM assumes underlying conditional orderings of the data. Different orderings can potentially lead to different model behaviors. MOPP uses ensembles of K ADMs with randomly permuted orderings for dynamics and behavior policy, which further enhances the overall model expressiveness.

4.2 Value Function Evaluation

Introducing a value function to extend the planning horizon in model-based planning algorithms have been shown to greatly accelerate and stabilize trajectory optimization in both online [Lowrey et al., 2019] and offline [Argenson and Dulac-Arnold, 2021] settings. We follow this idea by learning a Q-value function Qb(st, at) using fitted Q evaluation (FQE) [Le et al., 2019] with respect to actual behavior policy πb and γ ′ < 1:

The state value function Vb(st) = Ea∼πb Q(st, a) is further evaluated. This provides a conservative estimate of values bond to behavioral policy. MOPP adds Vb to the cumulative returns of the trajectory rollouts to extend the planning horizon. This helps shorten horizon H needed during planning. Besides, MOPP uses Qb to perform the max-Q operation and guide trajectory rollouts toward potentially high reward actions.

4.3 Offline Planning

MOPP is built upon the finite-horizon model predictive control (MPC) framework. It finds a locally optimal policy and a sequence of actions up to horizon H based on the local knowledge of the dynamics model. At each step, the first action from the optimized sequence is executed. In MOPP, we solve a modified MPC problem which uses value estimate Vb to extend the planning horizon:

Obtaining the exact solution for the above problem can be rather costly, instead, we introduce a new guided trajectory rollout and pruning scheme, combined with an efficient gradient-free trajectory optimizer based on an extended version of MPPI [Williams et al., 2017; Nagabandi et al., 2020].

Guided Trajectory Rollout. The key step in MOPP is to generate a set of proper action sequences to roll out trajectories that are used by the trajectory optimizer. Under offline settings, such trajectory rollouts can only be performed with the learned dynamics model fm. Using randomly generated actions can lead to large exploitation errors during offline learning. MBOP uses a learned behavior policy as a prior to sample and roll out trajectories. This alleviates the OOD error but has several limitations. First, the learned behavior policy could have insufficient coverage on good actions in low-density regions or outside of the dataset distribution. This is common when the data are generated by low-performance behavior policies. Moreover, the dynamics model may generalize reasonably well in some low-density regions if the dynamics pattern is easy to learn. Strictly sampling from the behavior policy limits sufficient exploitation of the generalizability of the learned dynamics model. Finally, the lack of diversity in trajectories also hurts the performance of the trajectory optimizer.

MOPP also uses the behavior policy to guide trajectory rollouts, but with a higher degree of freedom. Let µa (st) = [µa1 (st), · · · , µa|A| (st)]T , σa (st) = [σa 1 (st), · · · , σa |A| (st)]T denote the mean and standard deviation (std) of each dimension of the actions produced by the ADM behavior policy fb(st). MOPP samples and selects an action at time step t as:

where σM > 0 is the std scaling parameter. We allow it to take larger values than maxσa to enable more aggressive sampling. In MBOP, the actions are sampled by adding a very small random noise on the outputs of a deterministic behavior policy, which assumes uniform variance across different action dimensions. By contrast, MOPP uses the means µa and std σa boosted by σM to sample actions (µa , σa from the ADM behavior policy fb). This allows heterogeneous uncertainty levels across different action dimensions while preserves their relative relationship presented in data.

We further perform the max-Q operation on the sampled actions based on Qb to encourage potentially high reward actions. Note that Qb is evaluated entirely offline with respect to the behavior policy, which provides a conservative but relatively reliable long-term prior information. MOPP follows the treatment in PDDM and MBOP that mixes the obtained action ˆat with the previous trajectory using a mixture coefficient β to roll out trajectories with the dynamics model fm.

Trajectory Pruning. The guided trajectory rollout produces a set of trajectory sequences T = {T1, . . . , TN}, with Tn = {(ant , snt )}H-1 t=0 , n ∈ {1, . . . ,N}, which may contain undesirable state-action pairs that are out-of-distribution or have large prediction errors using the dynamics model. Such samples need to be removed, but we also want to keep OOD samples at which the dynamics model can generalize well to extend the knowledge beyond the dataset B. The uncertainty quantification method used in MOReL [Kidambi et al., 2020] provides a nice fit for our purpose, which is evaluated as the prediction discrepancy of dynamics models f lm, l ∈ 1, . . . ,K in the ensemble: disc(s, a) = maxi,j ∥∥f im(s, a) - f jm(s, a)∥∥22 .

Let U be the uncertainty matrix that holds the uncertainty measures Un,t = disc(snt , ant ) for each step t of trajectory n in T . MOPP filters the set of trajectories using the trajectory pruning procedure T rajP rune(T , U). Denote Tp := {Tn|Un,t < L, ∀t, n}, trajectory pruning returns a refined trajectory set for offline trajectory optimization as:

where L is the uncertainty threshold, Nm is the minimum number of trajectories used to run the trajectory optimizer (set as Nm = 0.2⌊N⌋ in our implementation). The intuition of trajectory pruning is to remove undesirable state-action samples and produce a set of low uncertainty trajectories. MOPP first constructs a filtered trajectory set Tp that only contains trajectories with every state-action pair satisfying the uncertainty threshold. If Tp has less than Nm trajectories, we sort the remaining trajectories in T - T p by the cumulative uncertainty (i.e. ∑t Un,t with Tn ∈ T - T p). The top Nm - |T p| trajectories in the sorted set with the lowest overall uncertainty are added into Tp as the final refined trajectory set.

Trajectory Optimization. MOPP uses an extended version of the model predictive path integral (MPPI) [Williams et al., 2017] trajectory optimizer that is used similarly in PDDM [Nagabandi et al., 2020] and MBOP [Argenson and DulacArnold, 2021]. MOPP shoots out a set of trajectories T f using the previous guided trajectory rollout and pruning procedure. Let Rf = {R1, . . . , R|T f |} be the associated cumulative returns for trajectories in Tf , the optimized action is obtained by re-weighting the actions of each trajectory according to their exponentiated returns:

where ant is the action at step t of trajectory Tn ∈ Tf and κ is a re-weighting factor. The full algorithm is in Algorithm 1.

5 Experimental Results

We evaluate and compare the performance of MOPP with several state-of-the-art (SOTA) baselines on standard offline RL benchmark D4RL [Fu et al., 2020]. We conduct experiments on the widely-used MuJoCo tasks and the more complex Adroit hand manipulation tasks. All results are averaged based

Algorithm 1 Complete algorithm of MOPP

Require: Offline dataset B

• 1: Train Qb, K1 dynamics models f lm and K2 behavior policies f l b on B. Initialize A∗t = 0, ∀t ∈ {0, . . . , H - 1}. 2: for τ = 0...∞do 3: Observe sτ , initialize T ,R= ∅ 4: for n = 1, . . . , N do 5: s0 = sτ , Rn = 0, Tn = null 6: for t = 0 . . . H - 1 do 7: Sample action ˆat using f l b(st) (l randomly picked from 1 . . . K2) according to Eq.(3) 8: ˜at = (1 - β)ˆat + βA∗t+1, (A∗H = A∗H-1) 9: Append (st, ˜at) into trajectory Tn 10: st+1 = f l′m(st, ˜at)s , l′ randomly picked from 1 . . .K1 11: Rn ←Rn + 1 K1 ∑K1 k=1 f km(st, ˜at)r 12: Un,t = maxi,j ∥∥f im(st, ˜at) - f jm(st, ˜at)∥∥22 13: end for 14: Compute Vb(sH) = ∑KQ i=1 Qb(sH, ai)/KQ, {ai}KQ i=1 are randomly sampled from f l b(sH ) 15: Rn ←Rn +Vb(sH), T ←T ∪ {Tn}, R←R∪ {Rn} 16: end for 17: Compute Tf = TrajPrune(T ,U) according to Eq.(4) 18: Update A∗t , ∀t = {0, . . . ,H -1} using Tf and Eq.(5) 19: Return optimized aτ = A∗0 20: end for

on 5 random seeds, with 20 episode runs per seed. The reported scores are normalized between 0 to 100, with 0 and 100 correspond to a random policy and an expert SAC [Haarnoja et al., 2018] policy respectively. We also included a comprehensive ablation study on each component in MOPP and evaluate its adaptability under varying objectives and constraints.

5.1 Comparative Evaluations

Performance onMoJoCo Tasks. We evaluate the performance of MOPP on three tasks (halfcheetah, hopper and walker2d) and four dataset types (random, medium, mixed and med-expert) in the D4RL benchmark. We compare in Table 1 the performance of MOPP with several SOTA baselines, including model-based offline RL algorithms MBPO [Janner et al., 2019] and MOPO [Yu et al., 2020], as well as the SOTA model-based offline planning algorithm MBOP.

MOPP outperforms MBOP in most tasks, sometimes by a large margin. It is observed that MBOP is more dependent on its special behavior policy at = f ′b (st, at-1), which include previous step's action as input. This will improve imitation performance under datasets generated by one or limited policies, as the next action may be correlated with the previous action, but could have negative impact on high-diversity or complex datasets (e.g., random and mixed). On the other hand, MOPP substantially outperforms its ADM behavior policy fb especially on the med-expert tasks, which shows great planning improvement upon a learned semi-performance policy.

Comparing with model-based offline RL methods MBPO and MOPO, we observe that MOPP performs better in medium and med-expert datasets, but less performant on highervariance datasets such as random and mixed. Model-based offline RL methods can benefit from high-diversity datasets, in which they can learn better dynamics models and apply RL

to find better policies. It should also be noted that training RL policies until convergence is costly and not adjustable after deployment. This will not be an issue for a light-weighted planning method like MOPP, as the planning process is executed in operation and suited well for controllers that require extra control flexibility. MOPP performs strongly in the med-expert dataset, which beats all other baselines and achieves close to or even higher scores compared with the expert SAC policy. This indicates that MOPP can effectively recover the performant data generating policies in the behavioral data and use planning to further enhance their performance.

Performance on Adroit Tasks. We also evaluate the performance of MOPP in Table 2 on more complex Adroit highdimensional robotic manipulation tasks with sparse reward, involving twirling a pen, hammering a nail, opening a door and picking/ moving a ball. The Adroit datasets are particularly hard, as the data are collected from a narrow expert data distributions (expert), human demonstrations (human), or a mixture of human demonstrations and imitation policies (cloned). Model-based offline RL methods are known to perform badly on such low-diversity datasets, as the dynamics models cannot be learned well (e.g., see results of MOPO). We compare MOPP with two more performant model-free offline RL algorithms, BCQ [Fujimoto et al., 2019] and CQL [Kumar et al., 2020]. It is found that although MOPP is a model-based planning method, it performs surprisingly well in most of the cases. MOPP consistently outperforms the SOTA

offline planning method MBOP, and in many tasks, it even outperforms the performant model-free offline RL baselines BCQ and CQL. The better performance of MOPP is a joint result of the inheritance of both an imitative behavior policy and more aggressive planning with the learned dynamics model.

5.2 Ablation Study

We conduct ablation experiments on walker2d-med-expert task to understand the impact of key elements in MOPP. We first investigate in Figure 1(a) the impact of sampling aggressiveness (controlled by std scaling parameter σM), as well as its relationship with the max-Q operation and trajectory pruning. It is observed that reasonably boosts the action sampling variance (e.g., increase σM from 0.01 to 0.5) is beneficial. But overly aggressive exploration (σM = 1.0) is detrimental, as it will introduce lots of undesired OOD samples during trajectory rollouts. When most trajectory rollouts are problematic, the trajectory pruning procedure is no longer effective, as there have to be at least Nm trajectories in order to run the trajectory optimizer. When σM is not too large, trajectory pruning is effective to control the planning variance and produces better performance, as is shown in the difference between MOPPnoP and MOPP under σM = 0.5. Moreover, the max-Q operation in the guided trajectory rollout increases the sampling aggressiveness. When σM is moderate, MOPP achieves a higher score than MOPP-noMQ. But when σM = 1.0, the less aggressive MOPP-noMQ is the only variant of MOPP that is still possible to produce high episode returns. These suggest that carefully choosing the degree of sampling aggressiveness is important for MOPP to achieve the best performance.

We further examine the impacts of value function Vb and max-Q operation on different planning horizons in Figure 1(b). It is observed that even with a very short horizon (H = 2 and 4), MOPP can attain good performance that is comparable to results using longer planning horizons. Moreover, MOPP achieves significantly higher scores compared with MOPPnoMQ-noV. We found that using max-Q operation on sampled actions provides stronger improvements, as MOPP-noMQ consistently perform worse than MOPP-noV. This might because that max-Q operation is performed at every step, while the value function is only added to the end of the cumulative return of a trajectory, thus providing stronger guidance on trajectory

• (a) Impacts of max-Q operation and trajectory pruning with different sampling aggressiveness
• (c) Impact of uncertainty threshold L (σM = 0.5, H = 2)

Figure 1: Ablation study on walker2d-med-expert. noMQ, noP, noV indicate removal of max-Q operation, trajectory pruning and Vb.

3000

2000

1000

0

1000

2000

0.10

0.05

0.00

0.05

0.10

0.15

Data MOPP-oldR MOPP-newR MOPP-newR-

newQ

Policy

• (b) z-position of the trajectories

Data MOPP-oldR MOPP-newR MOPP-newR-

newQ

Policy

• (a) Performance under new reward

(c) Performance under x-velocity constraints.

Figure 2: Performance on halfcheetah-jump (a, b) and halfcheetah-constrained (c) tasks.

rollouts towards potentially high reward actions.

Finally, Figure 1(c) presents the impact of uncertainty threshold L in trajectory pruning. We observe that both strictly avoid (L = 0.1) or overly tolerant (L = 10.0) unknown stateaction pairs impact planning performance. Reasonably increase the tolerance of sample uncertainty (L = 2.0) to allow sufficient exploration leads to the best result with low variance.

5.3 Evaluation on Control Flexibility

A major advantage of planning methods lies in their flexibility to incorporate varying objectives and extra constraints. These modifications can be easily incorporated in MOPP by revising the reward function or pruning out unqualified trajectory rollouts during operation. We construct two tasks for evaluation:

• halfcheetah-jump: This task adds incentives on the zposition in the original reward function of halfcheetah, encouraging agent to run while jumping as high as possible.
• halfcheetah-constrained task adds a new constraint (x-velocity≤10) to restrain agent from having very high xvelocity. Two ways are used to incorporate the constraint: 1) add reward penalty for x-velocity>10; 2) add penalties on the uncertainty measures U to allow trajectory pruning to filter out constraint violating trajectory rollouts.

Figure 2 (a), (b) shows the performance of MOPP on the halfcheetah-jump task. By simply changing to the new reward function (MOPP-newR), MOPP is able to adapt and improve upon the average performance level in data and the original model (MOPP-oldR). The performance will be further improved by re-evaluating the Q-function (MOPP-newRnewQ). The offline evaluated value function Qb and the max-Q operation could have negative impact when the reward function is drastically different. In such cases, one only needs to re-evaluate a sub-component (Qb under the new reward) of MOPP to guarantee the best performance rather than re-train the whole model as in typical RL settings. Evaluating Qb via FQE is achieved by supervised learning, which is computationally very cheap compared with a costly RL procedure.

Figure 2(c) presents the performance on the halfcheetah -constrained task. The original MOPP without constraint (MOPP-oldR) has lots of constraint violations (x-velocity>10). Incorporating a constraint penalty in reward (MOPP-newR) and pruning out constraint violating trajectories (MOPP-RC) achieve very similar performance. Both models effectively reduce constraint violations and have limited performance deterioration due to the extra constraint. Adding constraint penalty in the reward function while re-evaluating the Qb via FQE (MOPP-newR-newQ) leads to the safest policy.

6 Conclusion

We propose MOPP, a light-weighted model-based offline planning algorithm for real-world control tasks when online training is forbidden. MOPP is built upon an MPC framework that leverages behavior policies and dynamics models learned from an offline dataset to perform planning. MOPP avoids overrestrictive planning while enabling offline learning by encouraging more aggressive trajectory rollout guided by the learned behavior policy, and prunes out problematic trajectories by evaluating the uncertainty of dynamics models. Although MOPP is a planning method, benchmark experiments show that it provides competitive performance compared with the state-of-the-art offline RL and model-based planning methods.

Episode Return(new reward)

Z Position

Acknowledgments

This work was partially supported by Tsinghua University (AIR) - AsiaInfo Technologies (China) Joint Research Center for 5G Intelligent Internet of Things (20203910074).