Diversification of Adaptive Policy for Effective Offline Reinforcement Learning

1 Introduction

Reinforcement Learning (RL) has emerged as a powerful approach for decision-making in complex real-world scenarios, as evidenced by its successes in domains such as the games of chess, shogi, and go [Silver et al., 2017], and real-time strategy video games [OpenAI et al., 2019]. However, traditional online RL methods involve exploratory interaction with the environment, which makes them impractical, costly, and potentially unsafe in various real-world applications.

Thereby, offline RL has recently garnered significant attention [Fujimoto et al., 2019; Levine et al., 2020]. Offline RL involves training a policy solely on a fixed dataset obtained from the environment, without any additional interaction with the environment. However, directly applying existing online RL algorithms [Mnih et al., 2013; Haarnoja et al., 2018] to the offline setting often results in poor performance due to the distribution shift between the learned policy and the data-collected policy. This distribution shift introduces extrapolation errors in the value function estimation for unseen actions. As a result, several model-free offline RL algorithms [Fujimoto et al., 2019; Kumar et al., 2019; Nair et al., 2021] have focused on solving the constrained policy optimization problem, where the learned policy is encouraged to select actions from the dataset.

Alternately, there have been offline RL methods that adopt model-based RL approaches, which demonstrate superior generalizability by leveraging the learned dynamics model, in the offline setting. However, these methods still suffer from model overfitting issues and extrapolation errors in regions that are not explored by the data-collected policies [Kidambi et al., 2020; Yu et al., 2021]. This is because the dynamics model is learned solely from the offline dataset, which may not capture the full dynamics of the underlying environment. To address these challenges, previous works have proposed pessimistic framework [Yu et al., 2020; Rigter et al., 2022]. For example, model-based offline policy optimization (MOPO; [Yu et al., 2020]) incorporates penalty terms to adjust the reward signals based on the degree of uncertainty inherent in the model.

Yet, these pessimistic approaches in model-based offline RL fall short in handling scenarios where the agent encounters out-of-support regions. Here, the Bayes-adaptive decision framework presents a valuable alternative method [Ross et al., 2007; Ghavamzadeh et al., 2015]. One of the primary benefits of the Bayesian method lies in its capacity to reason over all potential models by calculating their posterior distribution from the given dataset. A policy trained on this posterior distribution can naturally adapt during online execution, which may lead to enhanced performance in comparison to a policy trained by the pessimistic approach. AVE-P [Ghosh et al., 2022] capitalizes on this advantage by training the adaptive policy through belief state tracking, using an ensemble of value functions to interact indirectly with the models. In a similar vein, MAPLE [Chen et al., 2021] employs a related approach by explicitly constructing an ensemble of dynamics models. This ensemble enables the generation of simulations for trajectories, which are subsequently utilized to augment the training of the adaptive policy.

These prior works used to construct the set of dynamics models with ensemble structures, facilitating the adaptive policy learning in practical applications. However, when the dynamics models are similar, they might not generate sufficiently diverse behaviors within the out-of-support region.

The lack of diversity in behaviors can impose limitations on the algorithm's capacity for effective generalization, leading to sub-optimalitiy in real-world scenarios. In practice, these methods often necessitate a considerable number of ensemble members to achieve the desirable level of generalization ability [Ghosh et al., 2022; Ghosh et al., 2021]. Consequently, the establishment of a more diverse set of models holds significant importance.

In this work, our goal is to enrich the diversity of the adaptive policy, all the while utilizing a limited number of models within the model-based framework. To do that, we first discuss which dynamics models are considered to encourage the diversification of the adaptive policy within an offline setting. These dynamics models should adequately represent the support region of the offline dataset, and also manifest distinct transitions in out-of-support regions. We particularly focus on the optimal trajectories generated by the optimal policies associated with different dynamics models. When certain optimal trajectories demonstrate similarities, it suggests that the corresponding MDPs do not lead to significantly different decision-making scenarios. Leveraging this insight, we introduce an information-theoretic objective that seeks to maximize the mutual information between the optimal trajectory and the identity of the underlying dynamics models.

We implement this idea and propose a Model-based Diverse Adaptive Policy Learning (MoDAP) algorithm for offline RL. MoDAP is an iterative training approach involving both the policy and dynamics models. We train the policy to maximize its expected return across dynamics models while concurrently training the dynamics models to facilitate policy diversification through the information-theoretic objective. We provide further details about MoDAP in Section 4. Lastly, we evaluate MoDAP across a range of D4RL [Fu et al., 2020] and NeoRL [Qin et al., 2022] datasets with a limited number of dynamics models, demonstrating its superior and competitive performance. Further details regarding these experiments are provided in Section 5.

2 RelatedWork

2.1 Offline RL

In model-free offline RL, the distribution shift between the learned policy and the data-collected policy introduces extrapolation errors [Kumar et al., 2019] in the estimation of the value function for unseen actions in the dataset, presenting a challenge for naively adopting online algorithms. To overcome this challenge, extensive research has been conducted to formulate offline RL as a constrained optimization problem [Kumar et al., 2019; Fujimoto et al., 2019; Nair et al., 2021; Wang et al., 2020; Kostrikov et al., 2021]. The common approaches [Fujimoto and Gu, 2021; Fujimoto et al., 2019; Kumar et al., 2019] ensure that the optimized policy remains close to the behavior policy. Other strategies involve estimating conservative Q-values [Kumar et al., 2020a] by incorporating regularization terms that assign lower values to unseen state-action pairs compared to observed ones. EDAC [An et al., 2021], another notable technique, enforces the expected minimum action value through the use of diversified Q-ensemble and clipped Q-learning.

Moving to model-based approaches, these algorithms utilize the learned dynamics model derived from the dataset to simulate the policy. A significant challenge arises when the policy takes actions that were absent from the training dataset, leading to unseen action-state pairs. In such cases, the estimated model may provide transition dynamics that differ from the true environment, causing potential unreliability in the model's predictions. To mitigate this issue, a prevalent strategy in model-based offline approaches introduces constructing a pessimistic MDP with a penalized reward function. For instance, MOPO [Yu et al., 2020] introduces model ensemble uncertainty into reward signal, while MORel [Kidambi et al., 2020] utilizes the state-action detector to make penalized reward. Since these methods primarily concentrate on decision-making within the known portion of the MDP, they may have inherent limitations when the behavior policy is sub-optimal, as the constrained formulations do not guarantee optimal performance [Chen et al., 2021; Ghosh et al., 2022].

2.2 Bayesian RL and Offline Adaptive Policy

Bayesian RL has proven to be effective in striking a balance between exploration and exploitation in online RL, resulting in sample efficiency and optimality [Ross et al., 2007; Asmuth et al., 2009; Kolter and Ng, 2009; Poupart et al., 2006]. To apply Bayesian principles to model-based offline RL, we assume the set of dynamics models that involve the true environment. The agent takes the action based on the posterior distribution of the models, or a sufficient statistic of interaction history, at deployment time, which naturally makes the policy adaptive. An illustrative example of this approach is MAPLE [Chen et al., 2021], which employs a Recurrent Neural Network (RNN) policy to infer environmental contexts from historical information. Alternatively, in the model-free approach, APE-V [Ghosh et al., 2022] takes an indirect strategy by engaging with the value function of each model, updating its belief state based on historical information.

2.3 Diversity and RL

There exists various range of research that leverages the concept of diversity in RL, often optimizing informationtheoretic objectives. In the domain of unsupervised skill discovery, agents have to get valuable skills without explicit reward signals. A prominent example is DIAYN [Eysenbach et al., 2019], which attains diverse skills by maximizing the mutual information between states and skills. Additionally, DADS [Liu et al., 2021] focuses on skill diversification by maximizing mutual information between the subsequent state and the skill, conditioned on the current state.

On the other hand, SMEARL [Kumar et al., 2020b] aims to enhance adaptability to unseen tasks by generating a diverse set of sub-optimal policies within the specific MDP. This is achieved through the utilization of an augmented reward derived from a diversification objective. Even though our work also utilizes the information-theoretic objective, its purpose and detailed algorithm are clearly different as we concentrate on training a set of dynamics models to facilitate the diversification of adaptive policy.

3 Preliminaries

3.1 MDPs and Offline RL

We consider a Markov Decision Process (MDP) as a tuple m = (S, A, Pm, r, ρ0, γ), where S is the state space, A is the action space, Pm : S×A → ∆(S) is the state transition probability function, and r : S ×A → R is the reward function, ρ0 ∈ ∆(S) is the initial state distribution, and γ ∈ (0, 1] is the discount factor. The objective of RL is to obtain a policy, πm : S → ∆(A), that maximizes the discounted cumulative reward:

The solution to the optimization problem defined by Eq. (1) is denoted as π∗m.

While the standard RL assumes online interaction with the environment during training, offline RL does not allow such interaction. Instead, we can only access a pre-collected dataset that is gathered by (unknown) mixed behavior policies. Since the pre-collected dataset usually covers only a subset of the states and actions, the goal of offline RL is to find the best policy given the limited information available in the dataset.

3.2 Model-based Offline RL Algorithm

Model-based Offline RL algorithms aim to train a policy by leveraging a learned dynamics model Pϕ(s′ |s, a). This model is typically trained using maximum likelihood estimation on the offline dataset, maxϕ E(s,a,s′ )∼Doff [logPϕ(s′ |s, a)]. 1 Once the model is learned, it is utilized to simulate rollouts of the policy, and the resulting rollout data is then used to optimize the policy.

Instead of focusing on building a single MDP, we want to work with a set of MDPs, each element representing an MDP with transition dynamics consistent with the dataset but different underlying transition dynamics in other regions of the state-action space. These MDPs comprise an ensemble of hypotheses about the true environment, and act as simulation environments to train the adaptive policy. The policy learns to switch from one MDP to another based on the iteraction, making it adaptive and hence generalize better, compared to learning from a single MDP.

1 Since the transition and reward function are unknown, they are usually trained together, maxϕ E(s,a,r,s′)∼Doff [log Pϕ(r, s′ |s, a)].

One thing we need to notice is that the generalization capacity of the adaptive policy is significantly influenced by the characteristic of the MDPs within the ensemble. For instance, the performance of the approach is particularly sensitive to the number of ensemble models used. When working with a limited number of models, the policy's performance tends to be poor, conversely, as the ensemble size increases, the policy's effectiveness improves significantly. However, having a larger number of ensemble models for learning well-generalized policies might not always be feasible due to the practical constraint of computational resources. MAPLE [Chen et al., 2021], despite its objective of improving decision-making in out-of-support regions through enhanced generalizability, adopts a pessimistic approach to work with a limited number of models. It involves uncertainty over the MDPs as a penalty term in the reward signal. Thus, the challenge lies in the construction of a set of MDPs that remains manageable in size while effectively enhancing policy learning and generalization.

In this work, we propose a novel method to construct an effective yet small number of MDPs in the set for training the adaptive policy and the value function. Similar to established methodologies, our approach employs model-based policy optimization (MBPO) [Jiang et al., 2020], utilizing a standard actor-critic RL algorithm. To generate synthetic data for training, MBPO conducts k-step rollouts originating from states s ∈ Doff, subsequently adding this data to the DPϕ . During policy training, minibatches of data are sampled from the combined dataset Doff ∪ DPϕ . Each data point within this batch is drawn from either of the real data D with a probability f or the synthetic data DPϕ with a probability 1 - f. Although our methodology shares similarities with MBPO regarding the generation and usage of synthetic datasets for training, it distinguishes itself in the utilization of history for value function and policy, as described in the next section.

4 Method

Before introducing our method, we begin with discussing the dynamics models that are essential for effective learning of the adaptive policy in offline RL.

Which models do we need? To facilitate a clear understanding of our method, we consider a scenario where the MDPs share the same state space, action space, and reward function, but differ in their transition functions Pm with the subscript/identity m ∈ [n] being the index of MDPs. 2 Instead of directly focusing on different transition dynamics {Pm}nm=1 , we narrow our focus to the optimal trajectories, denoted as {τ π∗m}nm=1 , which are generated by executing the corresponding optimal policies {π∗m}nm=1 for each MDP. When certain optimal trajectories exhibit similarities, it indicates that the corresponding MDPs do not yield distinct decision-making scenarios. The presence of such dynamics models within the set is inefficient and restricts the diversity for effective policy adaptation and generalization. To address this, we emphasize the importance of constructing a set of MDPs in which each MDP exhibits significant differentiation from the others.

2 This assumption does not compromise the generality of our method, as it can be applied when the reward is MDP-dependent while the state space, action space, and transition function are shared.

4.1 Promoting Diversity in Dynamics Models

To achieve this goal, we employ an information-theoretic approach that aims to maximize the Mutual Information (MI)

where M and τ π∗M represent the random variables associated with the MDP's identity and its corresponding optimal trajectory, respectively. The equation quantifies the information gain concerning the optimal trajectory when the MDP is known, measuring trajectory-aware diversity.

To optimize Eq. (2), we decompose p(τ π∗M |m) as follows:

where π∗M is the optimal policy in the MDP M. Since M is a discrete random variable, we can compute the marginal p(τ π∗M ) as ∑m p(m)p(τ π∗M |m) where p(m) is prior.

From Eq. (2), maximizing this objective equates to increasing the entropy of optimal trajectories (H(τ π∗M)) while makingM informative about the corresponding optimal trajectory (-H(τ π∗M |M)). It promotes the diversity of policy behaviors among MDPs. We provide the detailed theoretical interpretation of Eq. (2) in Appendix A.

Adaptive policy. Instead of learning independent optimal policies {π∗m} of each MDPs, our approach aims to train a single, adaptive policy π that covers all MDPs. Specifically, it maps history ht = {s0:t-1, a1:t-1, r1:t-1}, a sequence of observed transitions up to timestep t - 1, to action at. We approximate the optimal policy π∗m with this history-based policy, π. It notes that as π interacts with the environment at test time, it can identify the underlying transition dynamics by tracking the history. Over time, it becomes gradually the optimal policy. As a consequence of this approach, we maximize I(τ π ;M) instead of Eq. (2), where τ π is the trajectory obtained from the adaptive policy π over MDPs.

Overall objective. We aim to determine the set of estimated dynamics models {Pϕm }nm=1 given a fixed adaptive policy π. To achieve this, we solve the following optimization objective for each model m:

The set of dynamics models {Pϕm }nm=1 should be capable of modeling the transitions observed in the offline dataset, also. It becomes necessary to include MLE term as usual to ensure proper model training:

Algorithm 1 MoDAP

Require: the parameters θ, ψ of the policy πθ and GRUψ, the offline dataset Doff and constant, λ

• 1: {Pϕi }ni=1 , ˆ r ← the set of dynamics models and reward function from Doff
• 2: for iter = 0, 1, 2... do
• 3: Randomly sample s, z fromDoff
• 4: Set s0, z0 ← s, z
• 5: Randomly select dynamics model from {Pϕi }ni=1
• 6: for t = 0, 1, 2, ..., k - 1 do
• 7: Sample at ∼ πθ(at|st, zt)
• 8: Rollout one step
• 9:
• 10:
• st+1 ∼ Pϕi (· | st, at), rt+1 ← ˆ r(st, at) zt+1 ← GRUψ(st+1, at, zt)
• 11: Add {st, zt, at, rt, st+1, zt+1} to DPϕ 12: end for
• 13: // Policy Update //
• 14: Update πθ, GRUψ by SAC with DPϕ ∪Doff
• 15: // Model Update //
• 16: Estimate LMI for the generated k-step trajectory
• 17: Update {Pϕi }ni=1 by maximizing λLMI +LMLE
• 18: end for

4.2 MoDAP

To train the adaptive policy, we utilize a recurrent neural network (RNN), which is known for its effectiveness in incorporating historical information in meta RL [Duan et al., 2017; Chen et al., 2021]. Specifically, we use GRU [Chung et al., 2014] to encode ht into the hidden state zt such that zt = GRU(ht). The policy takes action based on both the current state st and the hidden state zt. Overall, we optimize the adaptive policy πθ to maximize the expected returns over the MDPs:

and we assume that p(·) is a uniform distribution.

We now present our algorithm, MoDAP (Model-based Diverse Adaptive Policy Learning) in Algorithm 2. Before training the adaptive policy, the dynamics models and reward function are pre-trained using MLE on offline dataset Doff. Notably, ˆ r is defined as the mean value over the ensemble's output. Due to the finite number of models in the ensemble, planning during a long-horizon results in the generation of unrealistic trajectories in the training dataset for policy, leading to performance degradation. To mitigate this issue, we adopt a fixed-horizon rollout approach for planning. We start planning from an arbitrary time-step t within the offline dataset, collecting trajectories for a fixed horizon of length k, and gathering the dataset required for training the policy.

To provide further detail, our approach involves the following steps during each iteration: In each iteration, we uniformly choose a dynamics model from our ensemble since we assume a uniform prior distribuiton p(m). We employ the selected dynamics model and the adaptive policy to generate trajectories. These trajectories are added to the synthetic dataset DPϕ. For each step in the trajectory, we obtain the next state st using the chosen dynamics model. The reward r is determined using a shared reward function ˆ r(s, a). We learn the policy using SAC [Haarnoja et al., 2018] on the combined dataset Doff ∪ DPϕ . We estimate the objective defined in Eq. (4) by rolling out trajectories with k steps rather than rolling out till the end. In our training process, we apply gradient updates to the dynamics models, aiming to minimize both Eq. (4) and the regularization term presented in Eq. (5). For a more comprehensive understanding of our algorithm's architecture and implementation, refer to the details provided in Appendix B.

5 Experiments

In this section, we focus on the following questions: (1) How is the performance of our method compared to that of previous approaches in the standard offline RL benchmark tasks, (2) To what extent is the performance improved when utilizing a limited number of dynamics models, as opposed to the scenario where the dynamics models are not re-trained, and (3) How the number of models and the rollout length influence the performance. We investigate these questions through extensive experiments on both the conventional D4RL [Fu et al., 2020] offline RL benchmark and the near-real-world NeoRL [Qin et al., 2022] benchmark within MuJoCo task.

5.1 Experiment Setup

In the initial phase of pre-training the dynamics models, we divide the offline dataset into a training set and a validation set using an 8:2 ratio. For each task, we construct a set of estimated models by training either 7 (for D4RL) or 15 (for NeoRL) models. After this training, we proceed to select the top 5 (for D4RL) or 10 (for NeoRL) models based on their predictive accuracy, which is evaluated on the validation set.

Baselines. We conduct a comparison of our algorithm against various state-of-the-art offline RL methods. These include model-free approaches as follows: (1) BC that simply mimics the policy that collected the dataset, (2) CQL [Kumar et al., 2020a] that uniformly penalizes Q-values for outof-distribution samples, (3) APE-V [Ghosh et al., 2022] that uses the Bayesian approach for decision-making within insupport region, and (4) EDAC [An et al., 2021] that diversifies Q-value ensembles to effectively estimate the expected minimum Q-value for policy learning. Additionally, we consider model-based approaches, which include (5) MOPO [Yu et al., 2020] that uses the uncertainty of the transition prediction as a penalty on reward function, (6) COMBO [Yu et al., 2021] that applies the penalty function of CQL within the model-based regime, (7) MAPLE [Lee et al., 2021] that also addresses decision-making in the out-of-support region by utilizing an ensemble of dynamics models, and (8) RAMBO [Rigter et al., 2022] that trains the policy through maximizing the expected value while training the dynamics model to minimize the expected value.

5.2 D4RL Benchmark

Datasets. Our evaluation spans across twelve datasets encompassing three distinct environments (halfcheetah, walker2d, hopper) and four data types (random, medium, medium-replay, medium-expert) for each environment. We used the 'v2' version of the datasets, and the results are summarized in Table 1. The reported scores are obtained from the 10 episodes of online evaluation at the last iteration.

Performance comparison. As a notable model-free baseline, EDAC uses either 10 or 50 critics to estimate the pessimistic Q-value. Our approach in D4RL relies on a mere 5 dynamics models to enhance the synthetic dataset for fitting the value function. This implies that when we have a restricted number of neural networks, employing them to enhance generalization can yield better results compared to utilizing them for estimating pessimistic values. This underscores the efficiency of our proposed method.

Model-based baselines also utilize 5 ensemble models, except MAPLE (n = 14). These models are employed to create pessimistic MDP, through estimating uncertainty or regularizing values for state-action pairs that are unseen during training. The results of our experiments convincingly demonstrate that our algorithm's heightened capacity for generalization can outperform or rival these model-based alternatives.

We also conducted a comparison with MAPLE that does not re-train the dynamics model during policy training. As we excluded the results of MAPLE (n = 5) due to the relatively lower performance, the detailed comparison results with MAPLE are provided in Appendix D.1. It is worth to note that both MoDAP and MAPLE initiate policy training using the same dynamics models across different seeds in case (n = 5). Here, MoDAP exhibits significant performance improvement compared to MAPLE (e.g., +54.3 on halfcheetahmedium-expert or +39.6 on halfcheetah-medium-replay). Interestingly, even though MAPLE uses a larger number of dynamics models (n = 14) as shown in Table 1, its performance does not surpass that of MoDAP, which uses only 5 models. This highlights the importance of the diverse composition of the set of dynamics models as a critical factor in learning adaptive policy for model-based offline RL.

5.3 NeoRL Benchmark

Datasets. NeoRL benchmark [Qin et al., 2022] is designed to replicate real-world scenarios. They have collected datasets through the more conservative policy, which aligns more closely with data-collection procedures encountered in real-world settings. The constrained nature of its narrower datasets hinders the high performance of existing offline RL algorithms. We focused on nine datasets that encompass three environments (HalfCheetah-v3, Hopper-v3, Walker-v3) and are categorized into three quality levels (L, M, H), representing low, medium, and high, respectively. Notably, NeoRL offers varying numbers of trajectories for training data (100, 1000, 10000) for each task. We have specifically experimented on 100 and 1000 trajectories settings, in order to nuance the limited support region covered by the datasets.

Performance comparison. In Table 8, we compared the performance of our algorithm with five baselines, including both model-free and model-based offline RL algorithms. To accommodate the characteristics of the NeoRL benchmark, MoDAP employs 10 dynamics models, in contrast to the

5 models utilized in the D4RL experiments. Further details about the baseline results and hyperparameter settings can be found in Appendix C.2. Our MoDAP method consistently showed robust and superior performance especially when compared to the deteriorated performance of EDAC, which performed remarkably well in D4RL benchmark. As the dataset support of NeoRL is significantly constrained, our diversity policy learning demonstrates the potential benefits of making decisions in out-of-support regions.

Furthermore, MoDAP showed substantial performance improvement with the larger number of trajectories (increasing from 100 to 1000). The augmented dataset size significantly contributes to constructing a set of better dynamics models. Meanwhile, the performance of the CQL method, which demonstrated high scores on several NeoRL datasets, showed relatively marginal improvement. This can be attributed to the inherent nature of NeoRL datasets, where the introduction of more conservative data does not necessarily translate to performance improvement.

We have observed that the performance of MoDAP was lower than that of BC, especially in the Hopper-v3-high dataset. This suggests that incorporating synthetic data sometimes leads to worse results than relying solely on the offline dataset if there is a significant discrepancy between the synthetic data and the true environment. We believe that increasing the number of dynamics models can handle this discrepancy issues, potentially leading to performance improvement. Specifically, in the Hopper-v3-high dataset, the normalized scores of MoDAP with n = 40 demonstrate a significant improvement: 37.1±0.8 for 100 trajectories (compared to 20.2 for n=10), and 52.4 ± 3.2 for 1000 trajectories (compared to 28.4 for n=10). These results clearly indicate that increasing the number of dynamics models leads to substantial improvements in performance.

5.4 Analysis on Hyperparameters

The number of models n employed plays a pivotal role in the performance of our algorithm, and rollout length k can impact our algorithm's behavior. To investigate these considerations, we conduct experiments to analyze the asymptotic performance while varying the number of model n and the rollout length k. This decision was driven by the understanding that utilizing a comparatively small number of training trajectories would enable a clear observation of the distinct impact of varying the number of models. Furthermore, the use of a medium-quality dataset is particularly effective in showcasing the algorithm's inherent capabilities. This is because other types of datasets could potentially yield performances that are either too low or too high, thus diminishing the significance of performance differences.

The results, illustrated in Figure 1, reveal several trends. Notably, for all values of n except when using 5 models, increasing the rollout length k leads to improved performance of the converged policies. However, in the case of 5 models, all tested rollout lengths show worse performance. This suggests that utilizing only 5 models is insufficient in the Walkerv3-medium-100 task.

Interestingly, when a rollout length of k = 5 is employed, increasing the number of models does not significantly improve performance. This observation implies that a rollout length of 5 may not be adequate to generate the necessary diversity across the dynamics models. However, as the number of models becomes sufficiently large, such as in the case of 20, 40 models, a rollout length of 40 exhibits rapid and superior performance. This outcome underscores the effectiveness of our algorithm in making decisions in out-of-support regions.

6 Conclusion

Offline RL has wrestled with the fundamental challenge of policy learning from limited datasets that capture only a fraction of the environment's dynamics. While constrained optimization has been a prevailing approach to cope with this challenge, it tends to suffer when encountering unexplored region during test time. To tackle this dilemma and enable better decision-making in previously unexplored regions, the advent of adaptive methodologies has been instrumental. These approaches have the capacity to train highly generalizable adaptive policy by incorporating a broad range of possible dynamics models as potential candidates for representing the true environment. However, the practical construction of an exhaustive set of dynamics models remains elusive, often resorting to ensemble structures to approximate this diversity. In this paper, we delve into the crucial question of which dynamics models should be embraced in training adaptive policies. To address this, we introduce a novel framework for assembling a set of dynamics models, driven by a mutualinformation-based objective. Our method is put to the test through comprehensive experiments conducted on both the D4RL and NeoRL benchmarks. The results demonstrate the superiority and competitiveness of our approach when compared to state-of-the-art algorithms, affirming its prowess in enhancing policy learning from limited data.

Furthermore, there exist several potential extensions to our algorithm, particularly within the context of Offline Meta RL [Pong et al., 2022; Lin et al., 2022]. In such scenarios, a limited number of tasks are sampled from the task distribution. To facilitate adaptation to new tasks, it would be advantageous to construct novel tasks that can interact with the policy, ultimately enhancing its generalization capability.

Acknowledgments

This work was supported by IITP grant funded by MSIT (No.2019-0-00075 Artificial Intelligence Graduate School Program (KAIST); No.2020-0-00940 Foundations of Safe Reinforcement Learning and Its Applications to Natural Language Processing; No.2022-0-00311 Development of GoalOriented Reinforcement Learning Techniques for ContactRich Robotic Manipulation of Everyday Objects).