Multi-policy Grounding and Ensemble Policy Learning for Transfer Learning with Dynamics Mismatch

1 Introduction

Reinforcement learning (RL) learns a policy of high return through interactions with the environment. Despite its successes in various domains, a critical limitation of RL algorithms is that a large number of transition samples are required to obtain an effective solution. Thus, transfer learning in RL seeks to minimize interactions with the target environment through source knowledge [Taylor and Stone, 2009]. In this work, we address the specific transfer learning problem in which source and target tasks are different only with respect to their dynamics.

In general, knowledge obtained from source tasks with less dynamics differences to the target task offers better opportunities for a successful transfer across tasks with different dynamics [Wang et al., 2020]. This has motivated a handful of methods modifying the dynamics of source task to reduce dynamics mismatch between source and target task [Hanna and Stone, 2017; Desai et al., 2020], which aim to effectively ground the source environment through an action transformation. A recent work [Desai et al., 2020] shows that the dynamics matching problem based on action transformation can be cast as an imitation from observation (IfO) problem. Based on this intuition, generative adversarial reinforced action transformation (GARAT) extends generative adversarial imitation learning (GAIL) to learn an action transformation policy that modifies dynamics of a source environment to minimize the discrepancy with respect to target dynamics. While GARAT can successfully reduce the dynamics mismatch between the source and the target environments, doing so is challenging with a limited number of samples, which especially being important when the target system is physical such as robot.

In this study, we propose to solve multiple IfO problems in a single grounding process for better grounding with much fewer samples. GARAT uses a single rollout policy to collect samples from the target environment. However, grounding with a single rollout policy can lead to a locally grounded environment resulting in dynamics matching only near trajectories of the rollout policy. We tackle this issue by learning an action transformation policy with multiple rollout policies, which we call multi-policy grounding (MPG). Through the use of multiple rollout policies, we expect the improvement of grounding quality.

Moreover, when learning a policy in grounded environments, we employ an ensemble method that uses multiple grounded environments. This is to avoid the possibility of obtaining a policy that works well only in a particular grounded environment. Hence, we learn a single agent policy that performs well across multiple grounded environments with a hope that the resulting policy generalizes well to the target task. The contribution of our work is three-fold as follows:

• We propose multi-policy grounding (MPG) algorithm to obtain generalizable action transformation policy with a small number of samples;
• We employ ensemble method to find more generalizable policy in the target task;

• We provide performance guarantee of the MPG.

Due to space limitations, we provide the supplementary material including appendices and our code at https://github.com/ hyunrok/MPGE.

2 RelatedWork

In transfer RL for dynamics mismatch, many studies assume multiple source policies, usually from a library of multiple source tasks, are given a priori and propose a method to effectively choose or combine these policies. Option learning [Yang et al., 2020b], policy function aggregation [Barekatain et al., 2019], policy shaping [Plisnier et al., 2019], and bayesian inference based mixture density network [Gimelfarb et al., 2020] are different methods proposed to fully utilize multiple source policies. Another stream of research assumes the existence of hidden parameters that define transition dynamics of environment [Killian et al., 2017; Perez et al., 2020; Yang et al., 2020a]. These studies exploit the advantage of hidden parameters that simplify the estimation of dynamics differences, hence facilitate transfer learning. Our work differs with these methods as we assume access to only one source task whose parameters may not be easy to modify or estimate (for example, a black-box simulator). Modification on tasks itself is alternative approach for ef-

fective transfer. Potential-based reward shaping has been used for modifying reward function at target task through source knowledge while preventing possibility of negative transfer [Brys et al., 2015; Gimelfarb et al., 2020]. On the other hand, recent work set additional penalties/incentives in reward functions of source task to consider dynamics differences while policy learning [Eysenbach et al., 2020].

Our work is more relevant to studies that modify the source dynamics. Policy robustness methods [Pinto et al., 2017; Jakobi, 1997] are such examples that perturb the source task dynamics to train robust policy that perform well across a range of different dynamics. Simulation grounding in sim-toreal setting is relevant in terms of enhancing transferability by reducing discrepancy of dynamics [Chebotar et al., 2019; Farchy et al., 2013; Zhang et al., 2020]. Especially, action transformation policy based approaches such as GAT [Hanna and Stone, 2017] and GARAT [Desai et al., 2020] possess the advantage of modifying dynamics without direct manipulation of simulator. Our study also takes advantage of the action transformation based simulation grounding approaches that work for black-box simulators and propose a modification on top of the GARAT framework. We propose the idea of using multiple policies to ground the simulator in addition to using an ensemble of multiple simulated source domains inspired by previous work that show improved sample efficiency when using ensembles in RL [Rajeswaran et al., 2017; Chua et al., 2018; Kurutach et al., 2018; Lee et al., 2021].

3 Preliminaries

3.1 MDP and Dynamics Mismatch Setting

We formulate each RL task as a Markov Decision Process (MDP) denoted by the tuple ⟨S,A,R, P, γ⟩ where S represents the state space; A represents the action space; R represents the scalar reward function S × A × S ↦→ R; P (s′ |s, a) represents the transition dynamics corresponding to taking action a in state s resulting in a next state s′ and γ ∈ [0, 1) represents the discount factor. A policy π ∈ Π is a mapping from S to a distribution over A that specifies the behavior of the RL agent. The objective of the agent is to find a policy that maximizes the expected return J(π) = Eπ,P [ ∑∞t=0 γ t r(st, at, st+1)] where s0 is initial state, at ∼ π(·|st) and st+1 ∼ P(·|st, at) for t ≥ 0. The objective can also be rewritten in terms of the marginal transition distribution ρπ as J(π) = 1 1 - γ ∑s,a,s′ ρπ(s, a, s′ )r(s, a, s′ ). ρπ is

where p(st = s|π, P) is the probability of being at state s at time t under dynamics P and π. The current scenario of transfer learning with dynamics mismatch assumes that only the transition dynamics P varies across the different tasks.

3.2 Action Transformation based Grounding

The objective of grounding is matching the dynamics of source environment to that of the target task. Learning an action transformation policy πg is useful for grounding when modifying the source dynamics Psrc directly is difficult due to its black-box nature [Hanna and Stone, 2017]. The policy πg is applied to the source dynamics Psrc as an additional transition so that Pg(s′ |s, a) = ∑˜a∈A Psrc(s′ |s, ˜a)πg(˜a|s, a). Action transformation based grounding seeks to learn a πg that minimizes the difference between grounded dynamics Pg and target task dynamics Ptrg .

Generative Adversarial Reinforced Action Transformation (GARAT) [Desai et al., 2020] learns action transformation policy by matching marginal transition distributions across the source and target tasks under a single rollout policy. This is done by defining an Imitation from Observation (IfO) problem on an augmented MDP for action transformation. State space of the augmented MDP is the joint state-action space, with elements x = (s, a), of the original MDP and the actions correspond to the transformed action ˜a. A policy function for this MDP is πg(˜a|s, a), and transition dynamics for the underlying rollout policy π is defined as P(s′ , a′ |s, a, ˜a) = Psrc(s′ |s, ˜a)π(a′ |s′ ).

Transition samples (s, a, s′ ) obtained with rollout policy π in target dynamics Ptrg can be interpreted as demonstration data for IfO problem of the augmented MDP given by an optimal action transformation policy π∗ g . Based on this fact, GARAT leverages recent approaches from the IfO literature [Torabi et al., 2018]. For a discriminative classifier D(s, a, s′ ) : S × A × S ↦→ (0, 1), that distinguishes if the observed transition (s, a, s′ ) is from the grounded (label 1) or target environment (label 0), the objective of GARAT is given by

where Eπ,P(·) denotes expectation with respect to transition samples (s, a, s′ ) collected by following π in dynamics P(·). The policy πg is learnt with reward function - log D(s, a, s′ ) in the augmented MDP problem.

4 Multi-policy Grounding and Ensemble Policy Learning

Grounding-and-then-Policy Learning (GPL) approach aims to learn a policy πa ∈ Π that will be optimal in target task in two steps: (1) grounding the source environment and (2) policy learning in the grounded environment. In the first step, action transformation policy πg is found as an optimal solution to the augmented MDP. This step uses a limited number of transition samples obtained with rollout policy at the target task. In the second step, optimal policy πa is learned in the grounded environment defined through πg without additional interactions with the target task.

We observed that the GPL suffered from two major issues: (1) grounding is done only locally around trajectories visited by the used single rollout policy, and (2) policy found to be optimal in the grounded environment catastrophically failed in the target environment. To address the issues, we extend the GPL in two ways: adoption of multiple policies as rollout policies (to address the first issue) and ensemble learning over multiple grounded environments (to address the second issue).

Figure 1 provides a schematic representation of the multipolicy grounding and ensemble policy learning method (MPGE). Note that grounding and policy learning are independent so various combinations should work such as multi-policy grounding and non-ensemble learning (MPG), single-policy grounding and non-ensemble learning (SPG), and alike.

Multi-policy grounding algorithm. The MPG algorithm extends GARAT [Desai et al., 2020] by adopting K rollout policies. While a single grounding policy πg is sought for, MPG considers the mixture of marginal transition distributions of K rollout policies. The objective function of MPG is defined by aggregating the performance of πg in K MDP problems using a common discriminator across multiple rollout policies as follows:

where D(s, a, s′ ) : S × A × S ↦→ (0, 1) distinguishes if the observed transition (s, a, s′ ) is from the grounded or target environment when K rollout policies has been deployed. More intuitive interpretations of Equation (3) as a grounding objective is provided in Appendix A.

In case there exists ¯πg that can perfectly match the dynamics across the source and target tasks, MPG finds ¯πg provided unlimited target samples are available.

Proposition 1. If there exists ¯πg ∈ Πg such that Pg(s′ |s, a) = Ptrg(s′ |s, a) ∀s, s′ ∈ S, a ∈ A, then JMPG is minimized by ¯πg.

Proposition 1 provides the performance guarantee that MPG will find an optimal grounding policy if exists. Proof is provided in Appendix B.

The MPG algorithm, given in Algorithm 1, finds a solution to K augmented MDPs defined by rollout policies {π1, ..., πK} using a reward function - log D(s, a, s′ ). While a single policy network for πg(˜a|s, a) is learned across K augmented MDPs, K value networks are updated for value functions V πg i (s, a) of ith of K augmented MDPs. Algorithm 1 is presented assuming TRPO [Schulman et al., 2015] as the RL algorithm for learning πg, but any other actor-critic RL algorithms will do.

Algorithm 1 Multi-policy Grounding (MPG)

Input: Set of rollout policies {π1, ..., πK}, demonstration buffers Bi,trg for i ∈ {1, ..,K}

• 1: Initialize action transformation policy πg(˜a|s, a), value network V πg i (s, a), and discriminators D(s, a, s′ ).
• 2: Uniformly sample index i ∈ {1, ...,K}, and initialize state s
• 3: for ⌊total time steps/batch size⌋ do
• 4: Initialize buffers for transition samples from grounded environments Bi,g = ∅ ∀i ∈ {1, ..., K}
• 5: for batch size do
• 6: if s = terminal state then
• 7: Uniformly sample index i ∈ {1, ...,K}, initialize state s
• 8: end if
• 9: a ∼ πi(a|s), s′ ∼ πi(a|s)πg(˜a|s, a)Psrc(s′ |s, ˜a)
• 10: Add (s, a, s′ ) to Bi,g
• 11: s ← s′
• 12: end for
• 13: Update D(s, a, s′ ) to maximize
• 14: ∑Ki=1 [EBi,g [logD(s, a, s′ )]
• 15: +EBi,trg [log(1 - D(s, a, s′ ))]]
• 16: Update πg(˜a|s, a) and V πg i (s, a) using TRPO rule based on reward - log D(s, a, s′ ) and samples in Bi,g
• 17: end for

Diversification of rollout policy. As discussed earlier, a near perfect grounding given a rollout policy suffers from overfitting. This is due to the tendency that grounding is done locally along the trajectories generated by the rollout policy. As a result, dramatic degradation of policy performance can occur whenever optimal policy deviates from the trajectories of the rollout policies. Therefore, we focus on diversification of rollout policies. A naive approach can be to generate rollout policies using multiple random seeds. The random rollout policies, which are individually optimal in the source domain, may not cover the environment globally and hence still prone to overfitting. Therefore, we propose a sequential generation of rollout policies for more controlled diversification using entropy as a regularizer. Specifically, we add the state-specific entropy of previously generated rollout policies when a new rollout policy is sought for. The entropy regularization term acting as a bonus for a state s given l alreadygenerated rollout policies is defined as 1l ∑li=1 H(πi(·|s)). This bonus term is motivated by an observation that distribution over actions from a state away from optimal trajectory tends to remain uniform (i.e., high entropy). It makes an intuitive sense to avoid state transitions sufficiently explored by previously generated rollout policies.

Ensemble policy learning. After grounding the source dynamics Psrc to Pg, we learn the agent policy πa in the grounded environment using a RL algorithm. However, because we are using limited number of target samples for grounding, errors in action transformation are inevitable. This could lead to poor performance of πa, optimal in the grounded environment, when it is actually used in the target task. To mitigate the performance degradation, we learn πa in multiple grounded environments, hence called ensemble policy learning. The ensemble approach effectively regularizes the policy learning in grounded environment by preventing the learned policy from overfitting to a particular grounded environment. To this end, we learn πg with N random seeds and same Bi,trg ∀i ∈ {1, ..,K} to form a collection of such policies: {πg,1, ..., πg,N}. During the policy learning, we uniformly sample an action transformation policy to define dynamics Pg at the beginning of each episode. That is, Pg(s′ |s, a) = ∑˜a∈A Psrc(s′ |s, ˜a)πg,n(˜a|s, a) for chosen πg,n ∈ {πg,1, ..., πg,N}. As a result, we learn πa whose performance is strong across N grounded environments. Now we have N as a hyper-parameter, and expect πa to be more robust in the target task. Note that MPG and MPGE will not require additional computation over SPG and MPG, respectively. This is because MPS uses the same number of samples as MPG but simply adopts multiple policies for sampling, and because MPGE carries out the same number of training iterations for policy optimization but the training is done on multiple environments.

5 Experiments

In this section, we compare the proposed methods (MPG and MPGE) against the existing one (SPG or equivalently GARAT). In particular, we seek to answer the following questions by transferring an agent policy between environments with different transition dynamics:

1. Can the multi-policy grounding method lead to a better matching of the dynamics?
2. How does multiple rollout policies affect the final performance at the target environment?
3. Does the ensemble approach in policy learning provide any benefits?

Transfer scenarios. We validate the performance of the proposed methods in continuous control environments provided by OpenAI gym [Brockman et al., 2016], namely InvertedPendulum, Hopper and HalfCheetah. For more interpretable results, we use transfer scenarios where the true action transformation is known. This is done by crippling one of the joints of the robots in target domain for Hopper and HalfCheetah to create the BrokenHopper and BrokenCheetah environments, respectively [Eysenbach et al., 2020]. The fully functional robots are used as the source tasks. Alternatively, for the InvertedPendulum environment, the impact of the action on the robot is modified in the target task such that there is an additional positive "force" ϵ acting on the robot. These three transfer learning scenarios provide meaningful experiments to analyze the quality of the action transformation learnt. We also include a task where the true action transformation is not known to address more challenging mismatch between the source and the target tasks. Similar to [Desai et al., 2020], the torso mass of the HalfCheetah is increased to create a HeavyCheetah for the target task. A full description of the tasks and transfer learning scenarios is provided in the Appendix C.

Experimental settings. We constructed two different types of rollout policy sets. Multi-seed (MS) set has 5 optimal policies for source task learned by the SAC algorithm with 5 random seeds. Entropy-regularized (ER) set has 5 source optimal policies that are generated sequential according to the diversification scheme explained earlier in Section 4. The entropy regularization term is added as 0.1 × 1l ∑li=1 H(πi(·|s)) for l previously generated rollout policies. Rollout policies are trained for 2 million time steps in InvertedPendulum and 3 million time steps in the other environments. We also trained 5 target optimal policies at the target task with the same setting for comparison. Then, we collected transition samples from the target environment using these rollout policies, and validated the performance of the proposed grounding methods using six target sample sizes: 500, 1000, 5000, 10000, 50000, and 100000. We varied the number of rollout policies K as 1 (SPG), 3, and 5 in MPG. Note that we keep the total number of samples used across all rollout policies under control. When the total number of samples is 1000, for example, we collected 200 samples per rollout policy in case of MPG using 5 rollout policies. We used 5 action transformation policies obtained from MPG to define 5 grounded environments to MPGE.

Implementation of MPG algorithm is based on 'stablebaselines' library [Hill et al., 2018] and codes released for GARAT [Desai et al., 2020]. We learned πg with TRPO algorithm over 5 million time steps in InvertedPendulum and 20 million time steps for the others, and rollout policy is uniformly selected at the start of each episode. While the action transformation policy is updated every 2048 time steps, we divided the sample batch by 512 and updated the discriminator 4 times with normalized inputs. The discriminator consists of two hidden layers of 256 units each, with tanh as activation function, and a gradient penalty term is used as a regularizer. When learning the agent policy in the grounded environment, the source optimal policies obtained from the original source task are used as the initial policies, and the agent policies are trained with the same settings as when obtaining the rollout policy. Most hyperparameters for RL algorithms refer to 'RL-baselines zoo' [Raffin, 2018], except for setting the discount factor to 0 while learning πg to obtain less dispersed action transformation. Detailed parameters are fully described in the Appendix C.

Analysis of action transformation error. We examine whether MPG matches the dynamics better than SPG in three environments (except HeavyCheetah). We compute action transformation error as the expectation of L2-norm of differences between learned action transformation ˜a and optimal action transformation ˜a∗ : E(s,a,˜a)∼π,πg,Psrc [∥˜a∗ - ˜a∥2]. Figure 2 shows how the action transformation error changes during grounding. For MPG, we have two results according to rollout policy set used for grounding. MPG-ER and MPG-MS denote MPG algorithm using entropy-regularized set and multi-seed set, respectively. We plot the average of the action transformation error for 2048 transition samples obtained with a random policy π in the grounded environment for every 100000 time steps. Each algorithm has been evaluated five times. Note that grounding is done by πg and the evaluation of grounding is done by π.

A larger number of rollout policies used during grounding should intuitively reduce the discrepancy of the marginal transition distributions between the grounded and the target environments. As expected, both MPG-ER and MPG-MS perform better than SPG over training steps using the same number of samples in terms of the average grounding error and the variance of grounding error as shown in Figure 2. On the other hand, more rollout policies would be less effective if they are not diversified enough, which is supported by an observation that the improvement is less when going fromMPG K = 3 to K = 5 than going from K = 1 (i.e., SPG) to K = 5.

The advantage of MPG can be due to that MPG prevents local grounding owing to multiple rollout policies. We provide a further analysis on issues in grounding done by SPG in Appendix D. MPG-ER, which controls diversification of rollout policies, seems to perform better in BrokenCheetah, but the difference between MPG-MS and MPG-ER is less obvious in general. This suggests that grounding with multiple rollout policies could be overall beneficial but the marginal improvement is decreasing as the number of rollout policies grows. In the following experiments, we used five rollout policies.

Benefits of ensemble learning. Ensemble learning tries to improve the quality of policy πa evaluated in the target task by optimizing πa across multiple grounded environments. This way, the resulting policy is more generalizable as evident in Figure 3, where MPGE outperforms not only SPG (thanks to multiple rollout policy grounding) but also MPG (thanks to ensemble). Figure 3 shows that the performance gap of the three algorithms between the grounded and the target environment. Note that MPG-ER with 500 target samples is used in Figure 3 and we obtained similar results from MPG-MS. Interesting to observe that the ensemble (MPGE) makes more significant improvement than multi-policy grounding (MPG) does over SPG.

Performance at target task. Figure 4 shows the performance of the policy πa, optimal in the grounded environment, in the target task as the sample size increases from 500 to 100000. Each algorithm has been evaluated 5 times with different random seeds, each rollout policy set for MPG-ER and MPG-MS contains 5 rollout policies, and 5 action transformation policies obtained from MPG to define 5 grounded environments to MPGE. We evaluated the performance of a policy with the average of returns in 50 episodes. We also normalized performance value by setting the performance when rollout policies are directly transferred to the target task as 0 (direct transfer) and the performance of policies learned in the target task as 1 (target optimal).

MPG-ER and MPG-MS generally outperform SPG with a few exceptions. This improvement can be due to more global grounding via multiple rollout policies. Performance of MPG without ensemble method (i.e., MPG-MS and MPGER) is decent as the target sample size varies frommere 500 to 100000. When ensemble is turned on, the performance improvement is quite noticeable. MPGE achieves more than 75% of the target optimal value with target transition sample size as few as 1000 and more with a single exception from BrokenHopper, in which MPGE shows at least 65% of the target optimal with sample size 1000 and above. In particular, for BrokenHopper and HeavyCheetah environments, MPGE was beneficial to find an agent policy that performs better on the target tasks compared to MPG. In HeavyCheetah, the ensemble methods outperform the optimal policy trained up to 3 million time steps in the target domain, which is why the normalized performance shown in Figure 4 is larger than 1.

6 Conclusions

In this work, we propose a transfer reinforcement learning method for dynamics mismatch setting by modifying the source dynamics through action transformation-based grounding. The existing methods have been found to suffer from dramatic degradation in target performance despite near perfect grounding, which motivated us to introduce multiple policy grounding (MPG). The target performance is further enhanced with ensemble policy training. Specifically, we re-write the objective function to allow multiple rollout policies yet a single discriminator to avoid overfitting. Ensemble method is to reduce the performance gap between the grounded and the target environments. Numerical experiments in four transfer scenarios based on continuous control task show that MPG algorithm and the ensemble method for policy learning could improve overall grounding quality and achieve better performance at target task.

While MPG and ensemble policy learning improves the stability of grounding algorithm and performance at target task, small errors in action transformation can still cause devastating performance collapse in more challenging scenarios. For more stable and effective grounding, we plan to extend the current approach in many directions. First, the MPG can be re-design so that it should reduce the discrepancy of trajectory distributions, rather than transition distribution, given our observation that globally grounded environment results in a better target performance. Second, more effective action transformation can be achieved based on the understanding of the dependency structure between state and action dimensions at the source task.

Acknowledgments

This research was supported by LG AI Research.