A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning

1 Introduction

Traditional online Reinforcement Learning (RL) is a promising path to obtain a near-optimal policy for complex systems within a general trial-and-error framework. However, this standalone mechanism faces numerous challenges [DulacArnold et al., 2021] when applied to real-world systems, especially in scenarios where interactions can be prohibitively expensive due to safety [Garcıa and Fern´andez, 2015; Achiam et al., 2017] or time [Weiss et al., 2016] considerations. In such context, one alternative is to leverage a cheap source environment, usually built as a simplification of the target environment. While these environments may also differ w.r.t. their observations [Gamrian and Goldberg, 2019] and rewards [Barreto et al., 2017], we study the off-dynamics setting where the mismatch is between their dynamics [H¨ofer et al., 2021]. It is a relevant consideration when it is possible to estimate or simplify the physics of the target environment, for instance within a simulator. We also consider that the transition probabilities of both the source and target environments are unavailable to reflect any kind of simulator.

The off-dynamics framework is particularly challenging: direct policy transfer from the source to the target environment usually fails [Muratore et al., 2019; Ju et al., 2022] due to compounding errors. The small discrepancies between the transition probabilities may accumulate over time, leading to increasing deviations between the trajectories of the source and target environments over time. Worse, modern optimization-based agents may exploit these discrepancies to find policies that perform exceptionally well in the source but result in trajectories that are impossible to replicate in the target.

In general, in addition to relying on a source environment, it is still possible to deploy the agent in the target system to collect data. However, this deployment is limited due to the mentioned safety and time considerations: the available data is this often limited to a few narrow trajectories. Two orthogonal approaches are possible to include this data in the derivation of the policy. The first one - well studied [Abbeel et al., 2006; Zhu et al., 2018; Desai et al., 2020; Hanna et al., 2021] - leverages this data to improve the source domain, then learns a traditional RL agent on the upgraded system. The second approach maintains the source environment fixed and biases the learning process to account for the dynamics discrepancies [Koos et al., 2012]. This second line of work is complementary to the first one, as both could be combined to make best use of the limited target samples. To the best of our knowledge, only a few works have taken this purely off-dynamics direction, and even fewer have focused on the low data regime scenario. Currently, the most prominent approach is DARC [Eysenbach et al., 2021] which modifies the reward function to search for parts of the source that behave similarly to the target system. Although this method is effective for a few classes of problems, such as the "broken" environments, we have found that it may fail in others, limiting its application to a restrictive class of discrepancies between the environments.

In this paper, we introduce the Few-shOt Off Dynamics (FOOD) algorithm, a conservative method that penalizes the derived policy to be around the trajectories observed in the target environment. We theoretically justify this method, which directs the policy towards feasible trajectories in the target system, and thus mitigates the potential trajectory shifts towards untrustworthy regions of the source system. Our regularization takes the form of a divergence between visitation distributions and can be practically implemented using stateof-the-art techniques from the Imitation Learning (IL) literature [Hussein et al., 2017]. Our method is validated on a set of environments with multiple off-dynamics disparities. We show that, compared to other baselines, our approach is the most successful in exploiting the few available data. Our agent is also shown to be relevant for a wider range of dynamic discrepancies.

2 RelatedWork

The off-dynamics setting has been studied in two distinct contexts, depending on the accessibility of the agent to transitions from the target environment, referred to as "zero-shot" and "few-shot" off-dynamics RL.

Zero-Shot Off-Dynamics RL. Sampling data from the target environment can be impossible due to strict safety constraints or time-consuming interactions. In such cases, the source environment is used to ensure robustness to guarantee a certain level of performance without sampling from the target system. It can take many forms. One possible choice is domain randomization [Mordatch et al., 2015] where relevant parts of the source system are randomized to make it resilient to changes. Another line of work focuses on addressing the worst-case scenarios under stochastic source dynamics [Abdullah et al., 2019]. Robustness can also be achieved w.r.t. actions [Jakobi et al., 1995; Tessler et al., 2019], that arise when certain controllers become unavailable in the target environment. These techniques are outside the scope of this paper as they do not involve any external data in the learning process.

Few-Shot Off-Dynamics RL. When data can be sampled from the target environment, two orthogonal approaches have been developed to propose efficient agents. The first approach, well established, is to improve the accuracy of the source environment. The parameters of the source physics can be optimized directly if available [Zhu et al., 2018; Tan et al., 2018]. Otherwise, expressive models can be introduced to improve the source dynamics model [Abbeel et al., 2006]. Within this category, a family of methods builds an action transformation mechanism that - when taken in the source system - produces the same transition that would have occurred in the target environment [Hanna et al., 2021]. In particular, GARAT [Desai et al., 2020] leverages recent advances in Imitation Learning from Observations [Torabi et al., 2019] to learn this action transformation and ground the source environment with only a few trajectories. All these algorithms are orthogonal to our work since once the source system has been improved, a new RL agent has to be trained.

The second approach, more closely aligned with our work, is the line of inquiry that modifies the learning process of the RL policy in the source to be efficient in the target environment. One group of approaches creates a policy - or policies - that can quickly adapt to a variety of dynamic conditions [Arndt et al., 2020; Yu et al., 2020;

Kumar et al., 2021]. It requires the ability to set the parameters of the source dynamics model which may not always be feasible, e.g., if the model is a black box. Amore general algorithm is DARC [Eysenbach et al., 2021]. It learns two classifiers to distinguish transitions between the source and target environments and incorporates them into the reward function to account for the dynamics shift. Learning the classifiers is easier than correcting the dynamics of the source environment, but as we will see in the experiments, this technique seems to work mainly when some regions of the source environment accurately model the target environment and others don't. Another related work is H2O [Niu et al., 2022b] which extends the approach by considering access to a fixed dataset of transitions from a target environment. It combines the regularization of the offline algorithm CQL [Kumar et al., 2020] with the classifiers proposed by DARC. However, the performance of H2O depends on the amount of data available. In fact, it performed similarly, or worse, to the pure offline algorithm when only a small amount of target data was available [Niu et al., 2022b, Appendix C.3].

3 Background

3.1 Preliminaries

Let ∆(·) be the set of all probability measures on (·). The agent-environment interaction is modeled as a Markov Decision Process (MDP) (S↪A↪ r↪ P↪ γ↪ ⇢0), with a state space S, an action space A, a transition kernel P : S ⇥ A ! ∆(S), a reward function r : S ⇥ A ⇥ S ! [Rmin↪Rmax], the initial state distribution ⇢0 and a discount factor γ 2 [0↪ 1). A policy ⇡ : S ! ∆(A) is a decision rule mapping a state over a distribution of actions. The value of a policy ⇡ is measured through the value function V ⇡ P (s) = E⇡↪P [P1t=0 γ t r(st↪ at↪ st+1)♣s0 = s]. The objective is to find the optimal policy maximizing the expected cumulative rewards J ⇡ P = E⇢0 [V ⇡ P (s)]. We also define the Q-value function Q⇡P (s↪ a) = E⇡↪P [P1t=0 γ t r(st↪ at↪ st+1)♣s0 = s↪ a0 = a] and the advantage value function A⇡P (s↪ a) = Q⇡P (s↪ a) - V ⇡ P (s). Finally, let d⇡P (s) = (1 - γ)E⇢0↪⇡↪P [P1t=0 γ tP(st = s)] the state visitation distribution, as well as its extension to state-action θ⇡P (s↪ a) and transition ⌫ ⇡ P (s↪ a↪ s0 ). All these quantities are expectations w.r.t. both the policy and the transition probabilities.

The off-dynamics setting involves two MDPs: the source systemMs and the target environment Mt. We hypothesize that they are identical except for their transition probabilities Ps 6= Pt. Our hypothesis states that while most of the MDP parameters are known, the underlying physics of the environment are only estimated. It encapsulates many real-world applications: a model of the dynamics may have been previously learned, or practitioners may have created a simulator based on a simplification of the system's physics. We do not assume access to any parameter modifying the transition probabilities Ps of the source physics to encompass blackbox simulators. For readability purposes, we drop the P subscripts for the value functions as they are always associated with the source environment Ms.

Many few-shot off-dynamics agents [Abbeel et al., 2006;

Desai et al., 2020; Hanna et al., 2021] typically employ the following procedure to handle complex environments. First, the policy and value functions are initialized in the source system. The choice of objective at this stage can vary, although a classical approach is to solely maximize the rewards of the source MDP. At each iteration, the policy is verified by experts. If it is deemed safe, N trajectories are gathered in the target environment and saved in a replay buffer Dt. These trajectories are then used to potentially correct the source dynamics and/or the training objective and induce a new policy. This process is repeated until a satisfactory policy is found. This setup is time-consuming and may be risky even when the policy is verified by experts, hence the need to learn with as few data as possible from the target environment.

This work focuses on how to best modify the objective with few trajectories. If handled properly, this could reduce the number of interactions required by the whole process overcoming the need to build a perfect source environment. For the purpose of our study, we assume that Mt remains fixed throughout the process.

3.2 Conservative Algorithms and the Visitation Distribution Constraint

Due to their efficiency and stability, conservative algorithms [Schulman et al., 2015; Schulman et al., 2017; Kumar et al., 2020] have shown strong efficiency in various RL settings. Many of them are based on [Kakade and Langford, 2002], where an iteration scheme improves the policy by maximizing a lower bound on the true objective. This process has been extended by refining the lower bound [Terpin et al., 2022; Moskovitz et al., 2021], for example by introducing a behavior b⇡P that encapsulates any additional property of the MDP [Pacchiano et al., 2020; Touati et al., 2020].

This family of algorithms is formalized as follows. A policy and a value function are parametrized with respective weights ✓ 2 ⇥ and ! 2 ⌦, that we denote from now on ⇡✓ and V ⇡✓ ! . At each iteration k, the policy is improved using the advantage function built from the approximated value function V ⇡✓k !k , with a penalization to respect the lower bound:

where D is any kind of similarity metric and ↵k is a hyperparameter, often set to a constant ↵. TRPO [Schulman et al., 2015] can be retrieved with b⇡P = ⇡, by setting D to be the Kullback-Leibler (KL) divergence and enforcing the penalization as a constraint with a step-size ✏k. Alternative behavior options can be found in [Pacchiano et al., 2020; Touati et al., 2020; Moskovitz et al., 2021]. In particular, [Touati et al., 2020] proposed to encapsulate the whole trajectories induced by ⇡ and P by setting b⇡P = d⇡P . It resulted in better results both in terms of sample efficiency and final cumulative rewards than most of its counterparts. This is natural as the new constraint between the state visitation distributions takes the whole trajectories induced by the policy into account, providing more information than the policy alone.

4 Few-Shot Off-Dynamics Reinforcement Learning

In this section, we propose a new objective to better transfer a policy learned in the source to the target environment. We extend the conservative objective to the off-dynamics setting. Then, we remind necessary results on Imitation Learning (IL) before deriving our practical algorithm Few-shOt Off Dynamics (FOOD) RL.

4.1 ANewConservative Off-Dynamics Objective

Given the discrepancies between the dynamics of the source and the target environment, applying the same policy to both environments may result in different trajectories. This poses a challenge as the agent may make the most of these differences to find policies that produce excellent trajectories in the source environment but are impossible to replicate in the target system.

We analyze the difference between the objectives J ⇡ P associated with the target and source environments, depending on a metric between visitation distributions. For this, we first apply directly the tools from traditional conservative methods [Schulman et al., 2015; Achiam et al., 2017] to the offdynamics setting, and propose the following lower bound using state visitation distributions.

Proposition 1. Let J ⇡ P = E⇢0 [V ⇡ P (s)] the expected cumulative rewards associated with policy ⇡, transitions P and initial state distribution ⇢0. For any policy ⇡ and any transition probabilities Pt and Ps, the following holds:

The Proposition also holds by replacing DTV(d⇡Ps ↪ d⇡Pt ) with DTV(θ⇡Ps ↪ θ⇡Pt ). We defer the proof to Appendix A. It illustrates how the performance of the optimal policy in the target environment may differ from that of the source due to two metrics. The first metric DTV(d⇡Ps ↪ d⇡Pt ) quantifies the difference between the visited states of the rollouts in the source and target environments. The second DTV(d⇡Ps ↪ d⇡Pt ) describes the difference between the transition probabilities associated with the visited states and the actions following the given policy. These terms must be controlled to allow a good transfer, especially given that they are exacerbated by the factor 2Rmax 1-γ . However, optimizing the second metric is difficult since the transition probabilities of both Mt and Ms are unknown. Hence, we propose the following simpler lower bound that considers transition visitation distributions instead.

with DTV the Total Variation distance and D⇡ TV (Ps↪ Pt) = Es⇠d⇡Ps (·)↪ a⇠⇡(·♣s) [DTV(Ps (·♣s↪ a) ↪ Pt (·♣s↪ a))].

Proposition 2. Let J ⇡ P = E⇢0 [V ⇡ P (s)] the expected cumulative rewards associated with policy ⇡, transitions P and initial state distribution ⇢0. For any policy ⇡ and any transition probabilities Pt and Ps, the following holds:

with DTV the Total Variation distance.

We also defer the proof to Appendix A. Here, the lower bound depends on the sole metric DTV(⌫ ⇡ Ps ↪ ⌫ ⇡ Pt ) that directly quantifies the difference in trajectories. As we will see, this term is easily minimized, especially given that the Total Variation distance could be replaced by other divergences. For instance, the Kullback-Leibler divergence or the JensenShannon divergence could be used thanks to Pinsker's inequality [Csiszar and K¨orner, 1981] or the one in [Corander et al., 2021, Proposition 3.2], provided the minimal assumptions of having a finite state-action space and the absolute continuity of the considered measures. Complete details can be found in Appendix A.

Overall, this lower bound highlights a good transfer between the source and target environment is possible when DTV(⌫ ⇡ Ps ↪ ⌫ ⇡ Pt ) is small, as the policy induces similar objectives J ⇡ P . Inspired by this insight, we adapt conservative methods to the off-dynamics setting and propose a new regularization between trajectories by setting the behaviors b⇡P to be the transition visitation distribution respectively associated with the transition probabilities of the source and target environment ⌫ ⇡ P :

The new penalization ensures that the policy is optimized for trajectories that are feasible in the target system, thus preventing the RL agent from exploiting any potential hacks that may exist in the source environment. In addition, remaining close to the data sampled from the target environment can be beneficial when the source system has been constructed using that data, as querying out-of-distribution data can yield poor results [Kang et al., 2022].

Unfortunately, the difference between the transition probabilities makes the regularization in Equation 4 difficult to compute. The previous work of [Touati et al., 2020] addressed this by restricting D to f-divergences

ing state-action visitation distributions. [Touati et al., 2020] used the DualDICE algorithm [Nachum et al., 2019] to directly estimate the relaxed ratio θ⇡✓ P θ⇡✓k P for any policy ⇡✓ sufficiently close to ⇡✓k , eliminating the need to sample data for each policy. However, this method is not applicable to our setting because DualDICE relies on a modified joined Bellman operator, which assumes that both distributions follow the same transition probabilities. Another solution would be to collect at least one trajectory per update. While this would not pose any safety concerns for the data would be sampled in the source system, it can be time-consuming in practice.

4.2 Practical Algorithm

In order to devise a practical algorithm for addressing Equation 4, we aim to get a surrogate objective for D⇣⌫ ⇡✓ Ps ∥∥∥ ⌫ ⇡✓k Pt ⌘. To construct such proxy, we leverage the recent results from Imitation Learning (IL) [Hussein et al., 2017] that we briefly recall in this paragraph. In this field, the agent aims to reproduce an expert policy ⇡e using limited data sampled by that expert in the same MDP with generic transition probabilities P. Most current algorithms tackle this problem by minimizing a certain similarity metric D between the learning policy's state-action visitation distribution θ⇡✓ P and the expert's θ⇡e P . The divergence minimization problem is transformed into a reward rimit maximization one, resulting in an imitation value function V ⇡ imit = E⇡↪Ps [P1t=0 γ t rimit(st↪ at↪ st+1)♣s0 = s]. Since these algorithms are based on data, they can be used to minimize the chosen similarity metric D between two transition visitation distributions with different transition probabilities. Applied to our setting where the target trajectories would be the expert ones, this is formalized as:

The choices for the divergence D are numerous, leading to different IL algorithms [Ho and Ermon, 2016; Fu et al., 2017; Dadashi et al., 2021], some of which are summarized in Table 1. For example, Equation 5 is exactly Theorem 1 in [Desai et al., 2020] in association with GAIL [Ho and Ermon, 2016], and is also straigthforward with PWIL [Dadashi et al., 2021].

DJS⇣X⇡✓

Pt ⌘ DKL⇣X⇡✓

Pt ⌘ DW⇣X⇡✓

Pt ⌘

Table 1: Objective function for well-known Imitation Learning (IL) algorithms. The variable X can be chosen as either d, θ, or ⌫. Other IL agents can be found in [Ghasemipour et al., 2020].

Ps ∥∥∥ X⇡✓k Ps ∥∥∥ X⇡✓k

These IL techniques enable efficient estimation of this value function using a small number of samples from ⌫ ⇡✓k Pt and unlimited access to Ms. Let ⇠ 2 ⌅ be the weights of this parametrized value function. The new regularization is A⇡✓k ↪⇠k imit (s↪ a), which can be learned with any suitable IL algorithm. It leads to the practical objective:

This new agent is quite generic as it could be optimized with different divergences. It takes as input an online RL algorithm [Babaeizadeh et al., 2016; Schulman et al., 2017] denoted O and an Imitation Learning algorithm denoted I. The whole off-dynamics algorithm process, which we denote Few-shOt Off Dynamics (FOOD) RL, is described as follows. First, the policy and the value weights are initialized in the source environment with O. At each iteration k, the agent samples N new trajectories with ⇡✓k 1 . Subsequently, the policy, traditional, and imitation value functions are retrained on the source environment with O and I according to Equation 4. The whole algorithm is summarized in Algorithm 1.

1 These trajectories could first be used to improve the source system.

Algorithm 1 Few-shOt Off Dynamics (FOOD).

Input: Algorithms O and I Initialize policy and value weights ✓0 and !0 with O Randomly initialize the weights ⇠0 for k 2 (0↪ ▷ ▷ ▷ ↪ K - 1) do Gather N trajectories ¶⌧i↪ ▷ ▷ ▷ ↪ ⌧N♦ with ⇡✓k on the target environment Mt and add them in Dt Learn the value function weights !k+1 with O in the source environment Ms Learn the imitation value function weights ⇠k+1 with I inMs using Dt Learn the policy maximizing (6) using Dt andMs with O end for

5 Experiments

In this section, we evaluate the performance of the FOOD algorithm in the off-dynamics setting in environments presenting different dynamics discrepancies, treated as black box simulators. The code can be found at https://github.com/ PaulDaoudi/FOOD.

The environments are based on Open AI Gym [Brockman et al., 2016] and the Minitaur environment [Coumans and Bai, 2016 2021] where the target environment has been modified by various mechanisms. These include gravity, friction, and mass modifications, as well as broken joint(s) systems for which DARC is known to perform well [Eysenbach et al., 2021, Section 6]. We also add the Low Fidelity Minitaur environment, highlighted in previous works [Desai et al., 2020; Yu et al., 2018] as a classical benchmark for evaluating agents in the off-dynamics setting. In this benchmark, the source environment has a linear torque-current relation for the actuator model, and the target environment - proposed by [Tan et al., 2018] - uses accurate non-linearities to model this relation.

All of FOOD experiments are carried out using both GAIL [Ho and Ermon, 2016], a state-of-the-art IL algorithm, as I. We find that GAIL performed similarly, or better than other IL algorithms such as AIRL [Fu et al., 2017] or PWIL [Dadashi et al., 2021]. FOOD is tested with its theoretically motivated metric between transition visitation distributions ⌫ ⇡ P , as well as with d⇡P and θ⇡P for empirically analyzing the performance associated with the different visitation distributions. The performance of our agent with the different IL algorithms can be found in Appendix C.5. We compare our approach against various baselines modifying the RL objective, detailed below. They cover current domain adaptation, robustness, or offline Reinforcement Learning techniques applicable to our setting. Further details of the experimental protocol can be found in Appendix C. Especially, a study with respect to the number of available target data is done in Appendix C.6.

• DARC [Eysenbach et al., 2021] is our main baseline. It is a state-of-the-art off-dynamics algorithm that introduces an importance sampling term in the reward function to cope with the dynamics shift. In practice, this term is computed using two classifiers that distinguish transitions from the source and the target environment.
• In this agent, an important hyper-parameter is the standard deviation σDARC of the centered Gaussian noise injected into the training data to stabilize the classifiers [Eysenbach et al., 2021, Figure 7]. We draw inspiration from the open-source code [Niu et al., 2022a].
• Action Noise Envelope (ANE) [Jakobi et al., 1995] is a robust algorithm that adds a centered Gaussian noise with standard deviation σANE to the agent's actions during training. Although simple, this method outperformed other robustness approaches in recent benchmarks [Desai et al., 2020] when the source environment is a black box.
• CQL [Kumar et al., 2020] is a purely offline RL algorithm that learns a policy using target data. It does not leverage the source system in its learning process and thereby serves as a lower bound to beat. This algorithm inserts a regularization into the Q-value functions, with a strength β. We use [Geng, 2021] to run the experiments.
• H2O [Niu et al., 2022b] is another off-dynamics algorithm that leverages data from the target system. It combines the classifiers from DARC to the CQL regularization. Similarly to FOOD, the agent is encentisized to stay close to the target data, but does so with a combination of DARC and CQL regularization. We also use [Niu et al., 2022a] to run these experiments. However, this method performed poorly, which was expected considering it was proposed for a setting where a large amount of target data is available. Hence, its results are omitted from Table 2 and deferred in Appendix C.9.
• We also consider two RL agents, RLs trained solely on the source system (without access to any target data) and RLt trained solely on the target environment. Both algorithms were trained to convergence. Even though the latter baselines do not fit in the off-dynamics setting they give a rough idea of how online RL algorithms would perform in the target environment. The online RL algorithm O depends on the environment: we use A2C [Babaeizadeh et al., 2016] for Gravity Pendulum and PPO [Schulman et al., 2017] for the other environments.

Experimental Protocol. Our proposed model and corresponding off-dynamics/offline baselines require a batch of target data. To provide such a batch of data, we first train a policy and a value function of the considered RL agent until convergence by maximizing the reward on the source environment. After this initialization phase, 5 trajectories are sampled from the target environment to fit the restricted target data regime. They correspond to 500 data points for Pendulum and 1000 data points for the other environments. If some trajectories perform poorly in the target environment, we remove them for FOOD, DARC, and CQL to avoid having a misguided regularization. FOOD, DARC, and ANE are trained for 5000 epochs in the source environment and were optimized with the same underlying agent. Both RLs and RLt are trained until convergence. CQL is trained for 100000 gradient updates for Gravity Pendulum and 500000 gradient updates for all other environments. Additional details can be found in Appendix C.

Hyper-parameters Optimization. We optimize the hyperparameters of the evaluated algorithms through a grid search for each different environment. Concerning DARC and ANE, we perform a grid search over their main hyper-parameter σDARC 2 ¶0▷0↪ 0▷1↪ 0▷5↪ 1♦ and σANE 2 ¶0▷1↪ 0▷2↪ 0▷3↪ 0▷5♦. The remaining hyper-parameters were set to their default values according to the experiments reported in the open-source code [Niu et al., 2022a]. For CQL, we perform a grid search over the regularization strength β 2 ¶5↪ 10♦, otherwise we keep the original hyper-parameters of [Geng, 2021]. For RLt and RLs we used the default parameters specific to each environment according to [Kostrikov, 2018] and trained them over 4 different seeds. We then selected the seed with the best performance in the source environment. For our proposed algorithm FOOD, the regularization strength hyper-parameter ↵ is selected over a grid search depending on the underlying RL agent, ↵ 2 ¶0↪ 1↪ 5↪ 10♦ for A2C and ↵ 2 ¶0▷5↪ 1↪ 2↪ 5♦ for PPO. This difference in choice is explained by the fact that the advantages are normalized in PPO, giving a more refined control over the regularization weight.

Results. We monitor the evolution of the agents' performance by evaluating their average return in the target environment during training. We do not gather nor use the data from those evaluations in the learning process since we work under a few-shot framework. The return of all methods is computed and averaged over 4 seeds for CQL, RLs, RLt and ANE, and 8 seeds for FOOD and DARC. For clarity, the standard deviation in Figure 1 is divided by 2 for readable purposes. In all figures, the x-axis represents the number of epochs where each epoch updates the policy and value functions with 8 different trajectories from the source environment.

5.1 Comparison Between the Different Agents

We evaluate the mentioned algorithms on the proposed environments. These experiments provide an overview of the efficiency of the different objectives in finetuning the policy, given reasonably good trajectories. Results are summarized in Table 2, where we also report the median of the normalized average return (NAR) J ⇡agent Pt -J ⇡RLs Pt J ⇡RLt Pt -J ⇡RLs Pt [Desai et al., 2020] as well as the median of the NAR's standard deviations. The associated learning curves can be found in Appendix C.2.

All the experiments demonstrate the insufficiency of training traditional RL agents solely on the source environment. The optimal policy for the source is far from optimal for the target system as we observe a large drop in performance from RLt to RLs on all benchmarked environments. For example, the RLs exploits the linear torque-current relation in Low Fidelity Minitaur and fails to learn a near-optimal policy for the target environment. Furthermore, RLs often exhibits a large variance in target environments as it encounters previously unseen situations. This is welcome as relevant trajectories are gathered to guide the agent in the source environment.

Overall, we can see that our algorithm FOOD exhibits the best performances across all considered environments against all other baselines, whether it is constrained by state, stateaction or transition visitation distributions. Two exceptions are on Heavy and Friction Cheetah where ANE has very good results. We also note that FOOD associated with its regularization ⌫ ⇡ P or θ⇡P has better results than when it is associated with d⇡P . This is expected for ⌫ ⇡ P as it optimizes the whole lower bound, and it seems using θ⇡P implicitely mimizes the second term of Proposition 1.

In addition, we find that the prominent baseline DARC is not efficient in all the use cases. It seems to be particularly good at handling sharp dynamics discrepancies, e.g., when one or two joints are broken or when friction is introduced but struggles for more subtle differences. In fact, it deteriorates over the naive baseline RLs by a large margin for the Gravity Pendulum and the Low Fidelity Minitaur environments. This may be explained by their reward modification ∆r (see Appendix B.1) which prevents the agent from entering dissimilar parts of the source environment but seems unable to handle systems with a slight global dynamics mismatch. Even when DARC improves over RLs, our algorithm FOOD is able to match or exceed its performance. The robust agent ANE is a strong baseline in most environments but may degrade the performance of traditional RL agents, as seen in Low Fidelity Minitaur, Broken Joint Ant, and Gravity Pendulum. CQL did not provide any good results, except on Gravity Pendulum and Low Fidelity Minitaur, but this was to be expected given the few target trajectories the agent has access to. Finally, we note that the three agents FOOD, DARC, and ANE often reduce the variance originally presented in RLs.

We attribute FOOD's success to its ability to force the agent to improve the rewards of the source environment along trajectories from the target environment. Its regularization seems to be efficient even in the low data regime.

5.2 Hyper-parameter Sensitivity Analysis

We previously reported the results of the best hyperparameters of the different methods. In practice, it is important to have a robust range of hyper-parameters in which the considered method performs well. Indeed, to the best of our knowledge, there currently exists no accurate algorithm for selecting such hyper-parameters in a high dimensional environment when the agent has access to limited data gathered with a different policy [Fu et al., 2020]. In this section, we detail the sensitivity of FOOD associated with PPO and GAIL to its hyper-parameter ↵ in 3 environments. They were specifically chosen to illustrate the relevant range for ↵. FOOD's complete hyper-parameter sensitivity analysis, as well as the one of DARC and ANE, can respectively be found in Appendix C.4, Appendix C.7 and Appendix C.8. They attest that selecting the right hyper-parameters for the baselines DARC and ANE does not have an intuitive pattern.

The hyper-parameter ↵ controls the strength of the regularization in FOOD. If it is too low, the agent will focus mainly on maximizing the rewards of the source environment and becomes very close to the naive baseline RLs. This can be seen in Broken Joint Cheetah. On the other hand, setting ↵ to a high value may induce the agent to solely replicate the trajectories from the target environment, in which case the agent may also be close to RLs. Even worse, a hard regularization may degrade over RLs, as shown in Low Fidelity Minitaur for ↵ = 5. However, 5 is an extremely high value as advantages are normalized in PPO, and this may increase the gradient too much and corrupt learning.

In any case, we have found that FOOD provides the best results when the regularization has the same scale as the traditional objective. This is also verified for the environments not displayed in this sub-section. We conclude that FOOD is relatively robust to this range of hyper-parameter, and recommend using PPO with ↵ close to 1 as the underlying RL agent. It is a natural choice given that PPO normalizes its advantage functions.

6 Conclusion

In this work, we investigated different objectives to optimize a policy in different few-shot off-dynamics scenarios, including the state-of-the-art method DARC. We found that these objectives are either too simplistic or unable to cope with complex dynamics discrepancies, thereby limiting their application to real-world systems. To address this challenge, based on theoretical insights, we introduced a novel conservative objective along with a practical algorithm leveraging imitation learning techniques. Through experimentations in different off-dynamics use cases, we have shown that our approach often outperforms the existing methods and seems to be more robust to dynamics changes.

Our agent could also benefit from new advances in the Imitation Learning literature to gain control in building its penalization. Finally, this penalization can be useful when the source environment has been improved using the available target trajectories as it avoids querying the source environment for Out-of-Distribution samples. This will be the primary focus of our future work.

Contribution Statement

Paul Daoudi wrote the main parts of the article and conducted the experiments. Merwan Barlier and Ludovic Dos Santos participated in the development of the main ideas and extensively revised the article. Christophe Prieur and Bogdan Robu oversaw the project and provided additional revisions to the article.