Exploring Parameter Space with Structured Noise for Meta-Reinforcement Learning

1 Introduction

Reinforcement Learning (RL) has achieved great success in various applications. In RL, the agent improves its future rewards by exploring insufficiently understood states and actions so that preventing premature convergence. Prior works have proposed a wealth of strategies, e.g., exploration by injecting random noise in the action space [Sutton and Barto, 2018], following task-agnostic criteria such as state visitation counts [Bellemare et al., 2016; Kearns and Singh, 2002; Tang et al., 2017], information gain [Houthooft et al., 2016] and curiosity [Sun et al., 2011; Pathak et al., 2017], and exploration by sampling and ensemble of models [Osband et al., 2016]. These techniques are task agnostic and do not contain any prior information on the way to better explore the task.

∗This work was primarily done during the author's internship at Tencent Robotics X.

†Contact Author: Jie Shao.

In practice, agents are often supposed to be able to handle distinct but related tasks. This leads to meta-RL. To utilize previous learning experiences. Model Agnostic Meta Learning (MAML) [Finn et al., 2017] propose to learn a metapolicy which contains common structures of various tasks. The agent can then adapt agilely to a new task from the learned meta-policy. Meta learning decouples the common structures and task-specific information of the tasks so that it can be utilized to acquire efficient and guided exploration for a specific task. Gupta et al. [2018] proposed Model Agnostic Exploration with Structured Noise (MAESN), which learns a task-specific latent variable z. z ∼ N(µ, σ2 ) containing the task-specific knowledge, is then concatenated to the states and used as structured prior noise for efficient and guided exploration. MAESN obtains an impressive exploration behavior in sparse reward tasks. However, computing dataset level features is always difficult and MAESN does not yield a better result than the original MAML in dense reward cases.

Inspired by MAESN, we propose a novel Exploration with Structured Noise in Parameter Space (ESNPS) approach by injecting structured noise directly in the policy parameter space. Different from MAESN which uses the task-specific latent variable as noise, we show that meta-policies trained on different partitions of meta-train tasks, containing diverse prior knowledge, can be used as additive structured noise up to a scaling factor. Besides the bonuses of prior guided exploration, training with parameter noise in high-dimensional neural network parameter space [Plappert et al., 2018; Fortunato et al., 2018] helps to escape local optima [Jin et al., 2017] in complex non-convex problems. To ensure consistency in actions and temporally coherent stochasticity, the policy network is perturbed only at the beginning of an episode.

Traditional parameter space noise injection techniques, however, are not directly applicable for meta-RL. In Plappert et al. [2018] and Fortunato et al. [2018], policy parameters are perturbed and updated with reparameterization trick [Kingma and Welling, 2014]. This optimization requires substantial gradient steps to avoid unpredictable high variance, but Meta-RL enjoys fast adaptation and converges in a few gradient steps on a new task. Instead of optimizing the original policy parameters, we directly adapt the perturbed policy parameters. Among the fine-tuned models perturbed by different structured noise containing different prior knowledge, the best performing one is picked.

Extensive experiments are conducted on cheetah velocity, cheetah direction, ant velocity and ant direction. The results show that our proposed ESNPS effectively explores testing tasks, outperforming a set of competitive baselines. The main contributions of our work are as follows:

• We propose ESNPS, which effectively utilizes previous search experiences as structured noise for directed efficient exploration in a new task. Specifically, to the best of our knowledge, this is the first time to use the meta parameters of policy networks as parameter space structured noise.
• We consider the high variance problem caused by injecting noise in parameter space in meta-RL and propose a new optimization strategy suitable for fast adaptation in meta-learning.
• Extensive experiments demonstrate the effectiveness of the constructed structured noise in directed exploration for a specific task as well as the superiority of our proposed method against a set of competitive baselines.

2 RelatedWork

A crucial problem of reinforcement learning is how to explore effectively, and prior works have made great efforts for it. State visitation methods [Kearns and Singh, 2002; Brafman and Tennenholtz, 2001] are based on optimism in the face of uncertainty, which is yet constrained by small state-action space. The concept of curiosity [Sun et al., 2011] is beneficial to the exploration. The information gain [Houthooft et al., 2016] and parameter space exploration [Plappert et al., 2018; Fortunato et al., 2018] are also used for exploration strategies. Additionally, Plappert et al. [2018] and Fortunato et al. [2018] use noise to perturb the policy network in parameter space. This ensures consistency in actions and temporally coherent stochasticity. However, these exploration strategies are largely task-agnostic, in that they aim to explore without exploiting the particular structure of the tasks. This approach using the samples from the perturbed policy to update the non-perturbed network is not suitable for fast adaptive metalearning method. In this paper, we investigate whether previous learning experiences can be leveraged to guide exploration of the current new task. This is achieved by incorporating the idea of meta-learning.

Meta-learning is also known as learning-to-learn [Mitchell and Thrun, 1992; Vilalta and Drissi, 2002; Lemke et al., 2015], where a model extracts common structures of related but distinguished tasks so that a similar and previously unseen task can be efficiently solved with the prior knowledge. Various meta-learning methods have been proposed [Hochreiter et al., 2001; Vinyals et al., 2016; Mishra et al., 2017] and the most relevant is the Model-Agnostic MetaLearning (MAML) [Finn et al., 2017]. MAML and its extensions [Finn et al., 2018; Yoon et al., 2018; Li et al., 2017;

Grant et al., 2018; Nichol et al., 2018] learn a set of metaweights as initialization from where a few gradient steps yield good performance on a previously unseen task. Metalearning decouples the common structures and task-specific information of the tasks so it can naturally be utilized to acquire efficient and guided exploration for a specific task. Closer to ours is MAESN [Gupta et al., 2018], which utilizes MAML and additionally learns a latent variable z ∼ N(µ, σ2 ), containing the task-specific knowledge as a structured noise for exploration. MAESN obtains an impressive exploration behavior in sparse reward tasks. However, in dense reward cases, MAESN does not yield better results than vanilla MAML. Compared with MAESN, Our ESNPS does not explicitly compute the task representation z, which is always difficult. Besides, as a parameter space exploration method, ESNPS enjoys the bonus of training with noised parameters, which helps to escape local optima.

3 Preliminaries

Suppose that a number of tasks {T } are sampled from a task distribution p(T ). Each task Ti is a Markov Decision Process (MDP) and is denoted as Ti = {S, A, Pi, Ri, γ, ρi}, where S is the state space, A is the action space, Pi : pi (st+1|st, at) is a transition probability distribution, Ri : S×A → R is the reward function, ρi the initial state distribution and γ ∈ (0, 1) is the discount factor. Policy parametrized by θ is denoted as πθ. For each task Ti, we collect a trajectory τij = {st, at, rt}H-1 t=0 where H is the actual trajectory horizon for the task, and the trajectory discounted return of the task R(τij) = ∑H-1 t=0 γrt. Now, the likelihood of a trajectory induced by policy πθ is given by p(τij |θ) = ρi (s0) ∏H-1 t=0 pi (st+1|st, at)πθ (at|st). Meta-RL considers how to utilize previous learning experiences for efficient search on a new task. To find a procedure that can generate a good policy to adapt to the new task Ti though a few gradient steps, Finn et al. [2017] proposed Model Agnostic Meta-Learning (MAML), which maintains a meta-policy πθ, from where a few gradient descent steps can lead to a significant increase of performance on previously unseen tasks Ti. The fast task adaptation is performed by:

where β is the adaptation learning rate of policy and R is the trajectory discounted reward. The meta policy is updated by solving the following problem:

The model learns the meta-policy πθ as good initialization for adaptation when Equation (2) converges. By solving Equation (2) and getting the meta-policy, the agent can reach high reward with limited examples on a previously unseen task.

4 Our Method

To better explore a new task in the meta-RL setting, we propose Exploration with Structured Noise in Parameter Space

(ESNPS), whose framework is presented in Figure 1. ESNPS uses meta-policies trained on different partitions of metatrain, containing diverse prior knowledge, as additive structured noise for better exploration of a new task in the metatesting phase. Among the perturbed models, after fine-tuning, the best performing one is picked. In the following, we first analyze how to apply parameter space noise without causing high variance problem in meta-RL setting. Then, we detail ESNPS in how to construct effective structured noise with meta-policies trained on different partitions of train tasks and discuss the intuition behind it. Finally, we introduce an adaptive scaling module to automatically determine the noise scaling factor for better performance.

4.1 Parameter Space Noise for Meta-RL

Injecting noise in high-dimensional neural network parameter space helps to escape local optima [Jin et al., 2017]. Without loss of generality, we use Gaussian noise to perturb the parameters of deep neural networks for effective exploration. For our policy gradient RL case, parameter noise can be incorporated in gradient steps [R¨uckstieß et al., 2008]. Given a policy πθ(a|s) parameterized by θ ∼ N(µ,Σ), with reparametrization trick [Kingma and Welling, 2014], the gradient can be:

where we consider N samples ϵi ∼ N(0, I), and τi ∼ (πµ+ϵΣ 1 2 , p), where p is the transition probability. Reparameterization trick decouples the mean, variance value and the stochasticity so that the original non-perturbed policy parameters can be updated with gradient steps. With this re-parameterization trick Plappert et al. [2018] show impressive exploration behavior on both high-dimensional discrete action tasks and continuous control tasks.

However, directly applying Gaussian noise and updating with re-parametrization trick as in Plappert et al. [2018] and Fortunato et al. [2018] requires lots of gradient steps to accurately estimate the unperturbed policy parameters. While in the meta-RL case, we focus on fast adaptation on new tasks and only a few steps are allowed. Instead of optimizing the original un-perturbed policy parameters during adaptation, we treat the perturbed as a variable to be optimized and directly fine-tune the perturbed models. The perturbed policy parameters are now ˜ θ = θ+ϵiΣ 12 and are updated as follows:

where β is the gradient step size, and we collect M trajectories to estimate the policy gradient. Altogether we sample N samples ϵi ∼ N(0, I) to conduct N perturbed policies and fine-tune these policies according to Equation (5). Then, the best performing model is picked from the N perturbed networks.

4.2 Exploration with Structured Noise

For RL problems, compared with task-agnostic random noise, structured noise containing prior knowledge of the underlying task can help the learner better explore the insufficiently understood states. This has been shown in MAESN [Gupta et al., 2018], which learns a task-specific latent variable z. z ∼ N(µ, Σ) containing the task-specific knowledge, is then concatenated to the states and used as structured prior noise for efficient and guided exploration. This also holds for parameter space noise. Hu et al. [2019] show that the parameters need to elaborate noise with prior knowledge rather than a task-agnostic random noise. In this part, we detail our Exploration with Structured Noise in Parameter Space (ESNPS) algorithm and apply the perturbed policy optimization strategy developed in Section 4.1.

In ESNPS, we partition meta-train tasks into J partitions T = ⋃Jj=1 Sj and conduct meta-training on each partition separately, resulting in altogether J + 1 meta-policies {πθj }Jj=1 (including one base model trained on the whole meta-train set). The intuition behind this is that the existing meta-learning algorithms would force the model to learn common structure (prior knowledge) from the training set [Finn et al., 2018], but do not adequately model specific structures of different tasks [Lan et al., 2019]. By partitioning the tasks and meta-train separately, meta-polices {πθj } contain diverse prior knowledge. When used as structured noise, they potentially provide more task-specific information, and can be much more helpful for directed exploration on a new testing task than plain spherical noise. Specifically, we use the network meta-policy πθ0 as a base model θ(1) 0 , and other meta-polices {πθj } as structured noise {θ(1) j }. After adapted policy with Equation 1 on a specific task, we obtain fine-tuned models θ(2) 0 , θ(2) j respectively. Then, we apply the following noise injection scheme:

where ˜ θi is the perturbed model, and α is a scaling factor which will be discussed later. Then, as discussed in the previous part, we directly fine-tune the perturbed neural network ˜ θi with the help of structured noises for randomized but taskaware exploration:˜

where β is the gradient step size. Besides the bonus of directed exploration based on the information contained in the structured noise, perturbing by parameter noise equals to changing the parameter position of policy parameter in the high-dimensional parameter space, and such a perturbation may help to escape the local minimum or saddle point in complex none-convex optimization problems [Jin et al., 2017].

Like ordinary meta-learning algorithms, our ESNPS algorithm has two phases: meta-training and meta-testing. The architecture of ESNPS is depicted in Figure 1, and the algorithm is summarized in Algorithm 1. In the meta-training phase, we train J +1 MAML models on J +1 partitions {Si} of the task set T . In the meta-testing phase, we first fine-tune

Algorithm 1 ESNPS algorithm

• Require: J + 1 MAML models {θ(1) 0 , θ(1) 1 , θ(1) 2 , θ(1) 3 , · · · , θ(1) J } trained from different partitions Si of task set T , where T = ⋃Jj=1 Si and θ0 trained on the whole task set S0T .
• 1: Initialize fine-tuned model collection C = ∅
• 2: Initialize perturbed model collection E = ∅
• 3: for all MAML model θ ∈ {θ(1) 0 , θ(1) 1 , θ(1) 2 , θ(1) 3 , · · · , θ(1) J } do
• 4: Fine-tune θ with k gradient steps and put it to finetuned model collection C
• 5: end for
• 6: Set θ(2) 0 as the base model
• 7: for j = 1 to J do
• 8: Perturb the base model θ(2) 0 by structured noise θ(2) i to generate perturbed model, put the perturbed model ˜ θi into perturbed model collection E.
• 9: end for
• 10: for all model θ ∈ E do
• 11: Fine-tune θ with k gradient steps, put it into perturbed model collection E
• 12: end for
• 13: return The best performing perturbed model ˜ θ from the perturbed model collection E

all the models with k gradient steps and put them into the Fine-tuned model collection. The number of gradient steps k for fine-tuning is always set to 4. We set model θ0 trained on the whole training set S0 = T as the base model, and other J models as structured noises. Then, we iteratively perturb the base model θ0 by using structured noises to generate diverse perturbed models. The parameters of perturbed models are computed by Equation 6. After further fine-tuning, we take the best performing perturbed model as a return. To the best of our knowledge, this is the first study to use the parameters of a learned neural network as structured noise.

4.3 Noise Scaling

The noise level plays an important role in RL exploration. High-level noise encourages trials on insufficiently understood areas but potentially incurs high variance in the agent training. Noise level has been previously discussed in Plappert et al. [2018], where task-agnostic Gaussian noise is considered. In this section, we introduce a heuristic scheme that adaptively tunes the scaling factor α in our directed exploration ESNPS. Compared with perturbation in action space, injecting noise in parameter space is harder to depict. Let π and ˜πj denote the original policy and the policy perturbed by noise model j. Following Plappert et al. [2018], we measure the distance between the two policies by the variance in action space they induce:

where we average on M trajectories to estimate the distance between the two policies. As there are altogether J noise models, the averaged distance d(π, ˜π) = 1 J ∑J d(π, ˜πj) is used to measure the averaged noise level. The scaling factor α then adaptively increases or decreases depending on whether the distance is below or above a certain threshold:

where λ ∈ R+ is used to rescale α, which is set to 1.1 in our experiments. δ is a threshold controlling the acceptable change of actions due to noise injection.

With Equation (9), we change a policy parameter space searching problem into an action space searching problem, which can be much more easier to solve. In meta-RL, to get a certain level of exploration or perturbation, the proper value of scaling factor α can change rapidly for different tasks. However, as the meta-test tasks are often quite similar to each other, a certain level of exploration naturally requires a similar level of change in actions for the tasks, which means a shared threshold δ is enough for all the tasks.

This scaling factor searching scheme resembles the noise level adaptation strategy proposed in Plappert et al. [2018] in the formulation, but is for a different purpose and is carried out differently. We apply Equation (9) at the beginning of the policy gradient update of a specific task to get a proper α for further guided exploration. The scaling factor is kept unchanged during the latter exploration. In contrast, Plappert et al. [2018] focuses on a single task and dynamically varies the noise level during the whole exploration process.

5 Experiments

We evaluate the proposed ESNPS on four reinforcement learning tasks with MuJoCo simulator [Todorov et al., 2012]. To show the effectiveness of guided search with structured noise, we compare ESNPS with vanilla MAML and a competitive parameter space task-agnostic noise model proposed in Plappert et al. [2018]. We omit the comparison with MAESN, which learns a task-specific latent variable for directed task-specific search because MAESN does not yield better performance in dense reward task.

5.1 Task Setup

Meta-RL tasks. To evaluate the proposed ESNPS algorithm, we experiment on four tasks: cheetah velocity, cheetah direction, ant velocity and ant direction. For all experiments, we use a neural network policy with two hidden layers of size 100, and ReLU nonlinearities. For the velocity tasks, the reward is the negative absolute value between the velocity given by the agent and the goal velocity. For direction tasks, the reward is the difference between the current direction and the goal direction. The goals of cheetah and ant are sampled uniformly from 0.0 to 2.0 and from 0.0 to 3.0, respectively. The horizon is set to H = 200, with 20 rollouts per gradient step for all groups of tasks except the ant direction task. The ant direction uses 40 rollouts. For each group of tasks, we construct 40 new tasks as meta-test set. For all the experiments, we exactly follow the protocol proposed in Finn et al. [2017].

Hierarchical partition. To acquire structured noise containing diverse prior knowledge, we hierarchically partition the meta-train tasks into J splits. T = ⋃Jj=1 Sj. As metalearning models exact common structures shared across metatrain tasks, these partitions, used as meta-train sets, can provide different levels of abstraction and diverse prior knowledge. Specifically, S0 = T , and we bi-partition all the metatrain tasks into S1 and S2 according to the goal value of the tasks. Then, we equally partition the meta-train set into 4 splits forming {S}6j=3 and finally 8 splits forming {S}14 j=7 . For instance, S1 of cheetah velocity contains tasks with goals from U[0.0, 1.0) and S2 with goals from U[1.0, 2.0], where U is the uniform distribution. Similarly, tasks with goals from U[0.0, 0.5), U[0.5, 1.0), U[1.0, 1.5), U[1.5, 2.0] form {S}6j=3 respectively. Conducting meta-learning on {S}Jj=0 will result in meta-policies {πθj }Jj=0 . We set model πθ0 as baseline MAML and perturb on policy parameters. In all our experiments, the fast adaptation Equation (1) is computed using vanilla policy gradient, and the meta-training is carried out with Trust Region Policy Optimization (TRPO) [Schulman et al., 2015].

5.2 Performance Comparison

We get altogether 15 models after the hierarchical partition and meta-training, including a base model θ0 trained on the whole meta-train set. Parameters of the other 14 metapolicies are used as structured noises to perturb the base model. In this section, we compare ESNPS with vanilla MAML and a parameter space Gaussian noise method [Plappert et al., 2018] (denoted by PSNE). For a fair comparison, in the Gaussian noise case, we follow a similar noise injection and fine-tuning protocol: 14 random noises ϵi ∼ N(0, σ2 ) are used to perturb the base model θ0, generating 14 perturbed models and the one with the best reward after fine-tuning is picked. Both structured noises (ESNPS) and Gaussian noises (PSNE) perturb the neural network in the parameter space. The perturbed policies are then updated as in Section 4.1. To get better performance, we adapt the scaling factor α of ESNPS with Equation (9) and the shared threshold δ controlling the change in action is set to be 1. For the Gaussian noise baseline PSNE, we set the noise level to σ = 0.01. This is a manually tuned noise level for PSNE. The properness of this value will be shown in the ablation study discussing the influence of noise level on final performance.

From Figure 2, we see that both ESNPS and PSNE improve the performance on vanilla MAML, validating our hypothesis that fine-tuning with parameter noise in high-dimensional neural network parameter space helps to escape local optima thus performing better exploration. While exploring with task-agnostic Gaussian noises only improves marginally, perturbing with the structured noise achieves significant improvement in all the four groups of tasks. Besides, we see that during the final fine-tuning, injecting Gaussian noise introduces large fluctuations while structured noise leads to much more stable training. We own this to the informative directed task-aware exploration in ESNPS.

In this experiment, every fine-tuning operation needs k = 4 gradient steps. Therefore, both the ESNPS and PSNE need

15 ∗ 4 + 14 ∗ 4 = 116 gradient steps, including the 4 steps of pre-adaptation for {θ(2) j }Jj=0 and the 4 steps of final adaptation of the perturbed model from {˜ θj}Jj=1 . From Figure 2 we see that the final fine-tuning steps on the perturbed ˜ θj do not always yield further performance gain, so that these fine-tuning steps are not always necessary. In this way, we can simplify ESNPS by running only the 15 ∗ 4 = 60 preadaptation gradient steps. From Figure 2 we see that even the simplified ESNPS (corresponding to the step 0 reward) always significantly outperforms PSNE, no matter whether the final fine-tuning steps are carried out or not. Compared with the baseline vanilla MAML, ESNPS obtains a significant improvement. With this notable achievement, we believe it is worthwhile even at the cost of more gradient steps.

5.3 Effect of Noise Level

Finally, we discuss how different noise levels will affect the exploration performance with the tasks of cheetah direction and ant direction. The other two groups of tasks are much simpler and can easily achieve high performance. We test the performance of ESNPS with different scaling factors α ranging from 0.01 to 1.0 (for all tasks, α is shared). Similarly, PSNE is evaluated with different variances σ ranging from 0.005 to 0.1. The results are shown in Figure 3. From Figure 3 we see that both ESNPS and PSNE are sensitive to the noise level and thus careful tuning of scaling factor α or variance σ is critical. In most cases, perturbing with structured noise shows far better exploration behavior than with Gaussian noise.

One interesting phenomenon is that when relatively highlevel noise (e.g., α = 0.1) is applied, ESNPS still performs comparably with vanilla MAML, indicating that the perturbed model still maintains the common structure of metatrain tasks. On the contrary, high-level noise introduces destructive perturbation when task-agnostic noise is applied, which validates the importance of task-specific guidance in exploration. Finally, note that with noise level σ = 0.01, the learner reaches the best performance in cheetah direction and explores comparably well in ant direction among the testing values, validating our setting of σ = 0.01 as the default variance level in Gaussian noise models in Section 4.2.

6 Conclusion and FutureWork

In this paper, we introduce ESNPS to utilize previous search experiences as structured noise for directed exploration in new tasks. Experiments show the effectiveness of using diverse meta-policies as structured noise for exploration. Besides, we also demonstrate that directly fine-tuning the perturbed model is a suitable method for fast adaptive metalearning methods like MAML. Although our method is widely applicable, it has limitations. For instance, constructing diverse structured noise is time-consuming. For future work, we plan to construct more easy-to-get yet powerful structured noises.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 61832001 and No. 61672133) and Sichuan Science and Technology Program (No. 2019YFG0535 and No. 2018GZDZX0032).