Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemble

1 Introduction

Compared with simple simulation tasks, it is more difficult for RL algorithms to succeed in real-world applications. First, the observation contains much noise and the sampling cost is more expensive in real-world applications. Second, the environment shifts between the training and evaluation due to the complexity of the real world. For instance, in financial trading, the noise in the imperfect market information puts forward high requirements on the sample efficiency of the algorithm, and the volatile market requires the algorithms not to overfit to the training environment and retain the ability to generalize to unseen states during evaluation. However, typical RL algorithms cannot achieve satisfactory performance in these applications. Motivated by the superior performance of ensemble methods on improving the accuracy and generalization ability in SL especially for small datasets, we resort to ensemble methods to fulfill the aforementioned requirements. In our work, we focus on policy ensemble, which is an integration of a set of sub-policies, instead of value function ensemble for some reasons listed below. i) Value-based methods perform worse than policy-based methods in noisy applications like MOBA games [Ye et al., 2020], card games [Yang et al., 2022; Li et al., 2020] and financial trading [Fang et al., 2021]. ii) Previous ensemble techniques for RL are mainly applied to SL components, like environment dynamics modeling [Kurutach et al., 2018] and value function approximation [Anschel et al., 2017]. iii) Policy learning in RL algorithms is critical which is more different from SL. But it is not well studied and thus is worth exploring.

∗The work was conducted during Zhengyu Yang's internship at Microsoft Research. The corresponding author is Kan Ren.

Notice that in many real-world applications such as those mentioned above, Proximal Policy Optimization (PPO) [Schulman et al., 2017] is always the first choice of

performances. Therefore, in order to derive an applicable RL algorithm, in this paper, we take PPO as our backbone, and propose a simple yet effective policy ensemble method named Ensemble Proximal Policy Optimization (EPPO). EPPO rigorously treats ensemble policy learning as a first class problem to explicitly address: i) what is the reasonable yet effective policy ensemble strategy in deep RL and ii) how it helps to improve the performance of policy learning.

Some existing works for policy ensemble aim to attain a diverse set of policies through individually training various policies and simply aggregating them ex post factor [Wiering and Van Hasselt, 2008; Duell and Udluft, 2013; Saphal et al., 2020], which has few guarantees to improve the overall performance due to the neglection of the cooperation among different sub-policies. The other works incorporate the divideand-conquer principle to divide the state space, and derive a set of diverse sub-policies accordingly [Ghosh et al., 2017; Goyal et al., 2019; Ren et al., 2021]. But the difficulty in dividing the state space and the unawareness of the sub-policy on the whole state space may significantly hurt the performance especially in deep RL. Furthermore, whether and how the ensemble method would benefit policy optimization still remain unsolved and require additional attention.

In contrast, EPPO resolves the ensemble policy learning from two aspects. On one side, we argue that ensemble learning and policy learning should be considered as a whole organic system to promote the cooperation among the subpolicies and guarantee the ensemble performance. Thus, EPPO combines sub-policy training and decision aggregation as a whole and optimizes them under a unified ensembleaware loss. To fully exploit the data and improve sample efficiency, sub-policies are also optimized with the data collected by the ensemble policy which aggregates all the cotraining sub-policies for final decision. Furthermore, we theoretically prove that the decision aggregation of co-training sub-policies helps in efficient exploration thus improves the sample efficiency. On the other side, considering that ensemble methods benefit from the diversity among sub-policies and it is difficult to divide the state space to train diverse policies reasonably, EPPO incorporates a diversity enhancement regularization within the policy space to guarantee the diversity and further improve the ensemble performance. We empirically found that it can improve policy generalization in real-world applications because the diversity enhancement regularization prevents the sub-policies from collapsing into a singular mode or over-fitting to the training environment, which retains the ability of the ensemble policy to generalize to unseen states.

In a nutshell, the main contributions of the work are threefold:

• We propose a simple yet effective ensemble strategy in ensemble policy learning and prove that aggregating the cotraining sub-policies can promote policy exploration and improve sample efficiency.
• To the best of our knowledge, EPPO is the first work that adopts the diversity enhancement regularization to attain a diverse set of policies for policy ensemble.
• Demonstrated by the experiments on grid-world environments, Atari benchmarks and a real-world application, EPPO yields better sample efficiency and the diversity enhancement regularization also provides a promising improvement on policy generalization.

2 Background

2.1 Preliminaries

Sequential decision making process can be formulated as a Markov decision process (MDP), represented by a tuple M = ⟨S,A, p, p0, r⟩. S = {s} is the space of the environment states. A = {a} is the action space of the agent. p(st+1|st, at) : S × A ↦→ Ω(S) is the dynamics model, where Ω(S) is the set of distributions over S. The initial state s0 of the environment follows the distribution p0 : S ↦→ R. r(s, a) : S × A ↦→ R is the reward function. The objective is learning a policy π to maximize the cumulative reward η(π) = Eτ∼π [∑Tt=0 r(st, at)] where τ is the trajectory sampled by π. In this paper, we consider discrete control tasks where A is limited and discrete.

2.2 RelatedWorks

Ensemble in RL The recent works applying ensemble methods in RL are mainly focusing on environment dynamics modeling and value function approximation. For environment dynamics modeling, several environment models are used to reduce the model variance [Chua et al., 2018] and stabilize the model-based policy learning [Kurutach et al., 2018]. As for value function approximation, Q-function ensemble is popular in alleviating the over-estimation [Anschel et al., 2017], encouraging exploration [Lee et al., 2021] and realizing conservative policy learning in offline reinforcement learning [Wu et al., 2021].

Nevertheless, the mechanism behind environment dynamics modeling and value function approximation is similar to SL and it has a huge gap with policy learning in RL. Among the existing works on ensemble policy learning, some works follow the technique used in SL and simply aggregate individually trained policies ex post facto. To generate a set of diverse sub-policies, different weight initialization [Faußer and Schwenker, 2015; Duell and Udluft, 2013], training epochs [Saphal et al., 2020], or RL algorithms [Wiering and Van Hasselt, 2008] are used. Compared with SL tasks, RL agents must take a sequence of decisions instead of making a one-step prediction, which makes the cooperation among sub-policies more important to attain a good ensemble. Without considering policy ensemble and policy learning as a whole optimization problem, the cooperation among sub-policies can be neglected, so that these methods have few guarantees to improve the overall performance. The other works incorporate the divide-and-conquer principle to divide the state space and derive a set of specialized policies accordingly, and then these policies are aggregated to solve the original task, which coincides with the idea of mixtureof-experts (MOE) [Jacobs et al., 1991]. The essential of the MOE is how to deliver data and obtain a set of specialized policies (i.e. experts) which focus on different regions of the state space. EPPO, which proposes a new method (i.e. diversity enhancement regularization) to derive an ensemble of experts and shows better performance in experiments, is also a special case under the paradigm of MOE. To divide the state space, DnC [Ghosh et al., 2017] heuristically divides the whole task into several sub-tasks based on the clustering of the initial states while ComEns [Goyal et al., 2019] and PMOE [Ren et al., 2021] learn a division principle based on information theory and Gaussian mixture model respectively. However, it is hard to divide the state space in many environments and the unreasonable division can damage the performance. Moreover, the unawareness of the whole state space of the sub-policies caused by the division may significantly hurt the performance especially in deep RL scenarios and result in poor ensemble effectiveness.

Diversity Enhancement Diversity enhancement, which aims at deriving a set of diverse policies, is mainly used in population-based RL. To enhance the diversity, KL divergence [Hong et al., 2018], maximum mean discrepancy [Masood and Doshi-Velez, 2019] and determinantal point process [Parker-Holder et al., 2020] are adopted as bonus during training process. In policy ensemble, the diversity among sub-policies is also important, but almost all the existing methods apply state space division to enhance the diversity, which may hurt the performance as mentioned before. Thus, we impose the diversity enhancement regularization on the policy space to guarantee the diversity among the subpolicies.

3 Ensemble Proximal Policy Learning

In this section, we first motivate our design in EPPO on the training of sub-policies and data collection, and then introduce the details of the learning method. The overview of the architecture is illustrated in Figure 1.

3.1 Policy Ensemble

As widely used in the literature of ensemble methods [Zhou et al., 2002; Anschel et al., 2017], the approximated function is aggregated by a set of base components, each of which can be optimized and work individually in the target task. Thereafter, we consider maintaining K sub-policies {πθ1 , πθ2 , . . . , πθK }. For brevity, we denote πk to represent the sub-policy parameterized by θk. Then, the ensemble policy ˆπ can be derived through mean aggregation over the subpolicies. Formally, for a given state s, the ensemble policy ˆπ(·|s) is calculated as the arithmetic mean of the sub-policies:

Note that the parameters of the ensemble policy ˆπ are exactly the whole set of parameters of sub-policies {θk}Kk=1 . In RL tasks, sample efficiency is a key problem and we would expect that the performance improvement of the designed ensemble method comes from the algorithm itself instead of the more trajectories sampled by more sub-policies. Thus, only the ensemble policy ˆπ is allowed for data collection in EPPO and the agent samples the action a ∼ ˆπ(·|s) from the ensemble policy ˆπ when interacting with the environment. Then the trajectories sampled by the ensemble policy ˆπ will be further used for updating the sub-policies.

3.2 Policy Optimization

The previous ensemble works in SL also motivate a fact that better sub-models lead to better empirical results [Zhang et al., 2020], thus we apply PPO to maximize the expected return (i.e., η(πk), 1 ≤ k ≤ K) of the sub-policies and the loss is defined as:

where ˆπ′ is the ensemble policy, µ is an adaptive penalty parameter to constrain the size of the policy update, and ˆ Aˆπ′(t) = ˆ Aˆπ′(st, at) is the advantage function estimated by a generalized advantage estimator (GAE) [Schulman et al., 2015] which describes how much better the action is than others on average. It is worth noting the data used for optimization is collected by the ensemble policy ˆπ, and we only optimize the parameters of sub-policies through policy gradient, which does not impose any additional sample cost in the environment as that of a single policy.

Though simply aggregating the predictions of the submodels in the ensemble has shown effectiveness in improving the performance for supervised tasks [Zhou et al., 2018], there is less evidence showing that the aggregation of the subpolicies can improve decision making, because of the large gap between the essential of RL and SL, such as: i) there may be more than one optimal action at the current state in RL while there is only one ground truth for SL; ii) the aggregation of some well-behaved sub-policies may derive undesirable action distribution and lead to bad states, especially when there are many different ways to handle the task and the subpolicies are separately optimized. Thus, it is necessary to take the cooperation among sub-policies into consideration and optimize them in a consistent learning paradigm. In EPPO, we incorporate an ensemble-aware loss to encourage the cooperation among the sub-policies and ensure a well-behaved ensemble policy ˆπ. The definition of ensemble-aware loss which optimizes the ensemble policy ˆπ by PPO is

Taking Eq. (1) into Eq. (3), we can update the ensemble policy by updating all sub-policies in a unified behavior under the same target. To some extent, the ensemble-aware loss serves as a regularization that may promote the ensemble performance at the cost of the performance of sub-policies.

However, there is a potential risk of mode collapse in policy ensemble that all the sub-policies converge to a single policy, which makes policy ensemble useless. In our method, the problem is even worse because all the sub-policies share a similar training paradigm, which makes these sub-policies tend to behave similarly. Moreover, EPPO randomly chooses a sub-policy for action sampling at each step (Eq.(1)), so the diversity among sub-policies should promote the exploration of the ensemble policy. To this end, we propose a diversity enhancement regularization to prevent all sub-policies from collapsing into a singular mode and ensure diverse sub-policies to further improve the ensemble performance. Intuitively, in order to enhance the diversity, the regularization should encourage the action distributions proposed by different subpolicies to be orthogonal with each other. Specifically, for discrete action space, the diversity enhancement regularization adopted in EPPO is defined as

We note that there are many optional metrics to encourage the diversity among sub-policies such as KL divergence [Hong et al., 2018] and MMD [Masood and Doshi-Velez, 2019] in RL literature, yet we adopt Eq.(4) in EPPO due to its computationally efficiency [Li et al., 2012].

In conclusion, the overall loss to minimize is defined as

where α and β are the hyper-parameters.

3.3 Theoretical Analysis

Theorem 1 (Mean aggregation encourages exploration). Suppose π and {πi}1≤i≤K are sampled from P(π), then the entropy of the ensemble policy ˆπ is no less than the entropy of the single policy in expectation, i.e., Eπ1,π2,...,πK [H(ˆπ)] ≥ Eπ [H(π)].

The proof can be found in Appendix A in the supplemental materials. Theorem 1 illustrates that the ensemble policy ˆπ enjoys more effective exploration than the single policy in the policy learning procedure. Thus, aggregating the sub-policies during training can improve the sample efficiency. We have also observed the corresponding phenomenon in the experiments, which reflects the effectiveness of the mean aggregation operation for policy ensemble in our method.

4 Experiments

In our paper, we only consider discrete control tasks as it is commonly adopted in real-world scenario applications and the continuous control tasks can be discretized for the ease of optimization. To evaluate the performance of EPPO, we conduct experiments on Minigrid [Chevalier-Boisvert et al., 2018], Atari games [Bellemare et al., 2013] and financial trading [Fang et al., 2021], which span the simulated tasks and real-world applications.

The experiments and the analysis in this section are led by the following two research questions (RQs). RQ1: Does our method achieve higher sample efficiency through policy ensemble? RQ2: Is the generalization performance of our method better than the other compared methods?

4.1 ComparedMethods

We compare EPPO with the following baselines including two variants of EPPO.

• PPO [Schulman et al., 2017] is a state-of-the-art policy optimization method which has been widely used in realworld applications [Fang et al., 2021; Ye et al., 2020].
• PE (Policy Ensemble) is based on a traditional policy ensemble method [Duell and Udluft, 2013] which trains K policies individually by PPO and then aggregates them. In our paper, we consider two aggregation operation, i.e., majority voting and mean aggregation, which are defined as ˆπ(a|s) = 1K ∑Kk=1 I((argmaxa′ πk(a′ |s)) == a) and Eq. (1), respectively. We denote these two methods as PEMV and PEMA for short.
• DnC [Ghosh et al., 2017] partitions the initial state space into K slices, and optimizes a set of policies each on different slices. During training, these policies are periodically distilled into a center policy that is used for evaluation.
• ComEns [Goyal et al., 2019] uses an information-theoretic mechanism to decompose the policy into an ensemble of primitives and each primitive can decide for themselves whether they should act in current state.
• PMOE [Ren et al., 2021] applies a routing function to aggregate the sub-policies and deliver the data to different sub-policies during optimization.
• SEERL [Saphal et al., 2020] uses a learning rate schedule to get multiple policies in a round and selects a set of policies for ensemble according to performance and diversity.
• EPPO is our proposed method described above, which has two other variants for ablation study: EPPO-Div is the method without diversity enhancement regularization defined in Eq. (4) and EPPO-Ens is the method without ensemble-aware loss defined in Eq. (3).

Since we focus on policy ensemble, we omit Q-function ensemble methods like Sunrise [Lee et al., 2021]. In all compared baselines, we take PPO as the base policy optimization method for fairness; besides, they have roughly the same number of parameters (i.e., K times the parameter size of PPO and we set K = 4 as default for all experiments) and the same number of samples collected in one training epoch.

4.2 Improved Efficiency onMinigrid

We first investigate whether EPPO can improve the sample efficiency. In this part, we consider two partial-observable environments with sparse reward in Minigrid [Chevalier-Boisvert et al., 2018]: Distributional-Shift and Multi-Room, as shown in Figure 2, where the agent aims at reaching the given target position and a nonzero reward is provided only if the target position is reached. Specifically, the place of the second line of the lava in Distributional-Shift is reset, and the shape of Multi-Room is regenerated during the reset procedure. Due to the more complex structure and longer distance between the start position and the goal, Multi-Room is more difficult.

As shown in Figure 2, EPPO enjoys the best sample efficiency in both environments (RQ1). We notice that PEMA and PEMV fail (i.e., return = 0) in both environments while PPO can get better performance. And we conclude the failure from two perspectives. First, considering the number of samples needed for PPO to get a positive reward is large, PEMA and PEMV may need K times samples for a positive reward because they individually train K PPO policies. Second, due to the asynchrony among the sub-policies of PE method, the useful knowledge can be overwhelmed thus neglected due to the aggregation operation by the worthless knowledge. Their failure implies the necessity of an ensemble-aware loss and explains the rationality of sampling with ensemble policy ˆπ. For SEERL, its final performance is even worse than that at the last epoch because the ex-post selection cannot ensure an improved performance, which further emphasizes the necessity of the joint optimization of the sub-policies. Moreover, the failure of DnC, ComEns and PMOE in Multi-Room environment can be attributed to the division operation on the state space that may not only unreasonably divide the space but also hinder the ability on exploration to the full environment. And comparing the performance of the methods based on state space division and EPPO, we find that EPPO consistently achieves better performance, which implies that a diversity enhancement regularization is a better choice for diversity enhancement in policy ensemble.

The ablation study in Figure 2 shows that EPPO outperforms its two variants, thus both of diversity enhancement regularization and ensemble-aware loss appear to be crucial for the superior performance of EPPO. In addition, the returns of both EPPO and its variants improve quickly at first, which confirms the result in Theorem 1 that mean aggregation encourages exploration. In the last column of Figure 2, we analyze the effect of using various number of sub-policies in EPPO. The results indicate that an extremely small K cannot lead to a good performance and the performance cannot be further improved by increasing K by a large margin. When K is a small value like 2, the sub-policies tend to have fewer overlaps in the action space, thus the mean aggregation operation is unable to extract the valuable information from subpolicies and leads to worse performance.

4.3 Comparative Evaluations on Atari Games

Having seen the superior performance of EPPO in environments with sparse reward, we also want to evaluate EPPO on more difficult and widely used benchmarks. Particularly, we follow the settings in [Saphal et al., 2020] and choose four environments from Atari games as the testbed. As shown in Table 1, EPPO can still consistently achieve the best performance in 10M environment steps, suggesting a better sample efficiency (RQ1).

4.4 Generalizable Application: A Financial Trading Instance

To evaluate the generalization ability, we conduct experiments on order execution [Fang et al., 2021] which is a fundamental yet challenging problem in financial trading. In order execution, the environments are built upon the historical transaction data, and the agent aims at fulfilling a trading order which specifies the date, stock id and the amount of stock needed to be bought or sold. In particular, the environments are usually formulated as training, validation and test phases each of which is corresponding to a specific time range. To be specific, training environment and validation environment are used for policy optimization and policy selection respectively. Test environment is unavailable during training. Due to the shift of macroeconomic regulation or other factors in different time, test environment may differ a lot to the training and validation environment. Thus, the selected policy has to make decisions in unfamiliar states during testing and the performance in test environment is a good surrogate evaluation of the generalization ability.

Following [Fang et al., 2021], the reward is composed of price advantage (PA) and market impact penalty where PA encourages the policy to get better profit than a baseline strategy. Specifically, we take TWAP as the baseline strategy, which equally splits the order into T pieces and evenly executes the same amount of shares at each timestep during the whole time horizon. And an increase of 1.0 in PA can bring about a 0.5% annual return with 20% daily turnover rate. The averaged PA and reward of the orders in the test environment are taken as evaluation metrics in order execution. We conduct experiments on the two large datasets 1801-1908 and 1807-2002 published in [Fang et al., 2021] and the statistics of the datasets can be found in Table 2.

The results of different methods are reported in Table 3. As expected, EPPO achieves the best performance in both PA and reward in two datasets, suggesting that our proposed method has great potential in generalizing to unseen states (RQ2). And we find that PEMV has a worse reward than PPO in 1801-1908, which implies individually training sub-policies has no guarantee on the ensemble performance, thus an ensemble-aware loss, i.e., Le in Eq. (3) which can encourage the coordination among sub-policies, is necessary. In addition, the performance degradation of EPPO-Ens also illustrates the importance of the ensemble-aware loss. Moreover, from the comparison between EPPO and EPPO-Div, we find that the diversity enhancement regularization further improves the generalization performance. This phenomenon coincides with the observations in SL [Zhou et al., 2002] that the diversity among sub-models can reduce the variance, alleviate the over-fitting problem and improve the generalization performance of the ensemble method.

To evaluate the effect of K, we conduct experiments when K ∈ {2, 4, 8} in dataset 1801-1908 and the results are shown in Table 4. Similar to the experimental results in Minigrid, K = 2 leads to a poor performance, which can still be attributed to the difficulty of getting consensus during aggregation when K is small. In addition, a larger K does not always lead to better performance.

4.5 Analysing the Policy Diversity

After showing the performance of diversity enhancement regularization in improving the sample efficiency and policy generalization, we further verify that the regularization does diversify the sub-policies. Motivated by [Hong et al., 2018], we utilize the action disagreement (AD) to measure the diversity among sub-policies, which is defined as

where M is a set of states. In dataset 1801-1908, the AD values of EPPO and EPPO-Div are 15.9% and 14.3%, respectively, which demonstrates the ability of diversity enhancement regularization in improving diversity.

5 Conclusion and FutureWork

In this paper, we focus on ensemble policy learning and propose an end-to-end ensemble policy optimization framework called EPPO that combines sub-policy training and policy ensemble as a whole. In particular, EPPO updates all the subpolicies simultaneously under the ensemble-aware loss with a diversity enhancement regularization. We also provide a theoretical analysis of EPPO on improving the entropy of the policy which leads to better exploration. Extensive experiments on various tasks demonstrate that EPPO substantially outperforms the baselines for both sample efficiency and policy generalization performance. In the future, we plan to incorporate more flexible sub-policy ensemble mechanisms and dive deeper into the mechanism behind the ensemble policy learning.