Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning (Extended Abstract)

1 Introduction

Reinforcement learning (RL) has shown great promise in solving complex decision problems, such as Go [Silver et al., 2017], video games [Berner et al., 2019; Vinyals et al., 2019] and dexterous robotic control [Nagabandi et al., 2020]. Learning by trial and error, RL enables an agent to maximize its accumulated expected rewards through interaction with a simulator. However, RL deployment in real-world scenarios is still challenging and unreliable [Garcıa and Fernández, 2015; Dulac-Arnold et al., 2019]. One of the reasons is that real decision-makers need to consider multi-objective functions.

* Here is the original journal paper [Ma et al., 2022c].

The desired policy should perform well for broader metrics, not just for expectation. That raises the demand of risksensitive learning, which aims at balancing the return and risk in the face of uncertainty.

The risk-sensitive decision-making has been widely studied beyond the scope of RL, which can be traced back to the mean-variance (MV) optimization theory established by Markowitz [Markowitz, 1952]. Variance, which captures the fluctuation and concentration of random variables, is a natural choice of the risk measure. As Markowitz only considers the single-period problem, many studies focus on extending the results to multi-period scenarios, from stochastic control [Li and Ng, 2000] to Markov decision process [Sobel, 1982; Filar et al., 1989]. However, the variance of a multi-period problem depends on the average value of the whole process. It breaks the essential property of dynamic programming- time consistency, and makes it hard to design model-free learning algorithms under the standard RL framework. Developing an efficient algorithm to optimize MV is still an ongoing topic in the RL community [Xie et al., 2018; Bisi et al., 2020; Xia, 2020; Zhang et al., 2021; Ma et al., 2022b; Ma et al., 2022a].

While MV analysis is the most widely applied risk-return analysis in practice, variance metric is questionable as a risk measure. As a measure of volatility, variance penalizes upside deviations from the mean as much as downside deviations. It could be problematic as the upside deviation comes from the higher return which is desirable. In general, the outcome distributions in the real world are often asymmetrical, such as the ones in the stock market [Estrada, 2007; Bollerslev et al., 2020], suggesting that we should control the "good" and "bad" volatility separately. Hence, Markowitz 1959 presents the mean-semivariance (MSV) as an alternative measure, which only penalizes the "bad" volatility, performing as a downside risk indicator. Even if the distribution is symmetrical, optimizing MSV is at least effective as optimizing MV. To better illustrate the difference between variance and semivariance, we construct a simple MDP example shown in Figure 1. The two policies result in two reward distributions symmetrically, for which variances are indistinguishable. However, the policy going right is preferred since it results in a lower semivariance.

Though MSV is a more plausible measure of risk, optimizing MSV is even more complicated than MV. It inherits time inconsistency from variance and introduces a truncation function of mean, making the analysis non-trivial. Due to the complexity of this objective, existing works consider a subset of problems restricted with a fixed mean [Wei, 2019] or heuristic algorithms for MSV [Yan et al., 2007; Zhang et al., 2012; Liu and Zhang, 2015; Chen et al., 2019]. To the best of our knowledge, there are currently no relevant studies on MSV in the RL literature.

In this paper, we aim to fill the gap of the previous study on the single-period MSV problem and extend the static methods to online RL algorithms. To achieve that, we resort to Perturbation Analysis (PA) theory [Cao, 2007] (also called the sensitivity-based optimization theory or the relative optimization theory) for Markov systems, which lays the basis of many efficient RL methods, such as TRPO [Schulman et al., 2015], CPO [Achiam et al., 2017] and MBPO [Janner et al., 2019]. The contributions of our work are threefold. Firstly, instead of constructing a Bellman operator, we establish the MSV performance difference formula of two policies (see Section 3 for details). The result indicates that the performance difference can be decomposed into two parts: the improvement corresponding to a reward function depending on the current policy and the average performance change from the current to the updated one. Second, we iteratively optimize MSV by considering the shift in mean locally and constructing a surrogate reward function. We develop two algorithms based on the policy gradient theory and the trust region method, respectively. We show that optimizing the surrogate reward function in the trust region has a similar performance lower bound with the standard TRPO, which guarantees monotonic improvement if the trust region is tight. Finally, we conduct diverse experiments to examine the effectiveness of our proposed methods, including a bandit problem, a tabular portfolio management problem, and robotic control tasks based on MuJoCo. The results demonstrate that the proposed algorithms successfully improve the performance under the criterion of MSV, which is better than standard RL from a risk-averse perspective.

2 Preliminaries

In this paper, we focus on the infinite-horizon discretetime MDP as M = ⟨S,A, r, P, π0⟩, where S denotes the state space, A denotes the action space, r : S × A ↦→ [-Rmax,Rmax] denotes a bounded reward function and P : S × A ↦→ ∆(S) is the transition matrix and π0 ∈ ∆(S) denotes the initial state distribution. We assume that all the involved MDPs are ergodic. Let µ : S ↦→ ∆(A) denote a Markovian randomized policy and Π denote the randomized policy space.

We are interested in the long-run average reward

where Eπ0,µ stands for the expectation with s0 ∼ π0, at ∼ µ(· | st), st+1 ∼ P(· | st, at). Note that ηµ is independent of π0 when T →∞. The variance and semivariance w.r.t. µ are defined by

where (·)- := min{0, ·}. In this paper, we focus on the mean-semivariance criterion,

where β ≥ 0 is the parameter for the trade-off between mean and semivariance. Analogously, when mean-variance criterion is mentioned, we mean ξµ := ηµ - βζ µ .

We further respectively define the state-value function, action-value function, and advantage function for average reward as

Similarly, the value functions for semivariance are defined as

For notation simplicity, we will omit the superscript "µ" when the context is clear, e.g., the average rewards ηµ , ηµ′ are written as η, η′ instead. When r is mentioned, we omit (s, a) and use r in short.

3 Perturbation Analysis

In this section, we derive the MSV performance difference formula (MSVPDF), where the core concept-performance difference formula-comes from the PA for Markov systems, also called the sensitivity-based optimization theory. With the aid of MSVPDF, we obtain the necessary optimality condition for the MSV problem. It also lays the basis for developing optimization algorithms (see Section 4), such as the policy gradient method and the trust region method.

3.1 Performance Difference Formula

MSVPDF is formally stated below.

The MSVPDF in Equation 4 claims that the MSV improvement can be divided into two parts. The first term in Equation 4 is a standard MDP with f as the reward function, and the second term is caused by the perturbation of the mean. It clearly quantifies the difficulty of solving the MSV problem, i.e., the policy-dependent reward function breaks down the time-consistent nature of MDPs. Meanwhile, it also shows us the standard MDP algorithm such as policy iteration (PI) is unavailable. A PI-like algorithm may be efficient in improving the first term, but the sign of the remaining term (dependent on η′ ) is unpredictable. It suggests that we need novel tools to guarantee policy improvement.

3.2 Performance Derivative Formula

While Equation 4 describes the performance difference between any two policies, we still need the local structure of the MSV problem to guide the direction of optimization. Following the line of the last part, we present the MSV performance derivative formula in this subsection, which describes the performance derivative at µ towards another policy µ′ .

Theorem 2. Given any two policies µ, µ′ ∈ Π, we consider a mixed policy µν ,

where the action follows µ with probability 1 - ν, and follows µ′ with probability ν for ν ∈ [0, 1]. We have

The above equality indicates that the performance derivative is related to a pseudo another reward function:

and the derivative formula can be written as

where Aµg (s, a) is the advantage function w.r.t. g.

4 Optimization and Algorithms

In this section, we propose two approaches to optimize MSV with the parameterized policy. We firstly extend the policy gradient method to MSV with the pseudo reward function (cf. Equation 5) in Section 3. Following the same idea, we propose a trust region method to solve the MSV problem and prove the lower bound for its performance improvement. The two approaches together establish an iterative framework to solve the MSV problem.

4.1 MSV Policy Gradient Method

Policy gradient theorem is an essential foundation of modern deep RL algorithms, such as Actor-Critic methods. Here we consider the policy µ parameterized by θ ∈ Θ, which can be implemented with any differentiable function. We first give the MSV Policy Gradient (MSVPG) theory formally as follows.

Theorem 3. For a policy µ parameterized by θ, we have

The policy gradient for MSV can be easily proved by PA, which follows the same lines as the derivative formula.

4.2 MSV Trust RegionMethod

While PG has a concise form, it often suffers from the difficulty of selecting step-sizes and the sensitivity to initial points in practice, especially when it works with neural networks. To address these drawbacks, trust region method [Schulman et al., 2015] is proposed to solve a surrogate problem in a local trust region and perform an approximate policy iteration.

Monotonic Improvement Guarantee

We extend the idea of trust region in the standard MDP into MSV and propose the MSV Trust Region Policy Optimization (MSVTRPO) method. In MSVTRPO, we iteratively solve the problem below

where

Remark 1. The trust region method updates the policy via the direction of maximum derivative (cf. the performance derivative formula in Equation 6), constrained in the proximity policy space with the TV -divergence. In contrast, the standard policy iteration scheme updates the policy in the same direction without constraint, which breaks the monotonic improvement for MSV.

Next, we will show that MSVTRPO enjoys an analogous performance improvement bound. When the trust region is tight enough, i.e., ϵµ → 0, the lower bound is dominated by the first-order term.

Theorem 4. Let µ′ be the solution to the problem defined by Equation 8. We have

where ϵg = maxs |Ea∼µ′ [Aµg (s, a)]| and κ′ is Kemeny's constant under µ′ .

5 Experiments

In the previous sections, we analyze the properties of MSV problem and find that it can be solved by iteratively optimizing a surrogate reward function g (cf. Equation 5). We also propose two methods to solve the MSV problem in the parameterized policy space.

To validate the effectiveness of our proposed methods in solving the MSV problem, we conduct a series of experiments to answer the corresponding questions:

• Is the MSV really optimized by the surrogate reward function g? Specifically, what is the difference between optimizing g instead of f?
• What is the difference between the MV [Xia, 2020] and MSV criteria?
• Does the proposed algorithms work well with the current deep RL algorithms?

5.1 Bandit Problem

We start with a simple bandit problem. In this problem, there are three actions with only a single state. Different actions result in different rewards following the distributions shown in Figure 2(a). We compare three different agents, which optimize different reward functions. The first one optimizes g = (1 + 2η-)r - (r - η)2-, which is the derived reward function with β = 1 in this work. The second one optimizes f = r-(r-η)2-, which is the Monte-Carlo return of MSV. We further consider a third agent which optimizes r-(r-η)2 [Xia, 2020], an MV objective to illustrate the difference between MSV and MV problems.

The result tells us optimizing the reward f = r-β(r-η)2cannot optimize the MSV objective even in such a simple problem. This reflects the most essential difference between the optimization of policy-dependent reward and other problems. As discussed in Section 3, to optimize a problem with a policydependent reward function, we must consider the perturbation of the mean, at least in MSV problems.

5.2 PortfolioManagement

In this part, we compare the performances of MSV- and MVoptimal policies in a portfolio management problem, where we need to manage two independent assets and cash.

We change the risk preference parameter β and compare the MSVTRPI and MVPI. We depict the result in Figure 3, showing that with a fixed β, optimizing MSV always results in a larger return than that of MV. Besides, MV is more sensitive than MSV in terms of β, meaning that a small change of β will lead to a quick drop in both the return and risk. To better compare MSV and MV, we also show the "normalized" results of MSV, where we double β to provide the same penalty strength as MV. The result shows the normalized MSV also outperforms MV in terms of the average reward, illustrating that MSV is more plausible than MV.

5.3 Robotic Control

To demonstrate the effectiveness of our proposed method in more general problem setups, we implement a "deep" variant algorithm named mean-semivariance policy optimization (MSVPO), which is based on the recently developed method APO [Ma et al., 2021] for average-reward RL problems.

We evaluate MSVPO with different β's in the noisy Walker2d with different noise levels. When the agent falls, we penalize it with an extra cost -10 and reset the system. In the noiseless environment (noise level = 0), we interestingly find that risk-averse policy (β = 0.1) achieves competitive average reward with lower semivariance. It indicates that in complex scenes, optimizing a risk-averse metric may generate more robust policies with better performances compared with a risk-neutral one.

To better understand the performance difference with different risk preference policies, we visualize the reward distributions of typical agents in Figure 4, where each agent of noise level 0.1 is evaluated for 1000 steps. We can see that riskaverse policies successfully avoid unsafe states. Meanwhile, the agent uses smaller steps forward with the risk parameter β increasing. Instead, the risk-neutral agent tends to take the risk of falling for larger gains.

Acknowledgments

This work is funded by the National Natural Science Foundation of China (No. U1813216, 62192751, 61425027, 62073346, 11931018, U1811462), the National Key Research and Development Project of China under Grant 2017YFC0704100 and Grant 2016YFB0901900, in part by the 111 International Collaboration Program of China under Grant BP2018006, BNRist Program (BNR2019TD01009), the National Innovation Center of High Speed Train R&D project (CX/KJ-2020-0006), the Guangdong Province Key Laboratory of Computational Science at the Sun Yat-Sen University (2020B1212060032) and the Guangdong Basic and Applied Basic Research Foundation (2021A1515011984).