Teaching AI Agents Ethical Values Using Reinforcement Learning and Policy Orchestration

1 Introduction

Concerns about the ways in which autonomous decision making systems behave when deployed in the real world are growing. Stakeholders worry about systems achieving goals in ways that are not considered acceptable according to values and norms of the impacted community, also called "specification gaming" behaviors [Rossi and Mattei, 2019]. Thus, there is a growing need to understand how to constrain the actions of an AI system by providing boundaries within which the system must operate. To tackle this problem, we may take inspiration from humans, who often constrain the decisions and actions they take according to a number of exogenous priorities, be they moral, ethical, religious, or business values [Sen, 1974; Loreggia et al., 2018a; 2018b], and we may want the systems we build to be restricted in their actions by similar principles [Arnold et al., 2017]. The overriding concern is that the agents we construct may not obey these values while maximizing some objective function [Simonite, 2018; Rossi and Mattei, 2019].

∗This paper is an extended abstract of an article in the the IBM Journal of Research & Development [Noothigattu et al., 2019].

The idea of teaching machines right from wrong has become an important research topic in both AI [Yu et al., 2018] and related fields [Wallach and Allen, 2008]. Much of the research at the intersection of artificial intelligence and ethics falls under the heading of machine ethics, i.e., adding ethics and/or constraints to a particular system's decision making process [Anderson and Anderson, 2011]. One popular technique to handle these issues is called value alignment, i.e., restrict the behavior of an agent so that it can only pursue goals aligned to human values [Russell et al., 2015].

While giving a machine a code of ethics is important, the question of how to provide the behavioral constraints to the agent remains. A popular technique, the bottom-up approach, teaches a machine right and wrong by example [Allen et al., 2005; Balakrishnan et al., 2019; 2018]. Here we adopt this approach as we consider the case where only examples of the correct behavior are available.

We propose a framework which enables an agent to learn two policies: (1) πR, a reward maximizing policy obtained through direct interaction with the world, and (2) πC, obtained via inverse reinforcement learning over demonstrations by humans or other agents of how to obey a set of behavioral constraints in the domain. Our agent then uses a contextualbandit-based orchestrator [Bouneffouf and Rish, 2019; Bouneffouf et al., 2017] to learn to blend the policies in a way that maximizes a convex combination of the rewards and constraints. Within the RL community this can be seen as a particular type of apprenticeship learning [Abbeel and Ng, 2004] where the agent is learning how to be safe, rather than only maximizing reward [Leike et al., 2017].

One may argue that we should employ πC for all decisions as it will be more 'safe' than employing πR. Indeed, although one could use πC exclusively, there are a number of reasons to employ the orchestrator. First, while demonstrators may be good at demonstrating the constrained behavior, they may not provide good examples of maximizing reward. Second, the demonstrators may not be as creative as the agent when mixing the two policies [Ventura and Gates, 2018]. By allowing the orchestrator to learn when to apply which policy, the agent may be able to devise better ways to blend the policies, leading to behavior which both follows the constraints and achieves higher reward than any of the human demonstrations. Third, we may not want to obtain demonstrations in all parts of the domain e.g., there may be dangerous parts of the domain in which human demonstrations are too costly to obtain. In this case, having the agent learn what to do in the non-demonstrated parts through RL is complementary.

Contributions. We propose and test a novel approach to teach machines to act in ways blend multiple objectives. One objective is the desired goal and the other is a set of behavioral constraints, learned from examples. Our technique uses aspects of both traditional reinforcement learning and inverse reinforcement learning to identify policies that both maximize rewards and follow particular constraints. Our agent then blends these policies in novel and interpretable ways using an orchestrator. We demonstrate the effectiveness of these techniques on Pac-Man where the agent is able to learn both a reward-maximizing and a constrained policy, and select between these policies in a transparent way based on context, to employ a policy that achieves high reward and obeys the demonstrated constraints.

1.1 Problem Setting and Notation

Reinforcement learning defines a class of algorithms solving problems modeled as a Markov decision process (MDP). An MDP is denoted by the tuple (S,A, T ,R, γ), where: S is a set of possible states; A is a set of actions; T is a transition function defined by T (s, a, s′ ) = Pr(s′ |s, a), where s, s′ ∈ S and a ∈ A; R : S ×A× S ↦→ R is a reward function; γ is a discount factor that specifies how much long term reward is kept. The goal is to maximize the discounted long term reward. Usually the infinite-horizon objective is considered: max ∑∞t=0 γ tR(st, at, st+1).

Solutions come in the form of policies π : S ↦→ A, which specify what action the agent should take in any given state deterministically or stochastically. One way to solve this problem is through Q-learning with function approximation [Bertsekas and Tsitsiklis, 1996]. The Q-value of a stateaction pair, Q(s, a), is the expected future discounted reward for taking action a ∈ A in state s ∈ S. A common method to handle very large state spaces is to approximate the Q function as a linear function of some features.

Our problem is a multi-objective MDP. Instead of the scalar reward function R(s, a, s′ ), we have a reward vector ⃗R(s, a, s′ ) consisting of l dimensions representing the different objectives. However, not all components of the reward vector are observed. There is an objective v ∈ [l] that is hidden, and the agent is only allowed to observe demonstrations to learn this objective. These demonstrations are given in the form of trajectories {Tr(1) , Tr(2) , . . . , Tr(m) }. To summarize, for some objectives, the agent has rewards observed from interaction with the environment, and for some objectives the agent has only demonstrations. The aim is the same as single objective reinforcement learning: to maximize ∑∞t=0 γ tRi(st, at, st+1) for each i ∈ [l].

2 Proposed Approach

The overall approach we propose, aggregation at the policy phase, is depicted by Figure 1. It has three main components. The first is the IRL component to learn the desired constraints (depicted in green). We apply IRL to demonstrations depicting desirable behavior to learn the underlying constraint rewards being optimized by the demonstrations. We then apply RL on these learned rewards to learn a strongly constraintsatisfying policy πC. We augment πC with a pure reinforcement learning component applied to the original environment rewards (depicted in red) to learn a domain reward maximizing policy πR.

Now we have two policies: the constraint-obeying policy πC and the reward-maximizing policy πR. To combine them, we use the third component, the orchestrator (depicted in blue). This is a contextual bandit algorithm that orchestrates the two policies, picking one of them to play at each point of time. The context is the state of the environment; the bandit decides which arm (policy) to play at each step. We use a modified CTS algorithm to train the bandit. The context of the bandit is given by features of the current state (for which we want to decide which policy to choose), i.e., c(t) = Υ(st) ∈ Rd .

The exact algorithm used to train the orchestrator is given in Algorithm 1. Apart from the fact that arms are policies (instead of atomic actions), the main difference from the CTS algorithm is the way rewards are fed into the bandit. For simplicity, let the constraint policy πC be arm 0 and the reward policy πR be arm 1. First, all the parameters are initialized as in the CTS algorithm (Line 1). For each time-step in the training phase (Line 3), we do the following. Pick an arm kt according to the Thompson Sampling algorithm and the context Υ(st) (Lines 4 and 5). Play the action according to the chosen policy πkt (Line 6). This takes us to the next state st+1. We also observe two rewards (Line 7): (i) the original reward in environment, rR at (t) = R(st, at, st+1) and (ii) the constraint rewards according to the rewards learnt by inverse reinforcement learning, i.e., rC at (t) = ̂RC(st, at, st+1). rC at (t) can intuitively be seen as the predicted reward (or penalty) for any constraint satisfaction (or violation) in this step.

Algorithm 1 Orchestrator Based Algorithm

• 1: Initialize: Bk = Id, ˆµk = 0d, fk = 0d for k ∈ {0, 1}.
• 2: Observe start state s0.
• 3: Foreach t = 0, 1, 2, ..., (T - 1) do
• 4: Sample ˜ µk(t) fromN(ˆµk, v2B-1 k ).
• 5: Pick arm kt = argmax k∈{0,1} Υ(st)⊤ ˜ µk(t).
• 6: Play corresponding action at = πkt (st).
• 7: Observe rewards rC at (t), rR at (t), and next state st+1.
• 8: Define rkt (t) = λ (rC at (t) + γV C (st+1))

• 9: Update Bkt = Bkt + Υ(st)Υ(st)⊤, fkt = fkt + Υ(st)rkt (t), ˆµkt = B-1 kt fkt
• 10: End

To train the contextual bandit to choose arms that perform well on both metrics (environment rewards and constraints), we feed it a reward that is a linear combination of rR at (t) and rC at (t) (Line 8). Another important point to note is that rR at (t) and rC at (t) are immediate rewards achieved on taking action at from st, they do not capture long term effects of this action. In particular, it is important to also look at the "value" of the next state st+1 reached, since we are in the sequential decision making setting. Precisely for this reason, we also incorporate the value-function of the next state st+1 according to both the reward maximizing component and constraint component (which encapsulate the long-term rewards and constraint satisfaction possible from st+1). This gives exactly Line 8, where V C is the value-function according the constraint policy πC, and V R is the value-function according to the reward maximizing policy πR.

In this equation, λ is a hyperparameter chosen by a user to decide how much to trade off environment rewards for constraint satisfaction. For example, when λ is set to 0, the orchestrator would always play the reward policy πR, while for λ = 1, the orchestrator would always play the constraint policy πC. For any value of λ in-between, the orchestrator is expected to pick policies at each point of time that would perform well on both metrics (weighed according to λ). Finally, for the desired reward rkt (t) and the context Υ(st), the parameters of the bandit are updated according to the CTS algorithm (Line 9).

3 Demonstration on Pac-Man

We demonstrate the applicability of the proposed algorithm using the classic game of Pac-Man. The rules for the environment (adopted from Berkeley AI Pac-Man 1 ) are as follows. The goal of the agent is to eat all the dots in the maze, known as Pac-Dots, as soon as possible while simultaneously avoiding collision with ghosts. On eating a Pac-Dot, the agent obtains a reward of +10. On successfully eating all the PacDots, the agent obtains a reward of +500. In the meantime, the ghosts roam the maze trying to kill Pac-Man. On collision with a ghost, Pac-Man loses the game and gets a reward of -500. The game also has two special dots called Power Pellets in the corners of the maze, which on consumption, give Pac-Man the temporary ability of "eating" ghosts. During this phase, the ghosts are in a "scared" state for 40 frames and move at half their speed. On eating a ghost, the agent gets a reward of +200, the ghost returns to the center box and returns to its normal "unscared" state. Finally, there is a constant time-penalty of -1 for every step taken.

1 http://ai.berkeley.edu/project overview.html

For the sake of demonstration of our approach, we define not eating ghosts as the desirable constraint in the game of Pac-Man. However, recall that this constraint is not given explicitly to the agent, but only through examples. To play optimally in the original game one should eat ghosts to earn bonus points, but doing so is being demonstrated as undesirable. Hence, the agent has to combine the goal of collecting the most points while not eating ghosts.

3.1 Details of the Pure RL

For the reinforcement learning component, we use Q-learning with linear function approximation. Some of the features we use for an 〈s, a〉 pair (for the ψ(s, a) function) are: "whether food will be eaten", "distance of the next closest food", "whether a scared (unscared) ghost collision is possible" and "distance of the closest scared (unscared) ghost". For the layout of Pac-Man we use, an upper bound on the maximum score achievable in the game is 2170. This is because there are 97 Pac-Dots, each ghost can be eaten at most twice (because of two capsules in the layout), Pac-Man can win the game only once and it would require more than 100 steps in the environment. On playing a total of 100 games, our reinforcement learning algorithm (the reward maximizing policy πR) achieves an average game score of 1675.86, and the maximum score achieved is 2144. We mention this here, so that the results in Section 4 can be seen in appropriate light.

3.2 Details of the IRL

For Pac-Man, observe that the original reward function R(s, a, s′ ) depends only on the following factors: "number of Pac-Dots eating in this step (s, a, s′ )", "whether Pac-Man has won in this step", "number of ghosts eaten in this step" and "whether Pac-Man has lost in this step". For our IRL algorithm, we use exactly these as the features φ(s, a, s′ ). As a sanity check, when IRL is run on environment reward optimal trajectories (generated from our policy πR), we recover something very similar to the original reward function R. In particular, the weights of the reward features learned is given by 1/1000[+2.44,+138.80,+282.49,-949.17], which when scaled is almost equivalent to the true weights [+10,+500,+200,-500] in terms of their optimal policies. The number of trajectories used for this is 100.

Ideally, we would prefer to have the constrained demonstrations given to us by humans, but for the sake of simplicity we generate them synthetically as follows. We learn a policy π⋆C by training it on the game with the original reward function R augmented with a very high negative reward (-1000) for eating ghosts. This causes π⋆C to play well in the game while avoiding eating ghosts as much as possible. 2 Now, to emulate erroneous human behavior, we use π⋆C with an error probability of 3%. That is, at every time step, with 3% probability we pick a completely random action, and otherwise follow π⋆C . This gives us our constrained demonstrations, on which we perform inverse reinforcement learning to learn the rewards capturing the constraints. The weights of the reward function learned is given by 1/1000[+2.84,+55.07,-970.59,-234.34], and it is evident that it has learned that eating ghosts strongly violates the favorable constraints. The number of demonstrations used for this is 100. We scale these weights to have a similar L1 norm as the original reward weights [+10, +500, +200, -500], and denote the corresponding reward function by ̂RC.

Finally, running reinforcement learning on these rewards ̂RC, gives us our constraint policy πC. On playing a total of 100 games, πC achieves an average game score of 1268.52 and eats just 0.03 ghosts on an average. Note that, when eating ghosts is prohibited in the domain, an upper bound on the maximum score achievable is 1370.

4 Evaluation

We measure performance on two metrics, (i) the total score achieved in the game (the environment rewards) and (ii) the number of ghosts eaten (the constraint violation). We also observe how these metrics vary with λ. For each value of λ, the orchestrator is trained for 100 games. The results are shown in Figure 2. Each point is averaged over 100 test games.

The graph shows a very interesting pattern. When λ is at most than 0.215, the agent eats a lot of ghosts, but when it is above 0.22, it eats almost no ghosts. In other words, there is a value λo which behaves as a tipping point, across which there is drastic change in behavior. Beyond the threshold, the agent learns that eating ghosts is not worth the score it is getting and so it avoids eating as much as possible. On the other hand, when λ is smaller than λo, it learns the reverse and eats as many ghosts as possible.

Policy-switching. One important property of our approach is interpretability, we know exactly which policy is being played at each time. For moderate values of λ > λo, the orchestrator learns a very interesting policy-switching technique: whenever at least one of the ghosts in the domain is scared, it plays πC, but if no ghosts are scared, it plays πR. In other words, it starts the game playing πR until a capsule is eaten. As soon as the first capsule is eaten, it switches to πC until the scared timer runs off. Then it switches back to πR until another capsule is eaten, and so on. It has learned a very intuitive behavior: when there is no scared ghost, there is no possibility of violating constraints. Hence, the agent is as greedy as possible (i.e., play πR). However, when there are scared ghosts, it is better to be safe (i.e., play πC).

2 We do this only for generating demonstrations. In real domains, we would not have access to the exact constraints that we want to be satisfied, and hence a policy like π⋆C cannot be learned; learning from human demonstrations would then be essential.

5 Discussion

In this paper, we considered the problem of autonomous agents learning policies constrained by implicitly-specified values while still optimizing their policies with respect to environmental rewards. We have taken an approach that combines IRL to determine constraint-satisfying policies from demonstrations, RL to determine reward-maximizing policies, and a contextual bandit to orchestrate between these policies in a transparent way. This proposed architecture and approach for the problem is novel. It also requires a novel technical contribution in the contextual bandit algorithm because the arms are policies rather than atomic actions, thereby requiring rewards to account for sequential decision making. We have demonstrated the algorithm on Pac-Man and found it to perform interesting switching behavior among policies.

The contribution herein is only a starting point. We can pursue deep IRL to learn constraints without hand-crafted features and research IRL algorithms to learn from just one or two demonstrations (perhaps in concert with knowledge and reasoning). In real-world settings, demonstrations will likely be given by different users with different versions of abiding behavior; we would like to exploit the partition of the set of traces by user to improve the policy or policies learned via IRL. Additionally, the current orchestrator selects a single policy at each time, but more sophisticated policy aggregation techniques for combining or mixing policies is possible. Lastly, it would be interesting to investigate whether the policy aggregation rule (λ) can be learned from demonstrations.

Acknowledgments

This work was conducted under the auspices of the IBM Science for Social Good initiative.