Adversarial Behavior Exclusion for Safe Reinforcement Learning

1 Introduction

In the last two decades, RL has greatly evolved demonstrating its potential in a wide range of applications including robotics [Kober et al., 2013], and autonomous driving [Grigorescu et al., 2020]. To be deployed in the real world, ensuring safety is a crucial factor that RL inherently lacks due to its exploratory learning. One way safety can be ensured in RL is by modifying its objective to optimize both the safety goals and the task learning [Kim et al., 2020] [Geibel, 2006]. In this approach, the RL agent must explore a significant number of safetyviolating states which inherently leads to sub-optimal policies due to the conflict between the task learning and safety objectives [Thananjeyan et al., 2021]. Another way for ensuring RL safety is to enforce safe exploration by endowing explicit constraints. Manual specification of such constraints [Levine et al., 2018] though possible in environments with known dynamics, cannot be generalized to any slight changes in those environments. Besides, safety specifications based on estimating the environment dynamics during offline learning [Bastani, 2021] [Alshiekh et al., 2018] are not sufficient to assure that the RL agent will behave safely during runtime for two reasons. First, the details of the environment dynamics can't be fully known at training time, which partially invalidates the initial assumptions about the environment modeling used to design the safety specifications. Second, RL agents are susceptible to even subtle perturbations in their observations and actions, known as adversarial examples [Chen et al., 2019] [Lee et al., 2020] which introduces novel perturbations in the environment model. Although a vulnerable policy under adversarial attacks cannot be regarded as truly safe in the physical world, there is little research studying the robustness of the safe RL methods against those attacks. In this paper, we investigate the following question: how can we ensure a robust safety for the RL agent without impairing its task learning under deliberate perturbations?

We propose the Adversarial Behavior Exclusion (AdvExRL) framework for safe RL. AdvEx-RL first trains an adversarial policy to interactively extract unsafe behaviors by maximizing the safety violations in the environment. Then, it learns a safety policy by maximizing its divergence from the adversarial policy. This approach is different from adversarial training which requires the agent to learn its task in a zero-sum game with the adversary. Instead, we train the adversarial policy to estimate the safety of each encountered state and to learn the underlying behavior most likely violates the safety constraints in the environment. Our contributions are three-folded: (1) developing a task-agnostic safety learning framework, AdvEx-RL, where the RL agent can use it as a safety firewall to avoid unsafe behaviors, (2) introducing a new safety learning method separate from task learning by deriving a state-action distribution that is most divergent from the state-action distribution of an approximated optimal adversary, and (3) providing a theoretical proof for the safety of AdvEx-RL along with empirical evaluations against 7 baselines in 3 continuous MuJoCo environments from [Thananjeyan et al., 2021] and SafetyGym environments [Ray et al., 2019] . Unlike prior safe RL studies, in our empirical evaluation, we consider factors such as robustness and frequency of deadlocks. We also demonstrate the interpretability of AdvEx-RL through a study conducted on 41 end users using an extended version of CAPS [McCalmon et al., 2022].

2 RelatedWork

We adopt 3 algorithmic concepts from the literature: (1) safe exploration via online shielding (2) value functionbased safety estimation, and (3) the use of two policies. AdvEx-RL integrates the shielding concept from [Alshiekh et al., 2018; ElSayed-Aly et al., 2021; Bansal et al., 2017; Fisac et al., 2019] to prevent the agent from visiting unsafe states during runtime. We employ a post-posed [Alshiekh et al., 2018] shield in AdvEx-RL, but instead of deriving the safety probabilities using formal methods, we use the value-function based safety estimation [Geibel and Wysotzki, 2005], [Hans et al., 2008; Srinivasan et al., 2020; Thananjeyan et al., 2021].

In [Geibel and Wysotzki, 2005; Hans et al., 2008], Qlearning is used for the risk estimation and to derive the riskaverse and rescue policies [Mihatsch and Neuneier, 2002]. The safety violations is estimated in [Srinivasan et al., 2020] through a safety critic implemented using DQN. Then, the safety Q-function estimation is utilized to optimize a Lagrangian relaxation (LR) objective to derive a safety policy. [Srinivasan et al., 2020] shows that any policy constrained under a safety Q-function is guaranteed to be safe. RecoveryRL [Thananjeyan et al., 2021] adopts the concept of safety Q-function [Srinivasan et al., 2020] to develop a shielding mechanism. They use offline demonstration data collected from human-supervised policy to train a safety estimator, Qrisk then derives a model-free recovery policy by defining an LR objective function that minimizes the Q-risk. The performance of [Thananjeyan et al., 2021] is greatly reliant on the human-supervised offline data. In AdvEx-RL we employ a similar shielding mechanism by applying a threshold on the safety estimation from a critic network [Srinivasan et al., 2020; Thananjeyan et al., 2021]. However, instead of training a different DQN safety critic through exploring a pre-training environment [Srinivasan et al., 2020] or on human supervised data [Thananjeyan et al., 2021], we acquire the safety critic from the critic network of a trained adversary.

In AdvEx-RL, we follow the strategy of using two separate policies i.e. task policy and safety policy similar to[Bastani, 2021; Thananjeyan et al., 2021]. In [Bastani, 2021], a shield mechanism is used to switch between the task and safety policies. The safety policy is derived using a non-linear model predictive controller (NMPC) that in the advent of probable safety violations, resets the agent to some fixed initial safety point within the environment. Whereas [Thananjeyan et al., 2021] uses a similar concept but instead of resetting the agent to some fixed initial point, they reset the agent to nearby safe points. The safety policy in NMPC [Bastani, 2021] and MPC [Thananjeyan et al., 2021] requires prior knowledge about the environment dynamics or demonstration data.

3 Problem Statement

We consider the standard CMDPs, M = (S,A, µ,P(.|., .),R, γ, C) where S and A denote the state and action space; µ and P : S ×A× S -→ [0, 1] denote the initial state distribution and state transition dynamics respectively. R : S ×A -→ R is the reward function; γ denotes discount factor; and C = {ci : S ×A S-→ R ≥ 0; i = 1, 2, .., T} denotes the set of cost associated with safety constraint violations in any trajectory episode τ = {s0, a0, ..., aT-1, sT} with a maximum trajectory length of T. We assume that accomplishing the task goal or violating a safety constraint in M leads to episode termination. Let Π be the set of stationary MDP policies for M such that πtask , πadv , πsafety ∈ Π denote the task policy, adversary policy and safety policy respectively. The objective of task policy πtask is to learn the optimal control to maximize the expected discounted reward at time t; Rπtask = Eτ∼πtask[∑Tt′ =t γ t′ -t rt]. On the other hand, the objective of adversary policy πadv is to maximize the expected discounted cost associated with safety violations Cπadv = Eτ∼πadv [∑Tt′ =t γ t′ -t ct]. AdvEx-RL works like a protective safety firewall integrated with a safety shield allowing task policy πtask and safety policy πsafety to be learned and executed completely independent of each other as depicted in Fig.1.

4 Adversarial Behavior Exclusion for Safe Reinforcement Learning

AdvEx-RL (Fig.1) uses a post-posed shielding mechanism to assess the agent's current safety and act appropriately following either its task policy or safety policy. The task policy is any conventional RL policy optimized to learn a certain task while the safety policy is a task-agnostic policy that aims to only maximize the agent's safety in a certain environment. When the task policy takes the agent closer to a potentially dangerous/unsafe state, the safety policy is triggered by the shield to rescue the agent to a nearby safe state.

4.1 First, Learn to be Unsafe

AdvEx-RL first extracts unsafe behavior by interactively training an optimal adversary in M. The adversary, similar to any conventional RL, learns an optimal policy πadv to maximize the accumulated cost associated with the safety violations through the exploration-exploitation principle of RL. To improve the exploration of the adversarial policy, we use off-policy learning with entropy regularization by maximizing the following objective:

where H(πadv ϕ (.|st)) is the policy entropy and α is the temperature parameter. Using the maximum entropy policy objective [Haarnoja et al., 2018a], the optimal stateaction function Qadv∗ (s, a) can be approximated through soft Q-function that is evaluated with Bellman backup operator τ πadv as:

where,

The optimal Qadv∗ (s, a) provides the estimation of the expected cost over any trajectory τ ∈ M; Qadv∗ (s, a) = Eτ∼πadv∗ [∑Tt′ =t γ t′ -t c(st)]. Therefore, we approximate Qadv φ (s, a) by minimizing soft bellman residual [Haarnoja et al., 2018a] and use it to quantify safety violation as we will show in Section 4.3 . (Details on the training of the adversarial policy are given in Appendix 1 A, Algorithm 1.)

4.2 Learn to be Safe fromTheWorst Behavior

The second phase of AdvEx-RL is to learn a task-agnostic safety policy exploiting the adversarial policy πadv under the following assumptions:

Assumption 1: Given CMDP M, with the state space S = Ssafe ∪ Sunsafe , if a state s ∈ Ssafe have any neighboring unsafe state ˜ s ∈ Sunsafe , then the transition probability denoting safety violation upon taking an available action a at that state P(˜ s|s, a) > 0.

1 Appendix link

Assumption 2: The adversarial policy has sufficiently explored M and is optimal πadv∗ such that Qadv∗ ≥ Qadv′ where πadv′ is any sub-optimal adversary policy. Therefore Qadv∗ (.) can quantify the expected cost for any trajectory τ ∈M.

Assumption 3: Every state has at least one neighboring safe state i.e. at any state st, there exists at least one safe action that can lead the agent to a neighboring safe state; ∃at s.t P(st+1|st, at) > 0 where st+1 ∈ Ssafe .

Theorem 1 (Reduction of safety violation probability): The probability of following a trajectory that violates safety can be reduced by increasing KL divergence between the adversarial policy πadv and any arbitrary policy π′ .

Proof

Let's consider the objective of the optimal adversary in M which maximizes the expected cost (Eq.1). Stationary policy πadv ∼ πadv∗ represents a one-to-one correspondence [Puterman, 2014] with the state-action distribution in Mwhich can be computed by:

From inverse reinforcement learning principle [Ghasemipour et al., 2019], the expected return of an arbitrary policy π′ can be computed with respect to the optimal adversary policy (i.e. expert) πadv as:

This intuitively means that, as an arbitrary policy π′ becomes more similar to an optimal adversary πadv by minimizing the KL divergence DKL(π′ ||πadv ), the expected cost associated with safety violations increases. Meanwhile, according to the probabilistic inference [Levine, 2018], the probability of choosing a trajectory τ involving safety violation under πadv can be given by

Then optimal πadv can be denoted as:

The KL divergence term in the above equation acts like a penalty regularization [Goo and Niekum, 2022] that guides the arbitrary policy π′ closer to the optimal adversary i.e DKL → 0 which in turn maximizes the expected cost. This also indicates that if the trajectory distribution under the arbitrary policy π′ diverges from the trajectory distribution of the optimal adversary, it will then minimize the expected cost. Since diverging from a trajectory distribution is lower bounded by its state distribution [Ke et al., 2021] and considering assumption 3, maximizing KL divergence of the arbitrary policy with respect to the optimal adversary policy guarantees to reduce the probability of selecting a trajectory leading to safety violations.

We use this theorem as the learning principle of the safety policy, where the state-action distribution of policy πsafe is derived by maximizing its KL-divergence from πadv according to the following objective:

πsafe is trained by taking samples from a set of trajectories τπsafe; where the distribution of any trajectory τ = {s0, a0, s1, a1, ...., aT , sT} under πsafe is given by ρπsafe (τ) = µ(s0) ∏Tt=1 πsafe (at|st)P(st+1|st, at) then:

Here the KL divergence DKL(πsafe (τ)||πadv (τ)) is upper bounded by Pinsker's inequality i.e. the square root of total variational distance between the trajectory distribution under πsafe and πadv . To ensure that πsafe is not only divergent from πadv but also will rescue the agent to a safe state, we change the objective in Eq.8 to state-pairwise safety learning by optimizing πsafe as:

Where Qestadv (st, at) is derived by minimizing objective:

Maximizing Eq.10 ensures that the trajectory distribution of the safety policy πsafe is diverse from the trajectory distribution of πadv ϕ while increasing the probability of sampling trajectories that have lower expected safety violation cost. Consequently, πsafe will rescue the agent to nearby states that have lower values for the adversary or in other words safer for the agent.

Sampling trajectories directly under πsafe would lead to poor exploration. Therefore, we train πsafe using a rollout mechanism. This mechanism generates a demo trajectory τdemo using either the adversarial policy πadv ϕ , the task policy πtask , or randomly sampled actions. For each state st ∈ τdemo; safety policy πsafe generates new trajectories τπsafe ∼ (st, st+1,C(st)) which are then used to train the safety policy πsafe by maximizing the objective function of Eq.10 in an off-policy fashion. (More details on AdvEx-RL safety policy training can be found in Appendix B, Algorithm 2).

4.3 Online Execution with Safety Shielding

We adopt the strategy of using a post-posed shield from [Alshiekh et al., 2018] only during the online execution. Unlike [Srinivasan et al., 2020; Thananjeyan et al., 2021], which separately trains a DQN safety critic function Qπrisk for safety estimation, we instead utilize the critic function of the optimal adversary Qadv φ that we trained previously (Eq.2) to implement the safety shield as:

where Tsafety is a predefined threshold value such that at any state st and for any action at ∼ πtask (st); if Shield(st, at) is triggered, then the AdvEx-RL safety firewall replaces the selected action at by a safer action given by the safety policy asaf e t ∼ πsafety (st). The value of Tsafety is environmentspecific and can be chosen based on a sensitivity test for each environment (see Appendix C for details about the sensitivity test. Algorithm 3 in Appendix D shows the online execution of AdvEx-RL.)

Deadlock Side Effect: Providing safety in 2 separate policies is prone to deadlock as the agent may loop between the same states due to switching between πsafe and πtask . Suppose at any timestep t, for action atc ∼ πtask (.|stc ) at a critical state stc , the shield is triggered. From this state stc onward, the shield will make sure that the agent keeps following a safe trajectory using πsafe until it reaches a safe state sts such that for action atask ∼ πtask (.|sts ); Qadv φ (sts , atask) < Tsafety is satisfied. Afterward, the agent can select actions using its task policy πtask unless the shield is triggered again. However, due to the presence of external perturbation or inherent weakness within the task policy πtask , the agent might cycle back to the same old critical state stc while selecting action atc ; resulting in an inadvertent deadlock.

Although deadlock can hamper the agent's task, it can save the agent from dangerous situations. For example, a deadlock can be a safe, temporary solution for an expensive robot until it gets rescued by human operators. To examine the proposed AdvEx-RL for deadlocks, we empirically analyze the frequency of deadlocks in the tested environments using the deadlock detection proposed in [Ye et al., 2022].

5 Practical Implementation

The adversarial policy was trained using SAC [Haarnoja et al., 2018b] since it provides better exploration. The task policy was also trained using SAC but it can be trained using any RL algorithm. The safety policy was trained by performing gradient descent on the objective function in Eq.10. The postposed shield was implemented as a safety assurance layer that explicitly replaces any unsafe action selected by the task policy with a safe action chosen by the safety policy during execution.

6 Experiments

The experiments in this paper are conducted to answer the following questions: (1) how robust is AdvEx-RL compared to the baselines under deliberate uncertainty in form of external perturbations and altered environment dynamics? (2) how does AdvEx-RL's safety policy affect the agent's task performance? (3) how much does the safety policy contribute to the safety of AdvEx-RL? (ablation analysis) (4) how often does the deadlock occur in AdvEx-RL and the baselines? and (5) how transparent and interpretable is the behavior generated by AdvEx-RL to the end users? All the codes 2 relevant to the experiments are available online.

6.1 Environments

We conducted our experiments on three continuous MuJoCo CMDPs [Thananjeyan et al., 2021] (i) Maze (ii) Navigation 1, and (iii) Navigation 2. In these environments, the agent's task is to reach the goal state while avoiding collisions with obstacles, walls, or boundaries. In addition, we also conducted experiments on SafetyGym environments [Ray et al., 2019]. (See Appendix E for more details about the environments.)

6.2 Baselines

We have tested AdvEx-RL against 7 baselines; SAC (without any safety measures), Lagrangian Relaxation (LR) [Thananjeyan et al., 2020], Safety Q-Functions for RL (SQRL) [Srinivasan et al., 2020], Risk Sensitive Policy Optimization (RSPO) [Mihatsch and Neuneier, 2002], Critic Penalty Reward Constrained Policy Optimization (RCPO) [Tessler et al., 2018], Reward Penalty (RP) [Thananjeyan et al., 2021], and Recovery RL Model Free (RRL-MF) [Thananjeyan et al., 2021]. (More details on the baselines are in Appendix F. In addition, see Appendix G for further implementation details of AdvEx-RL and the baselines.)

6.3 Performance Metrics

Assuming a maximum episode length Tmax in any environment, the following cases might happen: (1) the agent accomplishes its task within Tmax without any safety violations, (2) it violates at least one safety constraint and terminates, or (3) it exhausts Tmax without accomplishing its task or violating any safety constraints. Considering these cases, we use the following two performance metrics in our experiments: (i) Safety(%): This metric measures the portion of time the agent acts safely over its maximum episode length. Given Tmax; the function F(.) counts the total number of timesteps before the episode termination caused by safety violation. Then Safety% is measured as a function of trajectory

2 https://github.com/asifurrahman1/AdvEx-RL

τ :

(ii) Success-Safety(%): Assuming the AdvEx-RL agent's task is to reach a goal state G, both the reward and success are measured in terms of how close the agent is to G. The agent is considered successful in accomplishing its task when the episode ends while it is within a predefined minimum distance from G Mindistance. If the maximum distance from G is given by Maxdistance and the agent's current Euclidean distance from the goal is D(st, G), then the Success-Safety % measures the trade-off between safety and task objectives by:

6.4 Robustness Analysis

While prior safe RL works focus on the performance optimality, we argue that optimality is not enough and we have to study the robustness of the safety solutions against deliberately crafted perturbations. Therefore, we evaluate the robustness of the safety performance of the baselines and AdvExRL under different magnitudes and types of uncertainty. We injected two types of uncertainty into the testing environments; external action space perturbations and changes to the environment dynamics (such as air resistance and noise). For the first type, we kept the environment dynamics the same as the training environment while incorporating two white-box action space perturbations; random and alternative adversarial action (AAA) perturbations from [Tessler et al., 2019]. (See Appendix H for details.)

6.5 Results

Evaluation of Safety Robustness: The results of the safety robustness analysis are presented in Fig.2 and Fig.3. The results are averaged over 100 test episodes. Evidently, the baselines showed optimal safety performance under no external perturbations or dynamics changes. However, a significant deterioration of both safety(%) and success-safety(%) can be seen for most of the baselines with increasing in perturbation rates. Performance deterioration is more visible in the case of the AAA perturbation than random perturbation, which indicates that the prior safe RL techniques are not robust against deliberate perturbations. Moreover, changes in the environment dynamics negatively impact the safety robustness of the baselines as well as AdvEx-RL. Interestingly, in Fig.2, we observe that SAC provides reasonable safety which is due to its default robustness properties [Eysenbach and Levine, 2021]. RRL-MF provides the second-best safety robustness, particularly for higher perturbation rates and the reason for this could be its usage of explicit unsafe demonstration data provided by experts during its training. In both uncertainty types, AdvEx-RL clearly outperforms the baselines with a minimum safety(%) over 75% and success-safety(%) over 80%. This empirically proves the robustness of AdvEx-RL

against perturbations and demonstrates its generalizability to unseen scenarios. (Please refer to Appendix I, to see results in SafetyGym environments. Also the switching between task policy and safety policy is provided in Appendix J.)

Ablation analysis: AdvEx-RL, has three components that contribute to its safety performance i.e (1) the task policy, (2) the safety policy, and (3) the shield. To evaluate the contribution of the safety policy to the overall performance, we conducted a thorough ablation study , the details of which can be found in Appendix L.

Deadlock analysis: We analyzed the presence of deadlock in the baselines while subjected to different rates of AAA

perturbation (ranging from 0% to 100%). Using a 20-step look-ahead for deadlock cycle detection across 100,000 test episodes, we found no deadlock cycles in the baselines and AdvEx-RL in all three environments. (See code supplements for deadlock experiment details).

Interpretability Analysis: Since safety-assurance approaches, in general, compromise system performance, we must ensure that human practitioners and users trust them, lest they ignore them and negate their effectiveness. Trustworthiness regarding how safe an agent is depends on how transparent its behavior is to the end users. For the joint task-safety optimization techniques [Srinivasan et al., 2020;

Tessler et al., 2018; Liu et al., 2022], it is not possible to pinpoint what influences the agent's behavior, task, or safety objective. AdvEx-RL, however, uses two separate policies for the task and safety and is, therefore, capable of explaining what influences its behavior at each time step. To analyze the interpretability of AdvEx-RL, we extend the explainable RL method CAPS [McCalmon et al., 2022] into safety-CAPS to explain the impact of safety violations on the agent's behavior and to provide a directed graph with an explanation of the agent's safety policy by extending CAPS with two criteria; risk estimation and the episode length in time steps. We use the safety-CAPS graphs to answer the question of "why the agent is taking certain actions at certain states?". We then add more details to the CAPS graph about how, when, and why the safety policy and safety shield are triggered during safety violations. Episode length (TS) measures the time steps the agent takes to accomplish its task. Perturbing the agent's policy during safety violations could result in the agent taking a longer time to reach its goal state. Hence, we use this metric to answer the question of "why the agent is not taking action a′ instead of a at state sc?" by attaching the timesteps to each abstract state of the CAPS graphs. The risk estimation (RE), on the other hand, measures how likely the agent will fail its task when it takes action a′ instead of following its policy and taking a at a critical state sc due to a safety violation. To estimate the risk, we build on the risk estimation approach proposed in [Uesato et al., 2018] for uncovering failures in RL agents by sampling the agent in states where its safety is violated. We use a neural network with two fully-connected layers of 64 neurons each, and train it to predict RE from each state st. An example of the safety-CAPS graph for Navigation 2 is displayed in Fig.4.

We conducted a user study using the Navigation 2 environment and presented 41 Amazon Mechanical Turk (AMT) workers with the safety-CAPS graphs and asked them 8 questions (See Appendix M for the study and the details of the questions) about their interpretability of AdvEX-RL policies. A summary of the accuracy of the users' answers to each question is shown in Figure.5.

In questions 1,2,5, we asked the users to identify optimal actions the agent will take at a certain state with its task policy (Q1), task policy under attack (Q2), and task policy under attack but with AdvEx-RL's safety policy (Q3). The participants demonstrated a good understanding of the environment, with an accuracy rate above 80% in those questions. Notably, the accuracy rate of Q5 is above 90%, which indicates that the users can understand the purpose and impact of the safety policy on the agent's behavior. This clearly shows the interpretability of AdvEx-RL for end-users regardless of their RL background.

Q3 and Q6 are true or false questions where we asked the users to identify if the agent will terminate in a dangerous state under an attack without (Q3) or with (Q6) the safety policy. Roughly 70% and 85% of the users correctly answered Q3 and Q6, respectively. The increase in the accuracy of Q6 demonstrates that the safety-CAPS graph with the safety policy can better convey the reason behind the agent's actions. Q4 and Q7 are counterfactual reasoning type questions where we asked the users why the agent terminated in dangerous states under attack (Q4) but successfully avoided the dangerous states using the safety policy (Q7). We think the low accuracy for those questions is due to their difficulty and demand in terms of logic and analytical skills, which can be challenging to non-technical participants who do not have RL background. Lastly, Q8 is a comprehensive measure of the users' understanding of the overall impact of the safety policy on the agent's behavior. Approximately 73% of the users successfully understood how the safety policy protects the agent from choosing unsafe actions. At the same time, we also need to consider the fact that the length of the study and the dependent relationship of questions generally pose challenges to the participants. If they fail to understand the central idea of the study, they tend to perform badly subsequently. All the participants who incorrectly answered Q8 have at least answered 2 or 3 questions incorrectly before. Therefore, we believe our AdvEx-RL is interpretable by nontechnical users using the graphs generated by safety-CAPS.

7 Conclusion

In this paper, we introduced an alternative view on safety learning for RL through our task-agnostic safety framework AdvEx-RL. We empirically showed that AdvEx-RL is effective in ensuring safety even in uncertain conditions. Through a user study conducted on 41 non-technical end users, we also demonstrated the transparency of AdvEx-RL by explaining its behavior using safety-CAPS. In future work, we plan to extend this framework to multi-agent settings along with an explainable safety method.

Acknowledgments

This material is based upon work supported by the National Science Foundation (NSF) under grant no. 2105007.