PNAct: Crafting Backdoor Attacks in Safe Reinforcement Learning

1 Introduction

Reinforcement learning (RL), as an important branch of artificial intelligence, is primarily focused on training an agent to develop an optimal policy that maximizes task gains through interactions with the environment. The metric used to evaluate task gains at every time step is the reward signal provided by the environment during the interaction process [Shi et al., 2023; Chen et al., 2024; Yang et al., 2023]. The greater the reward, the better the agent's action at that time step. However, the optimal policy in RL is often aggressive and may select actions with potential safety risks when making decisions, failing to ensure that the agent adheres to safety constraints. Therefore, existing research has proposed Safe Reinforcement Learning (Safe RL) based on traditional RL [Heger, 1994]. Safe RL introduces an auxiliary cost signal during the training process to measure the policy's compliance with safety constraints. In this context, the agent needs to simultaneously maximize cumulative rewards and control cumulative costs, ensuring task completion while complying with safety constraints. Existing solution methods for Safe RL include primal-dual-based methods [Bhatnagar and Lakshmanan, 2012; Fujimoto et al., 2019], uncertainty-aware methods [Fujimoto et al., 2019; Chen et al., 2021], and various other approaches [Achiam et al., 2017; Liu et al., 2022; Zhang et al., 2020; Yang et al., 2020].

However, beyond environmental safety constraints, RL faces other security challenges, such as adversarial attacks and backdoor attacks. While these issues are serious and deserve attention, research in this area remains limited. Existing studies [Liu et al., 2023a; Liu et al., 2023b] have shown that Safe RL can be vulnerable to adversarial attacks, which can lead to excessively high violation costs for safety constraints. These studies have also proposed corresponding defense methods. In contrast, backdoor attacks present a more challenging defense problem than adversarial attacks. Backdoor attacks are inherently more covert, as they involve injecting specific trigger conditions during training. Once triggered, these conditions cause the agent to take malicious or unsafe actions while the agent appears to behave normally under regular circumstances. Due to their stealthy, backdoor attacks are more difficult to defend against. If effectively implemented on Safe RL, backdoor attacks could significantly compromise the agent's decision-making safety and greatly increase the difficulty of implementing effective defenses. A recent study [Jiang et al., 2024] proposed a safety reinforcement learning backdoor based on Signal Temporal Logic (STL), demonstrating the vulnerability of Safe RL to such attacks. To the best of our knowledge, this is the first exploration of backdoor attacks in the context of Safe RL. However, this approach has certain limitations. The proposed backdoors require integrating logic parsers with RL frameworks and using different specifications tailored to specific scenarios, which limits their general applicability. To address this research gap, this paper makes the following contributions:

• We define the concepts related to backdoor attacks in Safe RL and propose relevant metrics to evaluate the effectiveness and stealthiness of these attacks.

• We propose a backdoor attack framework for Safe RL, named PNAct, which leverages positive and negative action samples along with a loss function modification technique. This framework enables the agent to take unsafe and risky actions when the environment reaches specific states.
• We apply the proposed attack framework to safetyconstrained environments and evaluate its performance using the backdoor metrics for Safe RL, demonstrating the effectiveness of our approach.

Unlike previous approaches, our attack framework does not affect the rewards obtained by the agent, making it less likely to be detected. However, it increases the cost of violating safety constraints, leading to riskier decision-making by the agent. Moreover, it can be applied to a variety of environments without requiring additional scenario-specific designs.

2 RelatedWork

2.1 Safe RL

Safe RL focuses on maximizing rewards while controlling costs. A popular method is the primal-dual approach, which introduces a Lagrange multiplier to penalize constraint violations, unifying reward and cost optimization. This approach serves as the foundation for our backdoor attack framework. Early studies manually selected the Lagrange multiplier [Borkar, 2005; Bhatnagar and Lakshmanan, 2012; As et al., 2022], but this often led to suboptimal policies. To improve, training can dynamically update policy parameters and the multiplier to balance rewards and costs [Chow et al., 2018; Liang et al., 2018]. Additionally, PID control [Stooke et al., 2020] helps stabilize training and reduce oscillations caused by phase shifts between the constraint function and multiplier.

2.2 Backdoor Attacks in RL

Backdoor attacks in RL resemble those in neural network classification, where an agent is manipulated to perform attacker-specified actions in certain states while behaving normally in other states. Most attacks decrease rewards. TrojDRL [Kiourti et al., 2020], uses poisoning methods and provides various strategies based on attack strength and target specificity. Building on TrojDRL, specialized methods have emerged: BACKDOORL [Wang et al., 2021] targets adversarial RL by triggering backdoors via adversarial action sequences; BadRL [Cui et al., 2024] employs sparse attacks to maximize success with minimal steps; MARLNet [Chen et al., 2023b] focuses on multi-agent scenarios using imperceptible triggers and reward manipulation to degrade performance. These attacks prioritize reward reduction without addressing safety constraint violations.

Recent research introduces the first backdoor attack in Safe RL using STL [Jiang et al., 2024], but its effectiveness is limited in complex, high-dimensional, or stochastic environments due to challenges in defining and verifying STL specifications and high computational costs. In contrast, our proposed method generalizes better across diverse environments.

3 Preliminary

3.1 ConstrainedMarkov Decision Process (CMDP)

The traditional Markov Decision Process (MDP) in reinforcement learning is a tuple that only considers the reward signal, whereas the CMDP [Altman, 1998] defines a cost signal. It is represented as ⟨S, A, P, R, C, ρ, γ, ⟩, where S is the set of all possible states in the environment, and A is the set of actions the agent can take. P(st+1|st, at) : S ×A×S → [0, 1] is the probability that, at time step t, the environment transitions from state st ∈ S to state st+1 ∈ S after the agent takes action at ∈ A. The reward at time step t is calculated using rt = R(st, at) : S × A → R, while the cost is calculated using ct = C(st, at) : S ×A → R≥0. The notation ρ(s0) : S → [0, 1] represents the probability distribution of the initial state. γ ∈ [0, 1] is the discount factor used to calculate both cumulative reward and cost. This paper uses the same discount factor for cumulative reward and cost calculations for convenience and to ensure stationary optimality.

CMDP provides a mathematical model for solving Safe RL. The objective is to find a policy π for the agent that satisfies the maximization of the reward value function:

At the same time, it also needs to ensure that the expected cumulative cost remains within a predefined range, i.e.,

Here, κ is a safety constraint threshold manually set. It is considered safe if the cost value function of policy π satisfies the above equation.

3.2 Threat Model

In this paper, we consider the following scenario: From the user's perspective, an RL policy is needed to guide the agent's behavior. In this case, the user may outsource training to a relevant party or download a pre-trained model from a thirdparty platform, deploying it directly in the user's environment or fine-tuning it before use. From the attacker's perspective, the attacker can implant a backdoor into the model during the training process and then provide it to the user. The backdoor can be activated during the model's decision-making process by altering the model's input state or modifying the environment.

3.3 Concepts and Notations

Elements of Backdoor Attacks

According to the definition of backdoor attacks, the agent is required to output actions desired by the adversary in specific states. These specific states are also designated by the adversary, often by adding anomalous elements to the state, such as inserting a colored patch in an image, a red sports car in a video, or a fixed sequence in text. Anomalous elements like the colored patch, sports car, or fixed sequence are referred to as triggers, denoted by δ. The specific state called backdoor state designated by the adversary is composed of the normal state and the trigger δ.

Policies

In this paper, we define the set of all policies for the agent as Π, which can be expressed as Π = Πs ∪ Πu ∪ Πf . Here, Πs represents the set of safe policies, where any policy πs ∈ Πs does not violate the task's safety constraints. Πu denotes the set of unsafe policies, where any policy πu ∈ Πu fails to comply with the safety constraints. Lastly, Πf is the set of failure policies, where any policy πf ∈ Πf is unable to accomplish the task. The optimal safe policy in Πs, which maximizes rewards while controlling costs, is denoted as π∗ s . Correspondingly, the optimal unsafe policy in Πu, which solely maximizes rewards, is denoted as π∗u . Figure 1 presents a simple case of a car reaching the goal, illustrating the relationship between the policy set and optimal policies.

The meaning of the two optimal policies in terms of the value function is primarily represented by the following two equations:

The relationship between the policy set and the optimal policy based on the value function is shown in Figure 2. In this paper, the backdoor policy is denoted as˚π, and its properties will be explained in detail in the following sections.

4 Method

4.1 Backdoored ConstrainedMarkov Decision Process (B-CMDP)

This paper introduces a new Safe RL-based mathematical model, the Backdoored Constrained Markov Decision Process (B-CMDP), built upon the CMDP framework. The mathematical definition of B-CMDP is given as ⟨S,X, ˚ X,A,P,R, C, ρ, γ⟩. Compared to CMDP, B-CMDP additionally considers sets of normal states X and backdoor states ˚ X. Obviously, S = X ∪ ˚ X. At time step t, the normal state or backdoor state are represented as xt ∈ X and ˚xt ∈ ˚ X, respectively. Since triggers represent redundant information within states, they do not affect the state transition probabilities, i.e., P(st+1|˚xt, at) = P(st+1|xt + δ, at) = P(st+1|xt, at).

In the scenario described in this paper, the backdoor trigger can be considered absent in normal states. Thus, we have:

Therefore, it can be derived that

Here, I(·) represents the indicator function, which is used for truth evaluation. If the condition inside · is true, the function value is 1; otherwise, it is 0.

In this paper, the backdoor policy˚π selects unsafe actions in the backdoor state ˚x and safe actions in the normal state x. Thus, we derive the following property:

Property 1 (Action-wise). From the perspective of action sampling, the backdoor policy executes the optimal unsafe action in states with the trigger while hiding the backdoor and executing the optimal safe action in normal states, i.e.,

The above formula represents the choice between two optimal policies made by the backdoor policy˚π at any given time step t.

In addition to representing the properties of the backdoor policy from the action sampling perspective, we can also derive its property from the value function perspective:

Property 2 (Value function-wise). From the perspective of the value function, the backdoor policy needs to violate the safety constraints in episodes where the backdoor is triggered and adhere to the safety constraints in episodes without the backdoor. Simultaneously, its reward value function also needs to attain optimality. Its value functions can be expressed as:

In the above property, τ = (s1, s2, . . . , sT ) represents a state sequence of an episode with T time steps, and ˜ τ is the state sequence containing backdoor states, with the number of backdoor states in the range (0, T ].

4.2 Metric Designing

The previous section has explained the basic properties of the backdoor policy ˚π. However, more detailed metrics are required to evaluate the backdoor policy's performance. We evaluate the backdoor policy from two aspects: effectiveness and stealthiness.

Definition 1 (Effectiveness). The effectiveness metric can be represented by a logical expression, specifically I˚πE = I ((V ˚π c (ρ, ˜ τ ) > V ˚π c (ρ, τ)) ∧ (V ˚π c (ρ, ˜ τ) > κ)). If I˚πE = 1, it indicates that the backdoor policy˚π satisfies effectiveness; if I˚πE = 0, then it does not satisfy effectiveness.

The effectiveness of the backdoor policy˚π is primarily reflected in two aspects: the comparison of cost value functions with and without backdoor states, and whether the safety constraints are violated in the presence of backdoor states.

Definition 2 (Stealthiness). The stealthiness metric can be represented by a logical expression, specifically I˚π S = I ((V ˚π r (ρ, ˜ τ ) ≥ V ˚π r (ρ, τ)) ∧ (V ˚π c (ρ, τ) ≤ κ)). If I˚π S = 1, it indicates that the backdoor policy ˚π satisfies stealthiness; if I˚π S = 0, then it does not satisfy stealthiness.

The stealthiness of the backdoor policy ˚π is primarily reflected in two aspects: the comparison of reward value functions with and without backdoor states, and whether the safety constraints are satisfied in the absence of backdoor states.

4.3 PNAct: Safe RL Backdoor Attack Framework

Based on the properties of the backdoor attack in Safe RL proposed earlier, the backdoor policy˚π is a fusion of the optimal safe policy π∗ s and the optimal unsafe policy π∗u . From the perspective of policy distribution, it resembles π∗u in backdoor states and π∗ s in normal states. On this basis, we need to find the optimal backdoor policy ˚π∗ that satisfies the conditions and maximizes the similarity.

Remark 1. The optimal backdoor policy˚π∗ satisfies the following conditions in terms of distribution:

where DKL is the Kullback-Leibler divergence between two policies.

We aim to obtain a Safe RL backdoor policy that satisfies the fundamental properties of backdoor attacks while meeting the effectiveness and stealthiness criteria. Based on this objective, we design an attack framework called PNAct, which leverages positive and negative action samples. This framework switches between positive and negative action samples based on the state, enabling the backdoor policy learned by the victim agent to better differentiate between safe and unsafe policies.

Attack Control

During the training process, we need to control the frequency and duration of attacks to adjust the ratio of backdoor and normal states, as well as positive and negative samples. To achieve this, we define an attack signal ψt = u(f, k, t), where f is the interval of backdoor injection during training (the number of time steps between two consecutive attacks ), and k is the duration (in time steps) of each injection. Their values can be adjusted based on the training scenario and the specific needs of the adversary. If ψt = 1, it indicates that a trigger needs to be added to the state at time step t; if ψt = 0, it indicates that no adjustment is required for the normal state at time t, i.e.,

We can systematically generate both backdoor and nonbackdoor samples through the attack signal generator u.

Loss Design

PNAct samples actions from π∗ s and π∗u based on different states in the training process, setting positive and negative action samples accordingly. The backdoor policy is trained to fit the two strategies based on the state conditions. Specifically, when in a backdoor state, positive action samples are obtained from π∗u , and negative action samples are obtained from π∗ s ; when in a normal state, the opposite applies, i.e.,

Considering that the choice of state depends on the attack signal generator, it follows that

Remark 2. Since the optimal safe and unsafe policies are unknown, we substitute them with a policy estimation method, namely using reinforcement learning-based approaches to obtain the fitted policy.

To satisfy Equation (9), we train the backdoor policy such that the action generated by the backdoor policy˚π is as close as possible to a+t and as far as possible froma-t . If the actions are continuous, the loss function for the action sample part during the PNAct training process is:

If the actions are discrete, the corresponding loss function can be described as:

λ represents the weighting factor balancing the effects of positive and negative action samples. In Equation (13), J can be a loss function for regression tasks, such as the mean squared error (MSE) loss function. In Equation (14), J can be a loss function used for multi-class classification, such as crossentropy loss, etc. In addition, the positive and negative action samples are in the form of one-hot encoding.

For the purpose of measuring the reward and cost value functions, PNAct references Lagrangian-based methods and adds two critics. They are both neural networks, and their loss functions for updating are as follows:

where Lr and Lc are the loss functions for updating the reward and cost value networks, respectively, with θr and θc being their corresponding parameters.

The update of the PNAct framework relies on minimizing the following loss function:

where α, β and µ are the weight coefficients that balance the importance of the three losses. The benefit of using a threepart loss function is that this approach ensures the backdoor policy satisfies the basic properties while also performing the functions of the reward and cost value functions. It allows the policy to meet the actor-critic structure and improves its stealthiness.

Training Process

Based on the method above, we outline the attack process of PNAct during the training procedure. The specific implementation approach is shown in Algorithm 1.

5 Experiments

To verify whether PNAct satisfies the safety reinforcement learning backdoor attack requirements, we apply it to the Safety-Gymnasium environment [Ji et al., 2023], which has strict safety constraints. The agent must reach the goal while avoiding hazards, with obstacles like vases blocking its view and movement.

5.1 Configuration

Environments and Trigger Design

We use Car, Point, and Ant as agents to complete the goalreaching task; the corresponding environments are called CarGoal, PointGoal, and AntGoal. All environments contain 8 hazards, 5 vases, and 1 goal. The agent's task is to avoid the hazards and reach the goal location.

To implement the backdoor injection, we use the absolute positions of the vases as triggers for the backdoor attack. This type of trigger is similar to color blocks, but the advantage is that it is not limited to image-based inputs. Moreover, during the agent's decision-making, only the positions of the

Algorithm 1: Training process of PNAct

Input: Optimal safe policy π∗ s ; optimal unsafe policy π∗u ; initialized backdoor policy˚π; initialized reward value function Vθr ; initialized cost value function Vθc ; trigger δ; initial state s0 ∼ ρ; attack signal generator u; attack interval f; attack duration k 1 for t=0,1,2,... do 2 Obtain the normal state xt and the backdoor state ˚xt = xt + δ according to st 3 Obtain the attack signal ψt = u(f, k, t) 4 if ψt then 5 st ←˚xt 6 end 7 else 8 st ← xt 9 end 10 Sample safe action as,t ∼ π∗ s (·|st) 11 Sample unsafe action au,t ∼ π∗u (·|st) 12 Agent takes action at ∼˚π(·|st) 13 if ψt then 14 a+t ← au,t, a-t ← as,t 15 end 16 else 17 a+t ← as,t, a-t ← au,t 18 end 19 The environment proceeds to st+1 and produce rt, ct 20 Agent adds (st, a+t , a-t , rt, ct, st+1) to its buffer 21 Calculate Lact, Lr and Lc according to Equation (13) or (14), and Equation (15) 22 Update˚π, Vθr and Vθc according to Equation (16) 23 end

objects in the environment need to be changed without directly interfering with the agent's state input. In the normal state, the positions of vases are random, while specific positions are selected as backdoor states. In the experiment, we use the positions of five vases, with their trigger positions being {(0, 0), (1, 0), (0, 1), (-1, 0), (0, -1)}. Taking the PointGoal environment as an example, we illustrate the agent's performance in both normal and backdoor states in Figure 3.

Training Configuration

During the training process, we set the attack interval of the attack signal generator as f = n × |τ|, where n ∈ Z>0. At the same time, we set the attack duration as k = |τ |, meaning that after several normal episodes, a backdoor is implanted in one episode.

Base Model Configuration

The base models generate the optimal policies in this paper. The base model for the optimal unsafe policy is trained using the classical reinforcement learning algorithm PPO, which only considers reward maximization without taking safety constraints into account. We select multiple safe reinforcement learning algorithms, namely PPO-Lag(rangian), TRPOLag(rangian), and RCPO [Tessler et al., 2019], to train the base models for optimal safe policies. These models are combined with the optimal unsafe policy to form the sampled policies for the optimal backdoor policy.

5.2 Results and Analysis

The experimental results of different policies in completing the task are shown in Table 1. Each experiment shows the average reward, cost, and corresponding variance of PNAct over 100 complete episodes with n ∈ {5, 10, 15, 20, 25}. We analyze the performance of PNAct from the following aspects.

Distinguishability

The results show that PNAct can generally meet the requirements for effectiveness and stealthiness in most cases.

Using the PPO+RCPO combination and the PointGoal task with n = 10 as an example, we sample the cumulative rewards and costs from 1000 rounds of the PNAct model in both backdoor and normal states. We then analyze the distribution of the corresponding rewards and costs to evaluate the distinguishability between the two states. The distribution plots are shown in Figure 4.

It can be intuitively seen from the figure that the backdoor model we train has data points mainly concentrated in the region where the cumulative cost is below the cost constraint value in the normal state, while the opposite is true in the backdoor state. At the same time, the cumulative reward of the backdoor model in the backdoor state is slightly higher

(a) Normal condition

(b) Backdoor condition

Figure 5: The trends in return and cost under normal and backdoor conditions are obtained from models trained with different attack signal intervals for AntGoal environment. (a) and (b) represent the trends in return and cost under normal and backdoor conditions, respectively. The dashed lines y = 25 represent the cost constraint.

than that in the normal state. This shows that our PNAct attack framework balances two metrics and exhibits a significant distinction in scenarios with and without triggers.

Attack Signal Generation Interval

To explore the impact of the attack signal generation interval on the models trained by the PNAct attack framework, we use line charts to show the trends in cumulative reward and cumulative cost under different values of n with n ∈ {5, 10, 15, 20, 25, 50, 75, 100}. Due to space limitations, we take the AntGoal environment as an example, with the corresponding results shown in Figure 5. Line charts for other environments are presented in the appendix.

Regardless of whether in the backdoor state or the normal state, for models trained using the PNAct framework, as n increases, both the cumulative reward and cumulative cost decrease, which may even affect the effectiveness of the backdoor attack. This result is related to the change in the proportion of samples affected by the attack signal. As n increases, the proportion of optimal unsafe policies decreases, and the samples are dominated by those generated by the optimal safe policy. As a result, the trained backdoor policy tends to make decisions in a safer and more conservative direction. Therefore, to ensure the performance of the backdoor policy, it is necessary to adjust the attack signal generation frequency.

Base Models

To compare the performance of PNAct with that of the baseline model, we present the cumulative reward and cost of the baseline model in both the backdoor and normal states, as shown in Table 2. Clearly, there are significant differences in performance across different Safe RL policies.

From the comparison, two phenomena can be observed. The first is that the reward of the PNAct backdoor policy tends to be similar to that of the corresponding base model's policy, but the cost is often significantly lower. The second is that while certain base models' safe policies violate the safety constraints, the PNAct experimental results indicate that the backdoor policy, even when trained with these unsafe policies, can still adhere to the safety constraints under normal conditions. These two phenomena may be due to the negative action sample terms in equations (13) and (14), which further constrain the cost. Additionally, due to the presence of the safety constraints, the backdoor policy tends to be more conservative in the normal state, leading to a lower reward. Therefore, ideal performance backdoor policies can be achieved by adjusting the weight factor λ in the loss function.

6 Discussion

6.1 Backdoor Attack vs. Adversarial Attack

The main differences between adversarial attacks [Zhou et al., 2024a; Zhou et al., 2024b; Guo et al., 2024; Guo et al., 2025] and backdoor attacks are reflected in the following two aspects:

• Stage of model access. Adversarial attacks can be carried out during the model inference stage by modifying the agent's input state to achieve the attack's goal. In contrast, backdoor attacks are implanted during the training stage to inject a backdoor into the model. During model inference, backdoor attacks can alter the agent's input state or even directly modify the environment without needing to interact with the model itself.
• Attack pattern. Adversarial attacks require specific methods to design perturbations, such as gradient-based optimization techniques (e.g., FGSM [Goodfellow et al., 2015], PGD [Madry et al., 2018]), to achieve significant attack effects. Random or simple perturbation patterns have limited effectiveness. On the other hand, backdoor attacks only need to apply simple attack patterns, such as color blocks or regular sequences, within the input state to achieve their objective.

Clearly, our attack method involves implanting a backdoor during the training process and altering the environment during the inference process, so the approach we adopt falls under the category of backdoor attacks.

6.2 Defense

Existing studies propose defenses against reward-reduction backdoor attacks, but none address secure backdoor attacks with increased costs. BIRD [Chen et al., 2023a] uses RL to detect triggers maximizing the value function, effective for poisoning-based attacks but inapplicable to our non-poisoning approach. PolicyCleanse [Guo et al., 2023] defends against adversarial RL backdoor attacks, but our method does not involve adversarial scenarios.

7 Conclusion

This paper introduces the concept of backdoor attacks in Safe RL, proposes corresponding evaluation metrics, establishes the mathematical model B-CMDP, and designs the universal backdoor attack framework for Safe RL, PNAct. Our attack framework has been applied to multiple Safe RL scenarios in Safety-Gymnasium, demonstrating significant experimental results under our evaluation metrics. In future work, we will conduct more in-depth research on the existing limitations of PNAct and propose improved solutions for backdoor attacks in Safe RL.

Ethical Statement

Our attack framework targeting safe RL agents may cause some ethical concerns. First, we clarify that attacks are not our goal. Instead, our research has two main goals: firstly, to identify potential risks in safe RL to avoid any real-world consequences that could arise from these vulnerabilities; and secondly, to lay the groundwork for future defense research in this area. While our current focus is on risk identification, we intend to focus on developing and implementing effective defense strategies in our future work.

Acknowledgments

This work was supported in part by the Space Optoelectronic Measurement and Perception Lab., Beijing Institute of Control Engineering under Grant LabSOMP-2023-03, in part by the National Nature Science Foundation of China under Grant 62172299 and Grant 62032019, and in part by the Fundamental Research Funds for the Central Universities under Grant 2023-4-YB-05. (Corresponding authors: Guanjun Liu)