BehaviorGuard: Online Backdoor Defense for Deep Reinforcement Learning

1 Introduction

Deep reinforcement learning (DRL) has shown strong performance in complex sequential decision-making tasks and is starting to be used in safety-critical domains such as games [Vinyals et al., 2019], autonomous driving [Wang et al., 2021b], and robotics [Tang et al., 2025]. As DRL moves into such settings, security and safety become central concerns [Gu et al., 2024]. Recent studies [Kiourti et al., 2019; Wang et al., 2021a] have revealed that DRL policies are highly vulnerable to backdoor attacks, where an adversary implants a hidden backdoor during policy training that causes the agent to behave maliciously once the backdoor is activated, while maintaining benign performance otherwise. Compared to backdoors in supervised learning [Gu et al., 2017;

∗Corresponding Author

Liu et al., 2018], backdoors in DRL are particularly dangerous due to the sequential nature of decision making and complex environment dynamics, which can amplify small behavior shifts into catastrophic outcomes.

This threat is further amplified in multi-agent reinforcement learning (MARL) settings. In competitive environments, an adversary may trigger a victim's backdoor purely through its own actions or interaction patterns [Wang et al., 2021a], whereas in cooperative environments, poisoning a single agent can compromise the entire team through coordination failure and cascading effects [Chen et al., 2022]. The inherent non-stationarity and inter-agent coupling in MARL significantly increase the difficulty of detecting and mitigating backdoors, making defenses developed for single-agent settings unreliable when transferred across environments.

Existing defenses against DRL backdoors primarily rely on reward anomalies, trigger reconstruction, or prior knowledge of trigger patterns [Guo et al., 2023; Chen et al., 2023; Yuan et al., 2024]. Although these methods show high defensive capabilities, they still face several challenges: (1) reward signals may appear normal when the backdoor is active [Rathbun et al., 2025], resulting in a high detection false-negative rate; (2) pattern-dependent methods fail against triggers based on sequential behaviors with spatial or temporal dependence [Yu et al., 2024a]; (3) most methods lack a unified treatment that generalizes across both single- and multi-agent scenarios; and (4) many methods require expensive policy retraining to mitigate backdoors.

To address these challenges, in this work, we shift our defense focus from recognizing diverse and attack-specific trigger patterns targeted by prior work to identifying the unified backdoor output manifestation. Specifically, we analyze the output behavioral differences between clean and backdoor DRL policies. Our key observation is that, regardless of trigger patterns or environments, backdoored policies inevitably induce subtle but persistent behavioral drift in the action distribution. Even under trigger-free inputs, the policy must bias its action probabilities to ensure successful activation, leaving detectable signatures in high quantiles and tails of the distribution. This insight enables trigger-agnostic backdoor defense without relying on reward signals or trigger priors.

Building on this finding, we propose BehaviorGuard 1 , a unified framework that jointly detects and mitigates backdoors online for different DRL scenarios. We first design a backdoor detection method based on the signature of behavioral drift, which quantifies action-distribution deviations using the Behavioral Drift Score (BDS) and density-based statistics. Second, we propose a lightweight, retrainingfree, drift-constrained backdoor mitigation strategy to mitigate backdoor actions online at runtime. Once backdoor outputs are detected, BehaviorGuard suppresses backdoor behaviors by projecting the abnormal action distribution back to a low-drift region, achieving practical protection without retraining. To the best of our knowledge, this is the first work that can detect and mitigate backdoor attacks against singleand multi-agent DRL at runtime.

1 https://github.com/c0d818/BehaviorGuard

Our contributions are summarized as follows:

• We identify behavioral drift as an attack- and environment-agnostic Trojan signature.
• We propose a unified backdoor detection framework for single- and multi-agent environments.
• We introduce a lightweight drift-constrained mitigation method that suppresses backdoor behaviors at inference time without retraining, enabling online defense.
• Extensive experiments across single-, cooperative, and competitive multi-agent benchmarks demonstrate that BehaviorGuard achieves powerful detection performance while effectively mitigating backdoor impact with minimal degradation on clean performance.

2 Background and RelatedWork

2.1 Deep Reinforcement Learning

Single-agent RL (SARL) is commonly modeled as an MDP M = (S, A, P, r, γ), where S and A denote the state and action spaces, P(s′ |s, a) is the transition kernel, r(s, a) is the immediate reward, and γ ∈ (0, 1) is the discount factor. At time t, the agent observes a state (or observation) xt and samples an action at ∼ πθ(·|xt); the environment then transitions to st+1 and returns reward rt. The policy πθ is typically parameterized by DNN, and is trained to maximize the expected discounted return Eπθ [∑∞t=0 γ t r(st, at)].

Beyond the single-agent setting, we also consider multiagent RL (MARL) [Busoniu et al., 2008]. In competitive MARL (com-MARL), agents' rewards are competitive or conflicting, and the environment evolves under the joint action at = (a1t , . . . , ant ) [Lowe et al., 2017]. In cooperative MARL (coo-MARL), agents share a team objective (e.g., a shared team reward), where coordination among policies is essential [Rashid et al., 2020; Foerster et al., 2018].

2.2 Backdoor Attack Formalization in DRL

Backdoor attacks aim to implant a hidden malicious behavior into a policy network such that it behaves similarly to a clean policy under normal inputs, while reliably switching to an attacker-specified behavior when a trigger is present. Formally, let δ denote the trigger and fδ(·) be the trigger injection function that applies the trigger pattern to an input: xδt = fδ(xt). Backdoor triggers can be classified into two types: instant and sequential triggers. The former can be instantiated as single-step state perturbations (e.g., visual patches [Kiourti et al., 2019] or specific states [Wang et al., 2021b]). The latter can be temporal state patterns [Yu et al., 2022] or interaction-based conditions among agents [Wang et al., 2021a; Fang et al., 2025] spanning multiple steps. The attacker obtains a backdoored policy πbd via poisoning the training process such that πbd mimics a clean policy πcl on trigger-free inputs but shifts toward a target behavior πtar on triggered inputs:

In SARL, this switch often manifests as a sharp return drop or persistent target actions; in com-MARL, the attack may reduce the victim's win rate or induce systematic failures in key adversarial states; in coo-MARL, deviations of one agent can further disrupt coordination and degrade team performance.

2.3 Backdoor Attacks in DRL

Backdoor Attacks in SARL. TrojDRL [Kiourti et al., 2019] is the first to show backdoor attacks on deep RL agents, demonstrating that a small amount of poisoned data and reward tweaks could implant malicious behavior. Yu et al. [Yu et al., 2022] introduce a new DRL backdoor whose trigger is a set of sequential observations, making it stealthy and persistent. BadRL [Cui et al., 2024] uses highly sparse poisoned samples during training and testing, maintaining a high attack success rate while reducing cost. BAFFLE [Gong et al., 2024] implants a backdoor by poisoning offline RL datasets.

Backdoor Attacks in com-MARL. BACKDOORL [Wang et al., 2021a] uses a sequence of actions by the opponent as a trigger, so that the adversary can activate the victim agent's backdoor through its own actions. SCAB [Liu et al., 2025] injects backdoors via pre-trained agents interacting with legitimate actions without the need for observation perturbations or reward modification.

Backdoor Attacks in coo-MARL. MARNet [Chen et al., 2022] successfully implements backdoor attacks on cooMARL systems by designing trigger, action poisoning, and reward hacking modules, causing the attacker's agent to behave abnormally in a malicious environment. One4all[Zheng et al., 2023] presents two poisoning attack techniques (SNPA and TAPA), which covertly poison the coo-MARL system by manipulating a single agent's observations or reward functions. BLAST [Yu et al., 2024b; Fang et al., 2025] introduces a unilateral filter that enables a single backdoor agent to quickly affect other clean agents, thereby enabling a backdoor leverage attack on the entire multi-agent system.

2.4 Backdoor Defenses in DRL

[Acharya et al., 2023] detects backdoor by analyzing discrepancies in how the policy evaluates state observations. [Vyas et al., 2024] presents a runtime detection method based on neural activation patterns, effectively detecting subtly hidden backdoor triggers. These methods are effective, but cannot mitigate detected backdoors. PolicyCleanse [Guo et al., 2023] detects Trojan agents in com-MARL according to abnormal accumulated rewards and mitigates the backdoor via machine unlearning. BIRD [Chen et al., 2023] and SHINE [Yuan et al., 2024] leverage trigger restoration and policy finetuning to detect and remove backdoors in DRL policies without requiring attack specifications. These methods all rely on reward anomalies and expensive policy retraining to detect and mitigate backdoors, which results in potential missed detection risks and high defense overhead. Bharti et al. [Bharti et al., 2022] proposed a provable, retraining-free backdoor defense mechanism that projects observations onto a safe subspace to suppress backdoor activation. However, it only supports the single-agent scenario.

Different from current methods, our BehaviorGuard is based on a novel behavioral drift metric without trigger priors, reward signals, or policy retraining, and can be deployed to defend against backdoors online. Moreover, it is effective against different trigger patterns, instant or sequential behaviors, environments (SARL, com-MARL, or coo-MARL), and retains defense power even when rewards appear normal.

3 Method

3.1 Threat Model

Attacker's Capabilities and Goal. The attacker has complete control over the training process, data, and reward functions, and can inject any backdoors with instant or sequential triggers into policies. These policies can strive to achieve their benign objectives, but once the backdoor is activated, they will immediately perform backdoor objectives.

Defender's Capabilities and Goal. The defender can only download or outsource the final policy, which may potentially contain a backdoor. We assume the defender does not have the resources to retrain policies. This enables higher practicality. The defender aims to detect whether the policy contains a backdoor and, if so, mitigate its impact.

3.2 Key Intuition

One key observation across different backdoor attacks (e.g., TrojDRL[Kiourti et al., 2019] and BACKDOORL[Wang et al., 2021a]) is that their policies systematically reshape the action distribution, leaving detectable behavioral drifts. As shown in Fig. 1, which compares the BDS distribution of clean policies and those injected with 3 types of TrojDRL attacks in SARL and 2 types of BACKDOORL attacks in comMARL, these drifts manifest in distinct patterns. In SARL, the clean BDS distribution is highly concentrated, owing to the stability of the single-agent environment. Backdoor attacks undermine this consistency, resulting in dispersed distributions with heavier tails. In contrast, in the non-stationary environment of com-MARL, a clean policy naturally exhibits a wider BDS distribution due to adversarial interactions. Intriguingly, BACKDOORL counteracts this by enforcing a fixed, cooperative strategy to achieve its attack goal, leading to an abnormally concentrated BDS distribution. In summary, despite variations in its directional pattern across environments, the behavioral drift induced by the inherent conflict between primary and backdoor tasks serves as a robust, universal clue for backdoor defense.

(b) com-MARL: sumo-humans

Figure 1: Behavioral drift comparison of clean and backdoor policies in single- and multi-agent environments.

3.3 Behavioral Drift Detection for Backdoor Detection

Based on the above observation, in BehaviorGuard, we design a behavioral drift detection (BDD) method based on BDS to detect backdoors against both SARL and MARL. However, their underlying mechanisms and observable characteristics differ substantially. Hence, to maintain conceptual clarity and analytical focus, here, we mainly describe our designs of BDD for SARL and leave its extensions for coo- and com-MARL in our full version [Yu et al., 2026].

In the single-agent scenario, once the backdoor is activated, it will induce systematic shifts in the policy's action distributions under attack conditions, while the policy remains indistinguishable from a clean model on benign inputs. BDD aims to identify these attacks by quantifying deviations between the policy's runtime behavior and its expected benign statistics. Given a trajectory of length Ts, BDS at each timestep t can be calculated as:

where pt,k denotes the predicted probability of the k-th action at timestep t, and (µk, σk) are the corresponding mean and standard deviation estimated from clean executions. The constant ϵ is used for numerical stability.

To characterize the distribution of behavioral drift over the entire execution, we collect the sequence {BDSt}Ts t=1 and estimate its probability density function using KDE:

where K(·) denotes a Gaussian kernel, h is the bandwidth parameter, and {BDSi}Ni=1 are the observed drift samples used for density estimation. Based on the estimated density, we define a tail statistic that measures the proportion of timesteps exhibiting rare, low-density deviations:

where τtail is defined as the density value corresponding to the low-density tail of the BDS distribution. It is calibrated

Algorithm 1 Behavioral Drift Detection of BehaviorGuard

t=1 ; benign stats (µk, σk); KDE (K, h); density threshold τtail; weights (w1, w2); decision

Require: Action probabilities {pt,k}Ts threshold τ Ensure: Detection result 1: Initialize B ← [ ] 2: for t = 1 to Ts do 3: BDSt ← 1 2 ∑Kk=1 ( pt,k-µk σk+ϵ )2 4: Append BDSt to B 5: end for 6: for t = 1 to Ts do 7: ˆ f(B[t]) ← 1 Nh ∑Ni=1 K( B[t]-B[i] h ) 8: end for 9: tail ← 1 Ts ∑Ts t=1 1[ ˆ f (B[t]) < τtail ] 10: BDS ← 1 Ts ∑Ts t=1 B[t] 11: Compute ̂ BDS according to Eq. (6) 12: if ̂ BDS > τ then 13: return "Potential Backdoor Detected" 14: else 15: return "No Backdoor Detected" 16: end if

by fitting a KDE to the clean baseline BDSt values and setting τtail to the 5th percentile of the estimated densities ˆ f (BDSt).

We further aggregate the drift magnitude over the trajectory by computing the mean drift score BDS = 1 Ts ∑Ts t=1 BDSt. Both BDS and tail are normalized using Z-scores:

Finally, they are combined into a composite detection score:

where we use 1 √2 to keep ̂ BDS's range being consistent with BDS. A policy execution is flagged as potentially backdoored when ̂ BDS exceeds a predefined threshold τ. We set τ as the 95th percentile of the composite score distribution computed from clean baseline executions, and fix it for all subsequent backdoor detection. The complete detection procedure is summarized in Algorithm 1.

The above detection formulation is designed for singleagent policies, and extending it to multi-agent settings requires additional considerations. In competitive multi-agent environments, backdoor behaviors often emerge through interaction-dependent deviations, where an agent's abnormal actions are amplified by the opponent's responses. In contrast, cooperative multi-agent scenarios exhibit correlated behavior shifts across agents, in which backdoor activation may manifest as synchronized or role-specific deviations. These characteristics render direct application of single-agent BDD insufficient and require dedicated designs that account for inter-agent dependency and joint behavioral dynamics. Those details of the extended BDD are described in [Yu et al., 2026].

Algorithm 2 Drift-Constrained Backdoor Action Mitigation

Require: Victim policy πθ; drift detector producing score BDSt; threshold τD;

• 1: reference action sequence Aref = {aref t }Tt=1; window length L; guard probability p

Ensure: Executed action ˜at 2: Initialize drift indicator buffer {zi} with length L to zeros 3: for t = 1, 2, . . . do 4: Observe current state st 5: Compute raw action at ← argmaxa πθ(a | st) 6: Obtain reference action aref t ←Aref [t mod T] 7: Obtain drift score BDSt from detector 8: zt ← I(BDSt > τD) 9: Update buffer and compute ct ← ∑ti=max(1,t-L+1) zi 10: gt ← I(ct ≥ L) 11: if gt = 1 and Unif(0, 1) < p then 12: ˜at ← aref t 13: else 14: ˜at ← at 15: end if 16: Execute action ˜at and observe next state 17: end for

3.4 Behavioral-Drift-ConstrainedMitigation

In the spirit of runtime shielding for RL [Alshiekh et al., 2018], we further design an online backdoor mitigation method in BehaviorGuard based on behavioral drift, which constrains policy execution by leveraging the output of BDD. This method is summarized in Algorithm 2. Without requiring trigger priors or policy retraining, our method intervenes online to suppress abnormal actions induced by persistent deviations from normal policy execution.

In the single-agent environment, we assume that typical action distribution characteristics under benign conditions can be approximated from limited offline interactions or historical execution logs, and use them as a normal behavior prior that remains stable under similar environment configurations. Formally, the prior is represented as a time-indexed set of reference actions:

where aref t denotes the most representative action at timestep t under benign conditions (e.g., the mode of the action distribution). During policy inference, the reference action is retrieved via modular indexing

which provides a lightweight behavioral reference without requiring exact state alignment.

At each timestep, BDD outputs a score BDSt. When it exceeds a predefined threshold τD, the output is considered to deviate from normal behavior. τD is calibrated offline as the 95th percentile of the clean baseline executions' BDS distribution, thus filtering out innocuous noise. To reduce false triggers caused by stochasticity or transient noise, we smooth the detection results over time by defining a drift indicator:

where I(·) denotes the indicator function. Drift occurrences are then accumulated within a temporal window of length L:

where L controls how many consecutive drift detections are required before mitigation is triggered. When ct ≥ L, the policy is regarded as entering a persistent abnormal behavior regime, and a mitigation gate gt = I(ct ≥ L) is activated.

At timestep t, the policy produces a raw action at = argmaxa πθ(a | st). If mitigation is not triggered (gt = 0), the raw action is executed. Otherwise, when persistent behavioral drift is detected (gt = 1), the action is corrected with probability p by projecting it back to the reference action:

p is a mitigation probability. This probabilistic correction avoids deterministic intervention at all timesteps and allows the mitigation strength to be controlled via parameters p and L, thereby suppressing abnormal behavior while preserving benign policy performance.

Overall, BehaviorGuard forms a unified closed loop between detection and mitigation: BDD determines whether the policy has entered an abnormal regime, while action constraints intervene at the execution level only when persistent drift is confirmed. This design enables effective and controllable backdoor suppression with a low computational overhead, without policy retraining or trigger knowledge. Extensions to multi-agent environments based on the same principle are provided in [Yu et al., 2026].

4 Evaluation

4.1 Experiment Setup

Evaluation Environments. We evaluate the performance of BehaviorGuard against backdoor attacks in discrete-action settings (SARL and coo-MARL) and a continuous-action setting (com-MARL).

SARL. We follow TrojDRL [Kiourti et al., 2019] and consider three attack variants: Strong-Targeted, Weak-Targeted, and Untargeted. Experiments are conducted on four Atari games from OpenAI Gym [Brockman et al., 2016] and ALE [Bellemare et al., 2013]: Breakout, Pong, Seaquest, and SpaceInvaders. These tasks exhibit diverse dynamics and action-selection patterns, allowing us to evaluate detection behavior across heterogeneous state distributions.

com-MARL. We adopt BACKDOORL [Wang et al., 2021a], which targets adversarial interaction settings, and evaluate on three tasks from the competitive benchmark suite [Bansal et al., 2017]: Sumo-Humans, Run-To-Goal, and YouShall-Not-Pass. The strategic coupling and non-stationarity induced by opponent behaviors make this setting particularly challenging for trigger-agnostic detection.

coo-MARL. We evaluate against the MARNet attack [Chen et al., 2022] on two representative coo-MARL algorithms (QMIX [Rashid et al., 2018] and COMA [Foerster et al., 2018]), which use specific textures as their trigger.

We set the mitigation probability p = 0.5 and the window length L = 5 to default across all environments. For each task, we evaluate multiple clean and backdoored policies to account for training variability.

Metrics. We assess the proposed approach using key metrics. We use the average episode reward in SARL tasks and the winning rate in MARL tasks to compare different policies. We use the False Positive Rate (FPR) and its AUC, which reflects the proportion of false positives. We also introduce a mitigation rate (MR) to compare defense performance across heterogeneous environments, which is a normalized recovery ratio of a defended policy from the poisoned performance toward the clean one. These metrics provide a comprehensive evaluation of the method's performance and allow for comparison with baseline approaches. All evaluations were conducted with an RTX 5090 GPU.

Baselines. We compare against representative backdoor defense baselines. First, we implement direct retraining via PPO-based retraining [Schulman et al., 2017] under the same environment and training budget for a fair comparison. Second, we include Neural Cleanse (NC) [Wang et al., 2019], STRIP [Gao et al., 2019], and FeatureRE [Wang et al., 2022] as classic trigger-oriented defenses, adapted to RL policies by operating on state observations (as the model input) and flagging abnormal responses. We further include RL-specific defenses, including BIRD[Chen et al., 2023], SHINE[Yuan et al., 2024], and PD[Bharti et al., 2022].

4.2 Experiment Results

Overall Defense Effectiveness

Table 1 demonstrates the defense effectiveness of our BehaviorGuard and 7 baselines under the clean and poisoned environments. Among the 3 classic defenses, STRIP outperforms NC and FeatureRE, mainly because the latter two suffer from low-fidelity trigger reconstruction. Direct retraining can partially mitigate backdoors in SARL and coo-MARL, but fails in all com-MARL environments. BIRD and SHINE defend against backdoors via RL-specific trigger restoration and finetuning, but SHINE has stronger defense performance due to its high-fidelity trigger reconstruction. PD, as a retrainingfree defense, performs poorly in both clean and poisoned environments due to its defective projection on all input states. BehaviorGuard does not introduce negative effects on the policy in the clean environment and achieves defense performance comparable to SHINE in the poison environment. While SHINE can accurately restore spatial triggers in SARL and coo-MARL, it struggles with complex sequential triggers used in com-MARL. In contrast, BehaviorGuard is triggeragnostic and does not rely on trigger recognition, enabling consistent robustness across diverse RL settings.

Detection Performance

We evaluated whether BehaviorGuard can reliably identify trigger activations. Fig. 2 demonstrates its detection quality for SARL and MARL via ROC curves and AUC values. Fig. 2(a) shows that BehaviorGuard achieves consistently strong discrimination on SARL Atari tasks, with ROC curves concentrated toward the upper-left region and AUC values above 0.90 across games, indicating reliable separation between clean and backdoored policies under heterogeneous dynamics. Fig. 2(b) further demonstrates that BehaviorGuard remains effective in multi-agent benchmarks, where interaction-induced non-stationarity and strategic coupling introduce substantial benign behavioral variability. Overall, these results indicate that our BDS provides a stable detection signal across single- and multi-agent systems.

Robustness Across Advanced Attacks

We further evaluate BehaviorGuard's defense effectiveness under four different backdoor attacks: (i) Normal attacks (macro-averaged from Tab. 1), (ii) Reward-constrained attack [Rathbun et al., 2025], which enforces bounded backdoor reward modifications to implant backdoors; (iii) Sequential attack in SARL [Yu et al., 2022], in which we use a patch to appear clockwise on the four corners of input images in 4 consecutive timesteps as a sequential trigger and perform backdoor actions after the 4 timesteps, and (iv) Sequential attack in coo-MARL [Yu et al., 2024b; Fang et al., 2025], whose triggers are defined by agent-wise sequential actions with spatial and temporal dependences. We use MR to compare defense performance across different tasks. MR= 1 indicates a clean-level recovery, and MR> 1 allows slight over-recovery due to regularization effects.

Table 2 compares robustness across attack types. Besides normal attacks, the other 3 advanced attacks suppress reward anomalies or embed triggers into sequential/interactiondependent states, leading to low performance of trigger identification in BIRD and SHINE. BehaviorGuard achieves the most consistent robustness across attacks, matching strong baselines under normal attacks while notably outperforming existing methods under advanced attacks.

Time Cost of Backdoor Defense

In Table 3, we evaluated the defense time cost of SHINE, BIRD, PD, and BehaviorGuard in 4 single-agent environments. In each environment, following the standard workflow of these methods, we first ran a set of episodes (10000, 1000, 10, and 1000) with a maximum of 2,000 steps per episode to perform their offline operations and then ran 1,000 episodes to evaluate their online operations and defended policies. We measured the time cost of dominant phases in these methods. Existing methods require hours due to their expansive operations, including trigger restoration in SHINE, policy finetuning in SHINE and BIRD, and subspace sanitization and state projection in PD. BIRD's trigger restoration can converge early, thereby being faster than SHINE. PD and BehaviorGuard perform backdoor mitigation online, but the high computational cost of PD's online state projection limits its applicability for policy inference. In contrast, BehaviorGuard finishes in 35.03 minutes (18.81m for offline baseline calibration and 16.22m for online detection and mitigation), contributed by its lightweight drift scoring and action correction. In summary, BehaviorGuard significantly outperforms existing defenses in terms of both efficacy and time cost, and its online defense capability ensures practical applicability.

4.3 Ablation Studies

We conducted ablations to assess the effectiveness and robustness of BehaviorGuard under different configurations. Results in Fig. 3 are averaged over multiple independent runs in Atari environments.

Attack types. Fig. 3(a) compares BehaviorGuard's MR under 3 TrojDRL attacks. Its performance under targeted attacks is more stable than under untargeted attacks, especially under strong targeted attacks. This indicates that BehaviorGuard is more reliable when the abnormal behavior manifests as sustained drift targeted by most backdoor attacks, rather than isolated, stochastic deviations.

Guard probability p. Fig. 3(b) studies the guard probability p used in Eq. (11), which controls the strength of probabilistic action correction after persistent drift is detected. Setting p = 0 effectively disables mitigation and results in limited recovery, whereas moderate values (e.g., p ∈ [0.25, 0.5]) provide a clear improvement across environments. Further increasing p yields diminishing returns and may introduce unnecessary interventions.

Window length L. Fig. 3(c) studies the impact of the window length L used in Eq. (10), i.e., the number of consecutive drift detections required to activate mitigation. Small L values tend to trigger overly early corrections, while overly large L delays intervention and reduces mitigation effectiveness. Intermediate L achieves the best trade-off, supporting the design choice of modeling persistent drift instead of reacting to instantaneous fluctuations.

Trigger size. Fig. 3(d) evaluates robustness to trigger size. Mitigation performance remains stable across different trigger magnitudes without catastrophic degradation, indicating that the strategy does not rely on a specific trigger scale and generalizes across attack configurations.

In summary, these results validate the necessity of combining temporal drift modeling with probabilistic action correction, and show that the key hyperparameters admit a reasonable operating range with consistent performance.

4.4 Discussion and Limitations

Our experimental results demonstrate behavioral drift as a trigger-agnostic signature for RL backdoor defense: it yields reliable separability across SARL and MARL and remains effective under interaction-induced non-stationary environments. BehaviorGuard's online guard achieves a favorable trade-off between utility and security, preserving clean performance while recovering poisoned performance without policy retraining or trigger reconstruction. The gains are especially clear for reward-constrained and sequential triggers where reward- or pattern-dependent defenses are weakened. BehaviorGuard still has a limitation. It is the reliance on benign baselines and a simple reference-action correction, which may require recalibration under distribution shift (e.g., new opponents or updated policies). Addressing this recalibration challenge and evaluating adaptive attackers that explicitly minimize drift are our future works.

5 Conclusion

We proposed a behavior-centric, online framework for backdoor detection and mitigation in DRL. Our key insight is that backdoored policies, despite differing trigger designs and objectives, tend to introduce persistent shifts in action distributions to ensure reliable activation, leaving detectable signatures in high-quantile regions and distribution tails even on trigger-free inputs. Based on this insight, we developed BehaviorGuard, which consists of a drift-based detector for backdoor activations without relying on reward anomalies, trigger priors, or trigger reconstruction, and a mitigator that suppresses backdoor actions at runtime and without requiring policy retraining. Extensive evaluations on SARL and MARL benchmarks demonstrate strong detection performance and a favorable utility-security trade-off: our defense substantially recovers poisoned performance while preserving clean-task performance, including under normal, reward-constrained, and sequential attacks. Future work will extend the drift constraint to continuous-action policies, improve calibration under distribution shift, and study adaptive attackers that explicitly optimize for drift evasion.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (62202387), the Guangdong Basic and Applied Basic Research Foundation (2025A1515011112), and the supporting funds for talents of Nanjing University of Aeronautics and Astronautics.