Counterfactual Explanations for Continuous Action Reinforcement Learning

1 Introduction

Reinforcement Learning (RL) has shown significant potential in tackling complex decision-making tasks across diverse fields, including healthcare [Yu et al., 2021] and robotics [Tang et al., 2024]. However, its adoption in highstakes applications is often hindered by a critical barrier: the lack of interpretability. Understanding an RL agent's decision-making process is essential for fostering trust and enhancing performance. While there is a growing body of work on explainable RL [Milani et al., 2023], most methods focus on summarizing policies or explaining individual actions using natural language or saliency maps. Counterfactual explanations, which address "what if" questions to improve interpretability, remain underexplored in RL.

In contrast, most research on counterfactual explanations has focused on supervised learning, aiming to identify minimal changes to input features that yield a desired classifier output [Verma et al., 2024]. These approaches often incorporate constraints such as validity (ensuring counterfactuals belong to the target class), proximity (minimizing deviations from the original instance), actionability (restricting changes to modifiable features), sparsity (minimizing the number of changes), data manifold closeness (ensuring realism by adhering to the training data distribution), and causality (preserving known causal relationships). However, as noted in [Gajcin and Dusparic, 2024], these methods cannot be directly applied to RL due to unique challenges, such as the temporal dependencies in decision-making and outcomes.

Existing research on counterfactual explanations for RL remains limited and often targets discrete action spaces. These approaches typically rely on heuristics [Amitai et al., 2024] or structural causal models [Tsirtsis et al., 2021] to generate counterfactual actions, or they focus narrowly on generating alternative state features [Olson et al., 2021]. A detailed discussion of these prior works is provided in Section 2.

To address these gaps, we propose a novel approach for generating counterfactual explanations in RL tailored for continuous action spaces. Our method computes an alternative sequence of actions for a given observed trajectory, aiming to achieve better outcomes (i.e., higher cumulative rewards) while minimizing deviations from the original actions. This is accomplished using a distance metric designed for continuous action sequences. Additionally, we extend the problem formulation to account for scenarios where actions in specific constrained states must adhere to a predefined policy.

A compelling example is the application of RL for managing blood glucose levels in diabetes patients [Tejedor et al., 2020]. Counterfactual explanations in this scenario can answer questions like: "What alternative insulin dosages could have resulted in better glycemic control?" These explanations must ensure minimal deviation from the original treatment plan while adhering to constraints, such as following a doctor's prescription (e.g., administering specified insulin doses when glucose levels are within defined ranges). This example highlights the practical relevance of our approach in highstakes decision-making tasks.

Instead of generating counterfactuals for individual observed trajectories one at a time, our approach computes a counterfactual policy that simultaneously generates desired counterfactuals for a set of observed trajectories (e.g., a patient's historical glycemic control trajectories over a specific period). To enhance interpretability, the counterfactual policy is deterministic (e.g., ensuring that a specific insulin dosage, rather than a probabilistic range, is recommended at a time).

Our approach extends the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm [Fujimoto et al., 2018], introducing novel mechanisms for generating counterfactual trajectories in continuous action spaces and a sparse rewardshaping framework to balance reward gains with minimal action deviations.

Finally, we evaluated our approach in two RL domains: diabetes control using the FDA-approved UVA/PADOVA simulator [Man et al., 2014] and Lunar Lander from OpenAI Gym [Brockman, 2016]. Experimental results demonstrate the effectiveness, efficiency, and generalization of our approach, paving the way for more interpretable and trustworthy RL applications in high-stakes settings.

2 RelatedWork

Existing work on counterfactual explanations has predominantly focused on supervised learning, with early efforts like [Wachter et al., 2018] framing the problem as an optimization task to identify minimal changes to input features that result in a desired classifier output. Following this foundational work, extensive research has explored incorporating constraints to ensure counterfactuals are actionable and realistic, as surveyed in [Verma et al., 2024; Guidotti, 2022]. [Chen et al., 2024] employed RL-based methods to generate optimal counterfactual explanations for classifiers and subsequently distilled decision trees to explain the counterfactual generation process.

Research on counterfactual explanations for RL is relatively limited compared to supervised learning. [Olson et al., 2021] investigated counterfactual state explanations for deep RL agents in visual environments like Atari, examining how minimal changes to game images could alter an agent's action. [Gajcin and Dusparic, 2024] extended counterfactual explanation concepts from classifiers to RL, defining various types to address questions such as "Had state features taken different values, action a′ (or policy π′ ) would be chosen instead of action a (or policy π)" and "Had the agent followed a different goal (or preferred a different objective) in state s, action a′ (or policy π′ ) would be chosen instead of action a (or policy π)." Conversely, our work focuses on computing counterfactual explanations to explore what alternative actions or policies the agent could take to achieve better outcomes.

Our definition of counterfactual explanations is inspired by [Tsirtsis et al., 2021], which identifies alternative action sequences differing in at most k actions to achieve better outcomes in finite-horizon MDPs. However, their approach is limited to discrete action spaces and relies on a Gumbel-Max structural causal model. In contrast, our work targets RL with continuous actions, introduces a novel distance measure for action sequences, and avoids dependence on causal models.

More recently, [Amitai et al., 2024] proposed a method for visually comparing an agent's chosen action to counterfactual alternatives, conducting user studies in the highway environment with videos illustrating counterfactual outcomes. However, their approach is limited to discrete action spaces, with counterfactual actions selected heuristically. Instead, our work introduces an optimization-based method to compute counterfactual actions in continuous spaces, ensuring minimal distance from observed actions.

Additionally, other studies have explored counterfactual reasoning to enhance RL techniques. [Bica et al., 2021] modeled expert decisions by defining reward functions based on preferences for "what if" outcomes and incorporating counterfactual reasoning into batch inverse RL. [Frost et al., 2021] generated counterfactual trajectories by guiding agents to diverse, unseen states, enabling users to view rollouts of agent behavior and gain insights into its actions under test-time conditions. Our work has a different goal and formalizes counterfactual reasoning in RL through a novel problem formulation tailored for continuous action spaces.

3 ProblemFormulation

We consider a problem setup of an RL agent interacting with an environment modeled as a Markov decision process (MDP), denoted as M = (S,A, P,R), where S represents the state space, A denotes the (continuous) action space, P is the (unknown) probabilistic transition function, and R is the reward function. At each time step t, the agent selects an action at based on the current state st, receives a reward R(st, at), and transitions to the next state st+1 ∼ P(st, at) as governed by the environment dynamics. The agent's execution over a finite number of n steps yields a trajectory τ = {(st+i, at+i)}ni=0 with cumulative reward G(τ ) = ∑ni=0 R(st+i, at+i).

Given an observed trajectory τ = {(st+i, at+i)}ni=0 , we define a (positive) counterfactual trajectory τ ′ as one that originates from the same initial state st but achieves a higher cumulative reward, i.e., G(τ ′ ) > G(τ), by adopting an alternative sequence of actions α(τ ′ ) = {a′t+i }ni=0 . We define the distance between two action sequences α(τ ) and α(τ ′ ) as:

where δ > 0 is a parameter to ensure numerical stability. For multi-dimensional action spaces, this equation is generalized using ℓp norms.

We formulate an optimization problem to compute a counterfactual trajectory with minimal action distance.

Problem 1. Given a trajectory τ = {(st+i, at+i)}ni=0 , find a counterfactual trajectory τ ′ starting from st that satisfies:

Next, we consider a variant where actions in certain constrained states (denoted by Sc ⊆ S) must follow a predefined policy πc , while actions in unconstrained states can be freely optimized to minimize the action distance and satisfy the reward constraint.

Problem 2. Given a trajectory τ = {(st+i, at+i)}ni=0 , find a counterfactual trajectory τ ′ starting from st that satisfies:

Motivating example. We illustrate these problems using the example of blood glucose (BG) control for Type 1 Diabetes patients. The state space represents BG levels, influenced by factors such as patient physiology and behaviors (e.g., eating, exercise). The action space consists of insulin doses. At each time step (e.g., every three minutes), the agent computes an insulin dosage to regulate BG levels. The reward function assigns positive rewards for BG levels within the target range and penalties for hypoglycemia (BG < 70 mg/dL) or hyperglycemia (BG > 180 mg/dL). Figure 1 shows an observed trajectory of a patient's BG level and insulin dosage over an hour, with a cumulative reward G(τ ) = 12.8.

The goal is to find a counterfactual explanation addressing the question: "What alternative treatment plan (i.e., insulin dosage sequence) could lead to better glycemic control outcomes (i.e., higher cumulative reward)?" The counterfactual explanation should involve minimal changes from the original treatment plan (i.e., minimal distance of actions) and may include constraints (e.g., following clinical prescriptions).

Figure 1 shows three counterfactual trajectories τ1, τ2 and τ3. Among them, trajectory τ1 achieves the highest cumulative reward G(τ1) = 13.7 with the smallest action distance D(α(τ), α(τ1)) = 1.13. We note that trajectory τ1 also adheres to the constraint of "injecting 0.03 units of insulin when BG levels fall below 100 mg/dL".

4 Approach

We propose a novel approach to solve the problems described in Section 3. Our approach extends the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm [Fujimoto et al., 2018], incorporating mechanisms for counterfactual generation and sparse reward shaping to account for trajectory action distance.

4.1 Solving Problem 1

We first relax the hard constraint G(τ ′ ) > G(τ ) in Problem 1 into a soft penalty term, yielding the following optimization problem:min

where λ is a hyperparameter reflecting the user's preferred weight for the penalty term. This is equivalent to

Directly searching over all possible trajectories τ ′ is challenging without knowledge of the transition function. Observing that trajectories are generated by policies, we approximate the problem by:

Here µ : S → A denotes a deterministic policy, and τ ′ (µ) represents the trajectory generated by following µ starting from st, the first state of τ. Specifically, τ ′ (µ) .= {(s′t+i , a′t+i )}ni=0 , where s′t .= st, a′t+i .= µ(s′t+i ), s′t+i+1 ∼ P(s′t+i , a′t+i ). Notably, while µ is deterministic, τ ′ (µ) remains a random variable due to the stochasticity in sampling s′t+i+1 from the transition function. The essential supremum (ess sup) is taken over all possible realizations of τ ′ (µ). We use the essential supremum here because Equation 3 requires finding the single best trajectory generated by µ.

Optimizing the essential supremum directly is difficult, as no existing methods are designed for this in sequential decision-making. Instead, leveraging concentration inequalities (e.g., Hoeffding's inequality) and large deviation theory, which indicate that deviations from the expectation are typically well-bounded, we optimize the expectation:

A policy µ that maximizes this expectation will, with high probability, achieve a large essential supremum. Here, E [G(τ ′ (µ))] corresponds to the expected total rewards of the deterministic policy µ. Thus, Equation 5 can be framed as a standard RL problem, where the goal is to maximize both the total rewards G(τ ′ (µ)) and an additional sparse reward -λD(α(τ ), α(τ ′ (µ))), provided only at the end of each episode.

Given the deterministic nature of the policy µ and the proven success of TD3 in optimizing deterministic policies, we extend TD3 to solve this problem, as outlined in Algorithm 1. The algorithm starts with initializing the critic networks Qθ1 and Qθ2 , the actor network πϕ with random parameters, and target networks as copies of the original networks to ensure training stability. A replay buffer B is also initialized to store counterfactual trajectories. Given a set of observed trajectories T , the algorithm samples a trajectory τ = (st+i, at+i)ni=0 from T and generates Nc counterfactual trajectories. Each counterfactual trajectory τk starts from the same initial state as τ and is rolled out by selecting actions through the actor network πϕ, with exploration noise added. To incorporate the action distance objective in Problem 1, the distance D(α(τ), α(τk)) is computed using Equation 1, and the reward is adjusted with a weighted function (line 12) following Equation 5. Each generated counterfactual transition is stored in the replay buffer B. The algorithm then samples a batch of transitions from B to compute the target action ˜a and target value y. The critic networks are updated by minimizing the mean squared error between the predicted Q-values

Algorithm 1: Counterfactual Generation

1 Initialize critic networks Qθ1 , Qθ2 , and actor network πϕ with random parameters θ1, θ2, ϕ 2 Initialize target networks: θ′1 ← θ1, θ′2 ← θ2, ϕ′ ←ϕ 3 Initialize replay buffer B 4 for e = 1 to No do 5 Sample a trajectory τ = {(st+i, at+i)}ni=0 fromT 6 for k = 1 to Nc do 7 Set initial state s′t ← st 8 for i = 0 to n do 9 Select action with exploration noise: a′t+i ∼ πϕ(s′t+i ) + ϵ, ϵ ∼ N(0, σ) 10 Observe reward r′t+i and next state s′t+i+1 11 if i = n then 12 Adjust reward: r′t+i ← r′t+i - λ · D(α(τ ), α(τk)) 13 Store (s′t+i , a′t+i , r′t+i , s′t+i+1 ) in B 14 Sample mini-batch of N transitions (s, a, r, s′ ) from replay buffer B 15 ˜a ← πϕ′ (s′ ) + ϵ, ϵ ∼ clip(N (0, ˜σ), -c, c) 16 y ← r + γ minj=1,2 Qθ′j (s′ , ˜a) 17 Update critics: θj ← arg minθj N -1 ∑ (y - Qθj (s, a))2 18 if e mod q = 0 then 19 Update actor ϕ using deterministic policy gradient: ∇ϕJ(ϕ) = N -1 ∑ ∇aQθ1 (s, a)∣∣a=πϕ(s)∇ϕπϕ(s) 20 Update target networks: θ′j ← ηθj +(1-η)θ′j , ϕ′ ← ηϕ+(1-η)ϕ′

and the target value. To stabilize training, the actor network πϕ is updated less frequently, every q iterations, using deterministic policy gradients. Finally, target networks are softly updated with a weighted average of the current and target parameters, further enhancing training stability. The algorithm terminates after sampling and processing No observed trajectories.

4.2 Solving Problem 2

To address Problem 2, we construct an augmented MDP, transforming Problem 2 into an instance of Problem 1. Specifically, the augmented MDP has a state space defined as ˆ S .= S \ Sc . In this augmented MDP, given a state-action pair (s, a), the transition proceeds as follows. The successor state is first determined using the original transition function P. If the successor state does not belong to the constrained set Sc , it is presented to the agent immediately; otherwise, the predefined policy πc is followed until the agent reaches a state outside Sc .

By constructing this augmented MDP, Problem 2 in the original MDP is reduced to Problem 1 in the augmented MDP. Consequently, we can directly apply Algorithm 1 in this augmented MDP to solve Problem 2.

5 Experiments

We implemented the proposed approach and evaluated it in two RL domains: (i) diabetes control using the FDAapproved UVA/PADOVA simulator [Man et al., 2014], and (ii) Lunar Lander from OpenAI Gym [Brockman, 2016]. Our implementation 1 is based on Stable-Baselines3 [Raffin et al., 2021].

Data Generation. We trained a baseline policy using the Proximal Policy Optimization (PPO) algorithm [Schulman et al., 2017] to generate trajectories for training and test datasets (details specific to each domain are provided later). Our approach is RL-method agnostic and can generate counterfactuals from trajectories produced by any RL algorithm.

Baseline Method. As no existing methods generate counterfactuals for continuous action RL, we compare our approach to a naive baseline that generates counterfactuals by rolling out the trained baseline policy in evaluation mode without state constraints or additional training.

Metrics. Our evaluation focuses on the effectiveness, efficiency, and generalization of the proposed approach. We define two metrics to assess the effectiveness.

• Positive Counterfactual Percentage (ρ+): This metric measures the percentage of test set trajectories for which at least one positive counterfactual trajectory (i.e., a trajectory with a higher cumulative reward than the observed trajectory) is identified among a fixed number of generated counterfactuals. This metric applies to both the baseline and the proposed approaches.
• Advantage Counterfactual Percentage (ρadv): This metric quantifies the percentage of test set trajectories for which the proposed approach generates a more advantageous counterfactual than the baseline. For an observed trajectory τ, let τ ∗ p and τ ∗ b denote the best positive counterfactual trajectories with minimal action distance produced by the proposed and baseline approaches, respectively, satisfying G(τ ∗ p ) > G(τ) and G(τ ∗ b ) > G(τ). The proposed approach is considered advantageous if ϕG > ϕD, where:

indicating that reward gains outweigh additional action distance. ρadv is computed as the percentage of advantageous counterfactuals among all test trajectories with valid positive counterfactuals.

Efficiency is assessed through learning curves of ρ+ and ρadv, illustrating performance improvements during the training of the proposed approach. Generalization is evaluated by testing the approach on unseen trajectory datasets across diverse single- and multi-environment settings.

1 Code is available at: https://github.com/safe-autonomy-lab/ CounterfactualRL

5.1 Diabetes Control

MDP Environment. The FDA-approved UVA/PADOVA Simulator [Man et al., 2014] was used to model glucoseinsulin dynamics and simulate the effects of insulin delivery, carbohydrate intake, and other factors on blood glucose levels in Type 1 Diabetes patients. Virtual patient profiles were generated with varying parameters (e.g., age, weight, insulin sensitivity). The MDP state space includes glucose reading, glucose rate of change, and carbohydrate intake, while the action space consists of the insulin dosage at each time step. A reward function from [Zhu et al., 2020] was used, providing the highest rewards for maintaining glucose levels within 90-140 mg/dL, smaller rewards for near-target ranges, and penalties for hypo- or hyperglycemic values, with increasing penalties for larger deviations.

DataGeneration. The proposed approachwas evaluated in two settings: single-environment (single patient) and multienvironment (population model with multiple patients). In the single-environment setting, a baseline policy was trained on a chosen patient profile for 100,000 steps, with a learning rate of 0.0001 and a gradient step size of 50. Observed trajectories were generated using a sliding window method, with each trajectory representing a 20-step segment (corresponding to a one-hour patient execution history). In the multi-environment setting, the baseline policy was trained on three patient profiles, with 3,000 steps per patient per round, continuing until stable performance was achieved. Other parameters were consistent with the single-environment setting. Both settings included 18 unique trajectories in each training and test set.

Experimental Setup. We evaluated three variants of the proposed approach: (P1) directly applying Algorithm 1 to solve Problem 1; (P2-base) adapting Algorithm 1 for Problem 2 with constrained states Sc (glucose levels below 100 mg/dL) and the baseline policy as πc ; and (P2-fixed) solving Problem 2 with the same Sc but using a fixed policy πc that injects 0.03 units of insulin per step. For each variant, Algorithm 1 trained an RL model to generate counterfactual trajectories with a distance reward weight of λ = 1. Counterfactual trajectories were computed for each training set trajectory and stored in a buffer. After a warm-up phase, batches of 256 trajectories were sampled for model updates using a learning rate of 0.0001 and 50 gradient steps. The RL agent was tested every 400 interaction steps, generating 10 counterfactual trajectories per test trajectory via policy rollout for evaluation. All three variants were compared to the same baseline method (rolling out the baseline policy without constrained states).

Results Analysis. We conducted seven independent trials for each method, and the results are analyzed below.

Effectiveness: Tables 1 and 2 present the ρ+ and ρadv metrics evaluated on test performance across seven trials after training. All three methods outperform the baseline in Positive Counterfactual Percentage (ρ+) across single- and multi-environment settings. In the single-environment setting, P1 and P2-fixed achieve the highest ρ+, with P2-fixed slightly surpassing P1 in the multi-environment setting. For Advantage Counterfactual Percentage (ρadv), P1 consistently demonstrates the best performance in both settings. P2fixed also performs reliably, while P2-base moderately improves over the baseline in the multi-environment setting but fails to generate advantageous counterfactuals in the singleenvironment setting.

Efficiency: Figures 2 and 3 show the learning curves of ρ+ and ρadv for each method in single- and multi-environment settings, with solid lines representing the mean and shaded areas denoting the standard error across seven trials. The baseline curve in Figure 2 represents counterfactuals generated by rolling out the baseline policy at each evaluation point, without additional training. In the single-environment setting (Figures 2a and 3a), P1 converges quickly and achieves the highest ρ+ and ρadv. P2-fixed steadily improves, matching P1 in ρ+ and delivering strong ρadv performance. P2-base, however, converges more slowly and inconsistently, underperforming relative to P1 and P2-fixed. In the multi-environment setting (Figures 2b and 3b), P2-fixed achieves the highest ρ+, while all three methods converge rapidly and maintain stable ρadv performance. Among them, P1 achieves the highest ρadv, followed closely by P2-fixed, with P2-base showing slightly weaker but reliable results.

Generalization: Performance across settings highlights each method's generalization capabilities. P1 generalizes well, maintaining strong and consistent performance in both settings. P2-fixed also generalizes effectively, excelling in the multi-environment setting. P2-base shows limited generalization, struggling in the single-environment but improving slightly in the multi-environment, with notable variability.

Summary: In the diabetes control experiments, the proposed methods consistently outperform the baseline in effectiveness, efficiency, and generalization. P1 is the most effective and efficient, while P2-fixed excels in generalization to multi-environment setting. P2-base, though weaker overall, demonstrates modest generalization potential.

5.2 Lunar Lander

MDP Environment. The Lunar Lander environment from OpenAI Gym [Brockman, 2016] simulates a 2D rocket attempting to land safely on a designated pad. The MDP state space includes eight continuous variables: the lander's x and y coordinates, x and y velocities, orientation angle, angular velocity, and two binary indicators for the left and right legs' contact with the ground. The action space includes two continuous variables controlling the throttle of the main engine and the lateral boosters. The reward function encourages safe landings, efficient fuel use, and minimal engine overuse.

Data Generation. The proposed approach was evaluated under single- and multi-environment settings. In the singleenvironment setting, a baseline policy was trained for 3,000 steps with a learning rate of 0.0001 and a gradient step size of 20. In the multi-environment setting, the baseline policy was trained across three gravity values, with 500 steps per environment per round, until stable performance was achieved. As in the Diabetes domain, trajectories were generated using a sliding window method, with each 20-step segment forming a trajectory. Both settings included 12 randomly sampled trajectories in each training and test set.

Experimental Setup. The proposed approach was used to generate counterfactual trajectories with a learning rate of 0.00001 and a gradient step size of 20, following a process

(a) Single-Environment

similar to the diabetes experiments. Three variants were evaluated: P1, P2-base, and P2-fixed. In P2-base and P2-fixed, constrained states Sc were defined as those with x-velocity in [-0.18, 0.18]. The policy πc was set as the baseline policy in P2-base, while P2-fixed used a fixed policy with action vectors set to (0, 0).

Results Analysis. We conducted seven independent trials for each method and analyzed the results below.

Effectiveness: As shown in Tables 1 and 2, P1 achieves the highest Positive Counterfactual Percentage (ρ+) and Advantage Counterfactual Percentage (ρadv) in both single- and multi-environment settings, surpassing all other methods and the baseline. While P2-base falls short of P1 in both metrics, it outperforms the baseline. P2-fixed shows the weakest performance, slightly below the baseline in ρ+.

Efficiency: The learning curves in Figures 4 and 5 depict the efficiency of each method in terms of Positive Counterfactual Percentage (ρ+) and Advantage Counterfactual Percentage (ρadv) for the Lunar Lander domain. In the single-environment setting (Figures 4a and 5a), P1 converges quickly and achieves the highest ρ+ and ρadv. P2base steadily improves, surpassing the baseline, while P2fixed converges slowly and lags behind other methods. In the multi-environment setting (Figures 4b and 5b), P1 consistently outperforms others in both ρ+ and ρadv, with rapid convergence and strong results. P2-base and P2-fixed demonstrate less consistent progress in ρ+ but maintain competitive performance in ρadv.

Generalization: All three methods generalize effectively

from single- to multi-environment settings. Notably, Advantage Counterfactual Percentage (ρadv) improves significantly in the multi-environment setting, while the learning curves remain comparable across both settings.

Summary: In the lunar landing experiments, P1 consistently outperforms other methods across both settings. P2base demonstrates moderate effectiveness and generalization, with gradual improvements and variability in performance. P2-fixed, while initially competitive, lags behind and performs worse than the baseline in some cases.

5.3 Discussion

Comparing Single- andMulti-Environment. The ρadv results in multi-environment settings show marked improvement over single-environment settings across all methods and domains (Table 2). This suggests that training in diverse environments enhances model performance by exposing it to a wider range of scenarios, enabling better generalization and the integration of more varied information. These findings highlight the value of tailoring counterfactual generation methods to the unique characteristics of each environment and leveraging diverse training conditions to achieve superior outcomes.

Comparing Results of Two Domains. The results from the Diabetes Control and Lunar Lander experiments highlight the influence of domain-specific characteristics on the performance of counterfactual generation methods. While P1 consistently performs best in both domains, P2-fixed outperforms P2-base in Diabetes Control, whereas P2-base surpasses P2fixed in Lunar Lander. These differences arise from the nature of state transitions and action impacts in each environment. In Lunar Lander, actions produce immediate and short-lived effects-adjusting the engine throttle instantly changes the rocket's velocity, with the effects quickly dissipating. Conversely, Diabetes Control involves delayed and prolonged effects, as an insulin dose takes time to affect blood glucose levels and its influence persists for an extended period. This delay likely contributes to the more stable learning curves in the Diabetes Control domain compared to the more variable curves observed in Lunar Lander.

6 Conclusion

This work presents a novel approach for generating counterfactual explanations in continuous action RL. By extending the TD3 algorithm with mechanisms for constraint handling and reward shaping, our approach efficiently generates counterfactual trajectories that improve outcomes while minimizing deviations from observed actions. Experiments in the Diabetes Control and Lunar Lander domains demonstrate our approach's effectiveness, efficiency, and generalization across diverse environments. Notably, the P1 variant (without constrained states) achieves the best overall performance, while the P2 variants (with constrained states) offer flexibility for incorporating user-defined constraints and preferences. Future directions include enhancing the interpretability of counterfactuals through real-time feedback and visualization tools, further improving user trust and understanding.

Acknowledgements

This work was supported in part by the U.S. National Science Foundation under Grants CCF-1942836 and CCF-2131511. The opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsoring agencies.