SPoRt - Safe Policy Ratio: Certified Training and Deployment of Task Policies in Model-Free RL

1 Introduction

Reinforcement Learning (RL) is an area of machine learning where an agent is trained to interact with its environment to maximize some (cumulative) reward [Sutton and Barto, 2014; Mason and Grijalva, 2019]. There has been great interest in applying RL to real-world control problems in fields such as robotics [Kober and Peters, 2014; Hwangbo et al., 2019; Singh et al., 2022], traffic management [Chu et al., 2020; Vertovec and Margellos, 2023; Lee et al., 2023] and autonomous driving [Isele et al., 2018; Ma et al., 2021; Li et al., 2022], to name just a few. Many of these domains typically fall into the realm of "safety-critical" applications, whereby we need to guarantee safety specifications, such as obstacle avoidance. Satisfying safety constraints becomes particularly challenging when we have little to no knowledge of our environment. This problem has been studied in a substantial body of literature, known as model-free safe RL.

Traditional policy gradient algorithms for model-free RL, such as Trust Region Policy Optimization (TRPO) [Schulman et al., 2015] and Policy Proximal Optimization (PPO) [Schulman et al., 2017], allow the agent to explore any behavior during training, including behaviors that would be considered unsafe; this is unacceptable for safety-critical applications. To encode safety into training, a popular formulation is the Constrained Markov Decision Process (CMDP) [Altman, 2021], which includes safety constraints and is typically solved using primal-dual methods [Achiam et al., 2017] and modifying the trust region to exclude unsafe policy updates [Milosevic et al., 2024]. However, the CMDP formulation is limited in its ability to model safety constraints; CMDPs constrain the expected discounted cost return, but for many practical applications we require an explicit bound on the probability that a sampled trajectory violates a safety constraint.

Alternative approaches based on control theory ensure safety by preventing the agent from taking actions that would eventually lead to safety violations; this is achieved using, e.g., Lyapunov and barrier functions [Chow et al., 2018], shielding [Alshiekh et al., 2018; Konighofer et al., 2023] or safety filters [Hsu et al., 2024]. However, these approaches require a model of the environment to predict future safety, and thus are generally limited to model-based setups. Meanwhile, formal methods-based approaches, such as [Hasanbeig et al., 2023], encode safety by leveraging Linear Temporal Logic (LTL) [Pnueli, 1977] as a formal reward-shaping structure. Unlike CMDPs, whereby the original objective is separate from the constraint, LTL formula satisfaction is encoded into the expected return itself, and under certain conditions the trained policy is guaranteed to maximize the probability of LTL formula satisfaction; however, no guarantees can be obtained during training.

Since we presume no knowledge of the environment, as in standard model-free RL, we will rely on finite-sample learning to evaluate the agent's ability to remain safe using probably approximately correct (PAC) guarantees. Finite-sample complexity bounds provide the number of samples needed to, with a given confidence, learn some target function with a certain accuracy [Vidyasagar, 2003]. Tools from statistical learning theory based on Vapnik Chervonenkis (VC) theory have successfully been able to provide finite sample bounds for learning in unknown environments [Vidyasagar, 2003; Tempo et al., 2005], with recent work providing finite sample bounds even under changing target assumptions [Vertovec et al., 2024]. Yet VC-theoretic techniques require the computation of the VC dimension, which is a difficult task for generic optimization problems. Under a convexity assumption, the so-called scenario approach offers a-priori probabilistic feasibility guarantees without resorting to VC theory [Calafiore and Campi, 2006; Campi and Garatti, 2008; Campi and Garatti, 2018].

The scenario approach traditionally relies on independent and identically distributed (i.i.d.) samples to establish its sample-complexity bounds. This creates a limitation in RL contexts, where the sampling distribution changes as policies are updated. As a result, safety guarantees established for one policy cannot be directly transferred when the policy changes. In this work, we overcome this limitation by extending the PAC guarantees to accommodate policy changes. Specifically, we derive a constraint on how much policies can shift while maintaining safety guarantees, and present SPoRt, an approach for adapting an existing safe policy to improve task-specific performance while maintaining a bound on the probability of safety violation, known prior to deploying or even training the adapted policy; this bound can be tuned by the user to trade off safety and task-specific performance.

Our technical contributions underpinning SPoRt are as follows:

1. A data-driven method for obtaining a bound on the probability that a property (e.g. safety), in general expressed as an LTL formula, is violated for trajectories drawn using a given 'safe' base policy (Section 3).
2. Novel theoretical results that provide, for an episodic, model-free RL setup, a prior bound on the probability of property violation for a new task-specific policy, based on a 'maximum policy ratio' computed with respect to the 'safe' base policy (cf. previous point) (Section 4).
3. A projection-based method for constraining the taskspecific policy to ensure that this prior bound holds (Section 5).
4. Projected PPO, an algorithm for training a new, taskspecific policy, while maintaining a user-specified prior bound on property violation, thus trading off safety guarantees for task-specific performance (Section 6).

We also test SPoRt on a time-bounded reach-avoid property and present experimental results demonstrating the safetyperformance trade-off, as well as comparison of the theoretical prior bound to posterior bounds based on empirical violation rates (Section 7 and 8).

All appendices and code 1 can be found in the supplementary material, which contains all proofs.

2 Models, Tasks and Properties

We consider a model-free episodic RL setup where an agent interacts with an unknown environment modeled as a Markov Decision Process (MDP) [Sutton and Barto, 2014], specified by the tuple ⟨S, A, p, µ, rtask⟩, with a continuous state space S and continuous action space A. p(s′ |a, s) : S × A →

1 Link to code: https://github.com/JacquesCloete/sport.

∆(S) and µ(s) ∈ ∆(S) are the (unknown) state-transition and initial state distributions, respectively. We will consider learning a stochastic policy π(a|s) : S → ∆(A) in a modelfree setup. We use τ p,π st,T = (st, st+1, . . . , st+T )p,π to denote a realization of a trajectory of the closed-loop system with state transition distribution p, starting at state st and evolving for T time steps, using policy π. rtask(s, a) : S × A → R is the task-specific reward, which encourages higher taskspecific performance (for example, max speed or min time).

2.1 Safety as a Temporally-Extended Property

We define safety in terms of satisfaction of a general temporal property φ. We denote that a trajectory τ satisfies property φ (and is therefore safe) by τ |= φ, while τ ̸|= φ indicates that τ violates φ (and is therefore unsafe). SPoRt addresses problems where the objective is to ensure that τ p,π s0∼µ,T |= φ with high probability, while maximizing the task-specific reward rtask. To evaluate the satisfaction of φ we introduce a robustness metric ϱφ, which encodes property violation as a real-valued signal that is non-negative only when τ |= φ.

Definition 1. A robustness metric ϱφ is a function ϱφ(τ) : S n → [-a, b], n ∈ Z+, a, b ∈ R+ such that ϱφ(τ) ≥ 0 only for trajectories τ ∈ S n that satisfy property φ (i.e. τ |= φ).

Any safety property φ can be expressed as a Linear Temporal Logic (LTL) formula [Pnueli, 1977], which ensures the existence of such a metric (see Appendix A.1). Notably, SPoRt extends beyond safety properties to encompass any property φ expressible as an LTL formula - the case study deals with 'reach-avoid' as we shall see. Accordingly, our theoretical results generalize to product MDPs in RL problems under general LTL specifications [Hasanbeig et al., 2023]. While Appendix A.2 provides detailed discussions on these extensions to general LTL formulae and hybrid-state models, for the remainder of the paper (and with no loss in generality) we focus exclusively on safety properties φ within continuousstate MDPs, as defined in Section 2.

3 Data-Driven Property Satisfaction

SPoRt provides a method for adapting an existing safe policy (πbase) so as to maximize some task-specific reward (rtask), without violating a given property φ.

As a first step, let us evaluate the property satisfaction of given traces for a general policy π. Given an initial state distribution, state transition distribution and stochastic policy (µ, p, π), the value of the robustness metric for an associated trajectory, i.e., ϱφ(τ p,π s0∼µ,T ) will be a random variable drawn from some distribution ∆p,π µ and the probability of satisfying the property φ will be encoded by

SPoRt first bounds the probability of property violation under an existing 'safe' base policy πbase, i.e, P{ϱφ(τ p,πbase s0∼µ,T ) ∈ ∆p,πbase µ : ϱφ(τ p,πbase s0∼µ,T ) < 0} ≤ ϵbase using the scenario approach [Campi and Garatti, 2018]; we roll out N scenario trajectories (τ p,πbase s0∼µ,T )i using πbase and record them in buffer Dp,πbase µ = {(τ p,πbase s0∼µ,T )i}Ni=1 . We use Theorem 1 to obtain an upper bound ϵbase on the probability of violating φ:

Theorem 1. If ϱφ((τ p,π s0∼µ,T )i) ≥ 0 for all N scenarios (τ p,π s0∼µ,T )i in Dp,π µ = {(τ p,π s0∼µ,T )i}Ni=1 , then with confidence 1-β, where β = (1 - ϵ)N , the probability of drawing a new scenario τ p,π s0∼µ,T such that ϱφ(τ p,π s0∼µ,T ) < 0 is at most ϵ.

If not all scenarios in Dp,πbase µ satisfy φ, we can leverage results from [Campi and Garatti, 2010] to identify a suitable ϵbase 'under k-constraint removal', as follows:

Corollary 1. Assume k scenarios (τ p,π s0∼µ,T )i in buffer Dp,π µ = {(τ p,π s0∼µ,T )i}Ni=1 are such that ϱφ((τ p,π s0∼µ,T )i) < 0, then with confidence 1 - β, where β = ∑ki=0 (Ni )ϵik (1 - ϵk)N-i , the probability of drawing a new scenario τ p,π s0∼µ,T such that ϱφ(τ p,π s0∼µ,T ) < 0 is at most ϵk.

In both cases, we first collect N scenarios, then choose our confidence 1 - β, and then compute the bound ϵbase.

4 Property Violation under ModifiedMDPs

Once ϵbase is obtained, SPoRt safely trains a task-specific policy πtask so as to maximize the (cumulative) reward rtask. For SPoRt to ensure safe training of πtask, we must upper bound the probability of property violation under πtask, i.e., P{ϱφ(τ p,πtask s0∼µ,T ) ∈ ∆p,πtask µ : ϱφ(τ p,πtask s0∼µ,T ) < 0}, by the probability of property violation under πbase, i.e., P{ϱφ(τ p,πbase s0∼µ,T ) ∈ ∆p,πbase µ : ϱφ(τ p,πbase s0∼µ,T ) < 0}, which is upper bounded by ϵbase. To do so, we first construct this bound for general (µ1, p1, π1) and (µ2, p2, π2), and then set (µ1, p1, π1) = (µ, p, πbase) and (µ2, p2, π2) = (µ, p, πtask).

Let S0, . . . , ST ⊆ S be a sequence of arbitrary subsets of the state space for each time step in the episode. We begin by characterizing the probability that a sampled trajectory τ p,π s0∼µ,T = (s0 ∼ µ, s1, . . . , sT )p,π is such that s0 ∈ S0, . . . , sT ∈ ST as a forward recursion, based on work in [Soudjani and Abate, 2013; Soudjani and Abate, 2015]. Let 1St (s) : S → {0, 1} be the indicator function for s ∈ St, and define functions W µ,p,π t (s) : S →R+, characterized as

It holds that P{τ p,π s0∼µ,T : s0 ∈ S0, . . . , sT ∈ ST} = ∫S W µ,p,π T (s)ds. We then use Theorem 2 to bound the probability that trajectory τ p2,π2 s0∼µ2,T remains within S0, . . . , ST throughout the episode in terms of the probability that trajectory τ p1,π1 s0∼µ1,T remains within the same S0, . . . , ST :

Theorem 2. Suppose that, for a set of coefficients αt ∈ R+, we could constrain µ2, p2 and π2 so as to enforce the following bounds for all t = 1, . . . , T :

It thus holds that

We want to make this bound as tight as possible, which is done by minimizing αt subject to Equation (1) for all s′ ∈ S and t = 1, . . . , T. Solving this problem is non-trivial due to p1 and p2 being unknown in a model-free setup. To this end, we introduce Theorem 3 to obtain a feasible solution by constraining p2 and π2 in terms of p1 and π1:

Theorem 3. Suppose the following constraint holds:

Assuming stationarity, under this constraint the bound is minimized when αt = α for all t = 1, . . . , T . Note also that under this constraint, we find that α ≥ 1.

It is important to observe that, while correct, this bound can be very conservative for applications with large episode length T ; a discussion on this conservativeness can be found in Appendix C.1. Alternative bounds from literature suffer from similar blowup [Soudjani and Abate, 2012; Soudjani and Abate, 2015].

Using the results from Theorem 2 and 3, we can now derive Theorem 4 to obtain a bound on the probability of property violation for (µ2, p2, π2) in terms of (µ1, p1, π1):

Theorem 4. Suppose that

It thus holds that

The proof for Theorem 4 considers all sequences S0, . . . , ST that correspond to property violation events, and sums over their probabilities under (µ2, p2, π2).

Now let (µ1, p1, π1) = (µ, p, πbase) and (µ2, p2, π2) = (µ, p, πtask); we see that constraining the policy ratio πtask(a|s) πbase(a|s) ≤ α for all a ∈ A, s ∈ S is sufficient to ensure that the bound holds. The total multiplicative increase on the upper bound for property violation going from πbase to πtask is thus αT , with the bound being ϵtask = ϵbaseαT .

This is a significant result, since we can now provide a prior bound on the probability of property violation for any πtask, based entirely on the probability of property violation for πbase and the maximum policy ratio between πtask and πbase across all states and actions, with no required knowledge of µ, p or the constraints under which property φ holds, so long as initial state distribution and state transition distribution remain the same. Furthermore, by adjusting the value of α we can directly trade off safety guarantees for deviation from the base policy, which can be leveraged to achieve a boost in task-specific performance.

However, note the exponential relationship between T and ϵtask; this means that, for even small increases of α from 1, our prior bound will always eventually explode to the point of becoming trivially 1 if T is made sufficiently large. Thus, if the prior bound is to be used, SPoRt is best suited to control problems with a low maximum episode length T . In practice, however, there are ways to overcome or otherwise mitigate this limitation, as we will see later in Section 7.

Note that Theorem 4 also provides a bound when µ2 and p2 differ from µ1 and p1. Thus, our theoretical results can also be applied to robust control settings for perturbed systems; see Appendix C.2 for further discussion.

5 Constraint Satisfaction for a Task Policy

For Theorem 4 to hold, we must maintain the hard constraint πtask(a|s) πbase(a|s) ≤ α for all a ∈ A, s ∈ S. Given a πtask, we can achieve this by projecting πtask onto the feasible set of policy distributions Πα,πbase at each time step:

While πtask represents the unconstrained (and potentially unsafe) task-specific policy network that we train or are provided, the projection πproj is a policy that we can safely roll out, including during training. Note that α defines the level sets of Πα,πbase , which is always non-empty for α ≥ 1 (since πbase itself is a valid πtask). Let us now look at how to simplify the computation of this projection step - we will henceforth assume diagonal Gaussian policies:

Assumption 1. Both πbase and πtask are diagonal Gaussian policies: π(a|s) = N (a; µ, Σ), where Σ = diag (σ2 ), and where the policy means µ(s) and standard deviations σ(s) are functions of MDP state and evaluated at each time step using (for example) a policy neural network.

At each time step, we can obtain πproj using Theorem 5:

Theorem 5. Assuming diagonal Gaussian policies and using KL divergence as the distance metric for projection, the means and standard deviations of projected policy πproj can be computed from πbase and πtask by solving the following convex optimization problem at each time step:

It is interesting to note that the standard deviations of πproj must all be strictly lower than those of πbase, in other words we require πproj to be less exploratory than πbase. This is intuitive considering that we aim to maintain safety by remaining 'close' to πbase. We also note that making πbase more exploratory (with a larger standard deviation) generally results in a larger Πα,πbase , allowing for greater policy change and thus task-specific performance boost by πproj; see Appendix C.3 for details.

We implement and solve this problem using CVXPY [Diamond and Boyd, 2016; Agrawal et al., 2018]; on a standard desktop PC the compute time remains in the order of milliseconds for even high-dimensional action spaces, making this method feasible for many RL applications. See Appendix D for implementation details.

6 Training for Tasks, Under a Bound on Property Violation

Above, we have shown how to obtain πproj from πbase and πtask. We have implicitly assumed that we already have πbase and a corresponding bound on probability of property violation ϵbase. There are many practical applications where we would also have access to πtask: as an example from robotics, we may have trained πtask in simulation, where safety is noncritical, but now want to safely deploy this policy on the real robot, for which a tried-and-tested πbase is known.

However, other applications may require that we train πtask to maximize cumulative rtask while maintaining a prior bound on safety during training, for example if a suitable simulator to train good policies for the real environment is not available. In this case, we would first choose an acceptable α ≥ 1 by rearranging ϵtask = ϵbaseαT ≤ ϵmax, where ϵmax is a maximum acceptable probability of property violation, and we then learn πtask (initialized as πbase) while only ever deploying πproj during training so as to ensure the bound holds.

Note that there is a distinction between what we train, πtask, and what we actually deploy, πproj, during training. To overcome this, SPoRt uses Projected PPO, outlined in Algorithm 1, to train πtask for tasks with continuous state-action spaces. The algorithm is inspired by clipped PPO [Schulman et al., 2017] but clips the surrogate advantage based on the policy ratio between the new πtask and previous πproj, rather than the previous πtask; we store the (log-)probabilities of πproj at each time step during data collection to avoid needing to recompute πproj during gradient updates. The advantage estimates are also computed using samples collected by deploying πproj rather than πtask.

The clipping sets the gradient to zero beyond the maximum/minimum allowed policy ratio of πtask to πproj, preventing πtask from drifting away beyond clipping ratio ξ of πproj, in this way, we maintain an acceptable amount of mismatch between πtask and πproj and stop πtask from drifting away from the feasible set of allowed policy distributions.

We also warm-start the value function network for rtask before training πtask, since an accurate value function is important for effective fine-tuning. This is done by training the value function using clipped PPO until convergence while keeping the policy network weights fixed as those of πbase.

Algorithm 1 Projected PPO

Input: θbase, α, T

• 1: Obtain initial critic parameters ϕ0 by warm-starting the critic using PPO (episode length T ) with πbase
• 2: Initial task-specific policy parameters θ0 ← θbase
• 3: for k = 0, 1, 2, . . . do
• 4: Collect trajectories Dk = {(τT )i} by running projected policy πproj,θk = projΠα,πbase (πtask,θk ) in the environment. Store πproj,θk (·|st) ∀st ∈ τT , τT ∈ Dk
• 5: Compute rewards-to-go ˆRt
• 6: Compute advantage estimates ˆ At using Vϕk
• 7: Update task-specific policy:

8: Fit value function:

9: end for

7 SPoRt in Action: Case Studies

We apply SPoRt to a reach-avoid property, wherein the agent must reach a goal within a time limit while avoiding collision with a hazard up until the goal is reached. Such an objective is standard within the control and verification literature, and indeed can be used to model many real-world problems. For completeness, we provide the LTL formula and robustness metric for the time-bounded reach-avoid property in Appendix E.1, with a reminder that SPoRt can be used over general LTL specifications.

We implemented the environment in Safety Gymnasium [Ji et al., 2023] using a point agent and with the goal and hazard being green and red circular regions, respectively (cf. Figure 1 and 2). This setup was chosen since it allows for easy interpretability of results while remaining a reasonable abstraction of a real robotic navigation task using a skid-steering mobile robot with a LiDAR sensor; a description of the MDP observation and action spaces can be found in Appendix E.2. The episode is reset if the agent enters the hazard or goal sets, or if the maximum episode length is exceeded. To mitigate the exponential relationship between maximum episode length T and bound ϵtask, we reduced the control frequency ten-fold from the default (up to 100 simulation steps per environment step), in-keeping with the observation in Section 4 that SPoRt is best suited to control problems with low T .

πbase was trained so as to achieve a high probability of satisfying the property (reach-avoid), while remaining fairly exploratory. This was achieved by training πbase using Soft Actor-Critic (SAC) [Haarnoja et al., 2018] with a sparse reward scheme (corresponding to property satisfaction across an entire episode). Alternative synthesis schemes are possible. Further details on training πbase can be found in Appendix E.3. Once trained, around N = 10000 scenarios were collected to determine ϵbase with high confidence (β = 1e-7, see [Campi and Garatti, 2018]) using Corollary 1. Note that while training πbase the maximum episode length was set to T = 100 yet by the end of training the average episode length was much lower, at around T = 14. Thus, to keep the value of ϵtask = ϵbaseαT as low as possible, the maximum episode length was reduced to T = 21 after training πbase (with ϵbase computed using scenarios of this length). Further discussions (including how SPoRt can be modified to do this automatically) can be found in Appendix E.4.

For our case studies we trained πtask to reach the goal as quickly as possible: accordingly, rtask was the standard dense reward for reaching a goal used by Safety Gymnasium. Notice that the set task (and corresponding reward) clearly leads to a potential violation of the property (reach-avoid) of interest. We consider two separate cases, as follows:

Case 1: Pre-Trained Task Policy. πtask is trained separately without any consideration for property violation. As a result, under πtask the agent quickly drives directly towards the goal with no hazard avoidance. This represents applications where πtask has been pre-trained in an environment where safety is not critical (for example in a robotics simulator) and we want to safely test it on the real environment (see Section 5).

Case 2: Task Policy Trained Using Projected PPO. This represents applications where we have πbase and ϵbase (as from above) and now want to fine-tune our policy to be faster (thus obtaining πtask), while maintaining an acceptable given bound on property violation (see Section 6).

Note that we use the same πbase for both cases.

8 SPoRt Report: Results and Discussion

Both cases were tested for 1000 episodes at different values of α (such that πproj = projΠα,πbase (πtask)), ranging from α = 1 (i.e. πproj = πbase) to the point where the empirical violation rate exceeded a threshold. For Case 2, πtask was trained until convergence at each value of α, prior to testing. Further details on training πtask for both cases can be found in Appendix E.3. Action seeding for each episode was controlled across different values of α and across the different cases, so all results depend on α and the training of πtask. From our results we seek to answer the following questions (Qs):

1. Does increasing α trade off safety for performance?
2. How does performance compare between Case 1 and 2?
3. How conservative is the prior bound ϵtask = ϵbaseαT ?

Figure 1 presents sample distributions of episode trajectories for both cases over different values of α. In both cases, we see that as α increases, the trajectories bend more tightly around the hazard, suggesting a reduction in action variance, 2 as well as an action mean that takes the agent closer to the hazard. In fact, in Case 1 for α = 100, the agent's mean trajectory crosses the hazard. Thus increasing α is shown to trade off safety for performance, answering Q1.

However, we can also appreciate the difference between Case 1 and Case 2: while Case 1 produces an action mean that drives the agent through the hazard for α = 100, Case 2 instead produces a more reduced action variance, while retaining an action mean that keeps the agent outside the hazard, as expected. Thus, to answer Q2, we see that Case 2 allows us to provide better performance, whilst retaining a 'better behaved' (and indeed 'safe') πproj compared to Case 1; accordingly, we argue that since training πtask using Projected PPO deploys πproj during training, πtask learns to optimize performance of πproj compared to na¨ıvely training πtask a priori with no consideration of how πproj will perform.

Figure 2 presents a more detailed view of the agent behavior over an example episode for Case 2, for α = 5 (representing a compromise between safety and performance). Looking at mean turning velocity over the episode, we see that while both πbase and πtask drive the agent clockwise around the hazard, πtask induces sharper turning, taking the agent closer to the hazard and drawing a tighter, shorter curve while maintaining the same or faster forward drive force. However, this sharper turning is constrained such that πproj always lies within the α = 5 level set (see Section 5). Note at πtask applies a reduced forward drive force at the very start of the episode compared to πbase, which makes sense given the agent is initially pointing away from the goal, so πtask reduces episode length by first pointing the agent closer to the agent before driving forward. Appendix E.5 provides a similar analysis for Case 1.

2 Recall we work with diagonal Gaussian policies, hence the consideration of action mean and variance.

Figure 3a presents violation probabilities over different values of α for both cases. The most striking observation is the conservativeness of prior bound ϵtask, which grows exponentially from ϵbase = 0.009 at α = 1 to ϵtask = 1 at around α = 1.25 (beyond which point the bound is no longer useful), yet the posterior bounds on property violation (obtained by applying Corollary 1 to the N = 1000 test samples) remain at around 0.025 over this range. We do also see exponential growth in the posterior bound for Case 1, but this happens over a completely different scale (α = 1 to 100 rather than 1 to 1.25). The posterior bound for Case 2 remains at 0.025 for even α = 100, suggesting much safer behavior compared to Case 1 for the same α.

Figure 3b presents the mean and standard deviation episode length for successful trajectories over different values of α for both cases. For both we see a similar reduction in mean and standard deviation as α increases until around α = 10, at which point the mean plateaus (to around 12.0 at α = 100, for 14.3% total reduction) but standard deviation shrinks for Case 2 while the mean continues to decrease for Case 1 (to around 11.3 at α = 100, for 19.3% total reduction)); this comes at the cost of substantially increased violation rate for Case 1, shown by Figure 3a. Another important observation is that while we know from Figure 3a that ϵtask is very conservative, we do see a measurable (2.1%) reduction in mean episode length for both cases from 14.0 at α = 1 (ϵtask = 0.009) to around 13.7 at α = 1.12 (ϵtask = 0.1, a fairly sensible (if high) value). Appendix E.5 provides the same figures but zoomed in to the scale across which ϵtask ≤ 1.

These observations further confirm our earlier answers to Q1 and Q2, whilst now we also have an answer for Q3: the prior bound can be very conservative, though it is possible to see measurable improvement in performance while the bound remains fairly sensible.

9 Limits to Sporting SPoRt

The most obvious limitation of SPoRt is the conservativeness of the prior bound ϵtask = ϵbaseαT , which prevents significant policy changes if the bound is to be used to guarantee safety. This conservativeness also results in the limitation of needing T to be as low as possible, making SPoRt unsuitable for applications where T is high (though we have seen ways to mitigate this limitation). Another limitations include the reliance on collecting many scenarios to obtain a useful bound ϵbase, which may not be practical for some applications, as well as the requirement of stochastic policies (and, ideally, a fairly exploratory πbase to achieve noticeable policy change). Despite these theoretical limits, we have displayed the usefulness of the end-to-end architecture of SPoRt in meaningful simulation studies, which are promising for upcoming real-world implementations of SPoRt.

10 Conclusions

We have presented novel theoretical results that provide a prior bound on the probability of (safety) property violation for a task-specific policy in a model-free, episodic RL setup, based on a new 'maximum policy ratio' established vis-a-vis a given 'safe' base policy. Based on these bounds, we have presented an end-to-end architecture, SPoRt, which combines a data-driven approach for obtaining such a bound for the base policy with a projection-based approach for training the task-specific policy while maintaining a user-specified prior bound on (safety) property violation, thus trading off safety guarantees and task-specific performance. In view of promising experimental simulation results, future work will focus on reducing the conservativeness of the prior bound, to improve its utility in practical real-world applications.

Acknowledgments

This work was supported by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems [EP/S024050/1].