DEPLOY-RL: Active Boundary Discovery and Conservative Certification for Deployable Reinforcement Learning in Safety-Critical Continuous Processes

1 Introduction

Reinforcement learning (RL) shows strong simulated performance in industrial process optimization, yet rarely reaches production in safety-critical settings. The barrier is not algorithmic performance but the absence of principled deployment criteria with finite-sample risk guarantees. Classical PID/MPC is stable but conservative; our framework enables a hybrid strategy: deploy RL when certified safe and fall back to robust PID/MPC otherwise. This setting differs from offline benchmark selection: the operator must make a single deployment decision for a trained controller under a specified operational stress distribution, where a false positive deployment can cause safety violations while an abstention only incurs fallback cost.

∗Corresponding author.

This gap represents a fundamental bottleneck for AIenabled critical technologies. Existing safe RL methods (CPO [Achiam et al., 2017], Lyapunov-based approaches [Chow et al., 2018], Recovery RL [Thananjeyan et al., 2021]) address training-time safety but assume deployment. Black-box validation [Corso et al., 2021; Feng et al., 2023] discovers failures but lacks statistical deployment guarantees. Recent conformal approaches [Lindemann et al., 2023; Luo et al., 2024] provide prediction intervals but not policylevel certification decisions. A critical question remains unanswered: how should an operator decide whether to deploy a trained policy, defer to fallback control, or reject deployment entirely, all with statistical guarantees?

DEPLOY-RL is a post-training certification layer applicable to any trained policy. Its key insight is that deployment certification is a decision-oriented boundary discovery problem: instead of uniformly reconstructing failure boundaries, it samples where boundary uncertainty most affects deployment decisions under an explicit contract. This reframes active testing from "find the boundary accurately everywhere" to "resolve the boundary mass that changes the deploy/no-deploy decision."

Contributions. Our work makes three contributions:

• Decision-oriented boundary discovery (Section 4). A contract-coupled acquisition function (Eq. 3) prioritizes certification confidence under the deployment contract and gives monotone ambiguity reduction (Proposition 1), yielding ≈ 2× sample efficiency empirically.
• Conformal deployment certification (Section 5). We cast deployment as selective conformal risk control under an explicit contract, with finite-sample false-go bounds (Proposition 2) and data-separation conditions (Lemma 1).
• Three-way industrial fallback. The rule outputs Deploy, No-Deploy, or Abstain; abstention triggers fail-

STAGE 1: Boundary Search

False-Go: P[Deploy ∧ Fail] ≤ α under Dc with (A1)-(A3)

Figure 1: DEPLOY-RL pipeline. Boundary search targets certification-critical regions, conformal calibration produces conservative risk bounds, and the final rule returns Deploy, Abstain, or No-Go with PID/MPC fallback.

safe PID/MPC fallback.

We empirically validate DEPLOY-RL on papermaking (proprietary digital twin) and the Tennessee Eastman benchmark (public) using deployment-centric metrics: false-go rate, coverage, and certification cost.

Pipeline overview (Figure 1). DEPLOY-RL has three stages: contract-aware boundary search, conformal calibration of ˆ pU , and a Deploy/No-Deploy/Abstain rule with guarantee P[Deploy ∧ Unsafe] ≤ α under Dcontract.

2 RelatedWork

Safe RL and deployment verification. Safe RL methods (CPO [Achiam et al., 2017], Lyapunov-based [Chow et al., 2018], Recovery RL [Thananjeyan et al., 2021]) address training-time safety but assume deployment will occur. Black-box validation [Corso et al., 2021; Feng et al., 2023], runtime failure detection [Xu and others, 2025], probabilistic safety filters [Tabbara et al., 2025], and conformal control-barrier approaches [Lindemann et al., 2023; Luo et al., 2024] improve safety evidence, but do not provide post-training policy-level deploy/no-deploy certification with false-go control.

Conformal risk control and selective prediction. Classical conformal prediction [Vovk et al., 2005; Angelopoulos and Bates, 2021] provides coverage guarantees; recent work extends it to risk control [Angelopoulos et al., 2024], decision-oriented conformal methods [Fisch et al., 2024], covariate shift [Barber et al., 2024], feedback adaptation [Wang and Ning, 2025], and adaptive abstention [Kumar et al., 2025]. Unlike general conformal risk control, we target deployment decision correctness: contract-aligned boundary discovery, data separation despite adaptive sampling, and three-way fallback decisions. Standard conformal statements certify set coverage, but industrial deployment needs a policylevel statement of the form "deploy only when false-go probability is controlled."

Active boundary discovery. Level-set estimation [Bryan et al., 2006; Gotovos et al., 2013], randomized straddle sampling [Inatsu et al., 2024], stability-informed acquisition [Hirt et al., 2024], ActSafe [Sukhija et al., 2024], and CROWS [Muthali and others, 2024] target geometry-oriented boundary recovery. Our acquisition instead weights boundary accuracy by impact on the deploy/no-deploy decision under Dcontract, giving monotone ambiguity reduction and ≈ 2× sample reduction (Table 2).

Industrial process control and RL deployment. Processcontrol RL benchmarks [Shin et al., 2019; Bloor et al., 2024; Downs and Vogel, 1993] enable evaluation, but deployment remains rare because certification is missing; DEPLOY-RL supplies that decision layer.

3 Problem Setting

We consider safety-critical continuous processes controlled by an RL policy π under stress parameters x ∈ X ⊂ Rd (noise, delay, drift). Our goal: minimize unsafe deployments (false-go) while maintaining useful coverage.

Deployability score. We define fπ(x) = 1-(wvv(π, x)+ wcCVaR0.05(π, x) + wrtrec(π, x)/H), where v is violation rate, CVaR (conditional value at risk) captures tail risk, and trec is recovery time. Weights (0.5, 0.3, 0.2) are informed by domain knowledge; ±20% variations yield ±1.2pp false-go (see Limitations). A failure event occurs when fπ(x) < h (threshold h = 0.8).

Key metrics. False-go: fraction of deployed policies that fail. Coverage: fraction receiving definitive decisions. Abstention: triggers PID/MPC fallback. We use ϵ = 0.05 (max failure probability) and α = 0.10 (conformal level).

4 Certification-Aware Boundary Discovery

4.1 FromLevel-Set Estimation to Certification-Driven Sampling

Grid/random stress testing wastes budget far from failure boundaries, while standard level-set estimation [Gotovos et al., 2013] optimizes boundary reconstruction rather than certification accuracy. Because deploy/no-deploy depends on whether pπ > ϵ, two policies with similar boundary geometry can require different certification confidence; this motivates a certification-aware acquisition function.

Let fπ(x) : X → R be a deployability score (higher is safer). Given threshold h, define the safe region and boundary:

4.2 Certification-Aware Acquisition Function

We model fπ(x) with a GP surrogate, assuming fπ : X → R is continuous and lies in the RKHS of the Mat´ern kernel (scalable to d ≈ 5-10 stress dimensions; see Limitations for higher-dimensional extensions). We balance (i) boundary proximity, (ii) uncertainty reduction, and (iii) certification relevance: regions most affecting deploy/no-deploy.

Let Dcontract denote the deployment contract distribution (the operational stress distribution). We define the certification-aware acquisition function:

where wcontract(x) = pDcontract (x)/puniform(x) is an importance weight that upweights stress scenarios likely under the deployment contract. The hyperparameters β and γ control exploration-exploitation and certification focus respectively.

Normalization. We divide |µ(x) - h| by the deployability range, normalize σ(x) by σf , clip wcontract(x) to [0, 5], and set puniform(x) = 1/vol(X). Unlike straddle acquisition (γ = 0), Eq. 3 focuses simulations on boundary regions that matter for the deployment contract.

4.3 Adaptive γ Scheduling

We propose an adaptive scheduling strategy:

where t is the iteration, γ0 = 0.5 is the initial weight, γmax = 2.0 is the asymptotic weight, and τ = 50 controls the transition rate. This schedule shifts from broad exploration to certification-focused refinement.

Theoretical justification.

Proposition 1 (Decision Ambiguity Reduction: Semi-Empirical Bound). Define the decision ambiguity set At(δ) = {x ∈ X : |µt(x) - h| < δ}. Let µamb,t = Ex∼Dcontract [I[x ∈ At(δ)]] denote the contract-weighted ambiguity measure. Assume (C1) Boundary separability: ∃δ0 > 0 such that P[|fπ(x)-h| < δ0] < 1 under Dcontract. Under certificationaware acquisition (Eq. 3) with γ > 0 and δ < δ0:

where η = p0·cd·(ℓ/diam(X ))d /σf > 0. Here ℓ is the kernel lengthscale, σf the kernel variance, cd = πd/2 /Γ(d/2 + 1) is the volume of the unit d-ball, and p0 = P[|fπ(x) - h| > δ | x ∈ At(δ)] is the boundary escape probability. Note: p0 is estimated empirically (p0 ∈ [0.3, 0.5] in our domains); we therefore term this a semi-empirical bound. Theoretical lower bounds on p0 require additional regularity (e.g., |∇fπ| > c > 0 on the boundary).

Proof sketch. The acquisition selects xt with high boundary proximity, uncertainty, and contract weight; the GP update reduces local variance. Under GP regularity and (C1), observations near the boundary reveal safe/unsafe status with probability p0 > 0, removing mass from At+1(δ), while contract weighting concentrates this reduction on Dcontract-relevant regions.

Implication. The acquisition directly minimizes contractweighted decision ambiguity rather than uniformly refining the boundary, yielding ≈ 2× sample reduction (152 vs. 280320 rollouts; Table 2). Its convergence follows the level-set rate in [Gotovos et al., 2013]: µamb,t → 0 as B → ∞. Equivalently, contract-coupled acquisition concentrates information gain on deployment-critical regions, explaining why it can reduce certification cost without improving boundary reconstruction uniformly.

Boundary search procedure. Boundary search repeatedly queries acert (Eq. 3), simulates fπ(x), updates the GP, and returns ˆ Sπ, ∂ ˆ Sπ, and features zπ (Figure 2).

5 Conformal Risk Control for Deployment Certification

5.1 Deployment Contract and Distribution Separation

A key challenge is the mismatch between distributions:

• Stress-test distribution Dtest: The distribution from which we actively sample during boundary discovery (biased toward boundaries).
• Deployment contract distribution Dcontract: The operational specification defining stress conditions the policy must handle in production.

Because active boundary samples are biased, we use contract-aligned calibration: labels are evaluated under Dcontract rather than Dtest. This separation is essential: using actively sampled boundary points for calibration would overrepresent rare stress states and invalidate exchangeability with operational deployment.

5.2 Certifier Learning Problem

For policy π, boundary search produces features

summarizing boundary coverage, CVaR, and recovery statistics; certifier gθ outputs ˆ pπ ∈ [0, 1] under Dcontract.

Contract-aligned labels. For each calibration policy, the empirical label is computed under Dcontract:

5.3 Conformal Risk Control with False-Go Guarantee

We cast deployment certification as conformal risk control [Angelopoulos and Bates, 2021] for the false-go rate.

Nonconformity score (asymmetric by design). For calibration policy π with predicted risk ˆ pπ and true label yπ = I[ ˜ pπ > ϵ], we define:

This asymmetric score penalizes underestimation (unsafe predicted safe); overestimation only increases abstention, so ˆ pU is conservative.

Conservative upper bound. Let n = |C| denote the calibration set size. Let q1-α be the ⌈(1-α)(n+1)⌉-th smallest value of {sπ}π∈C. The conformal upper bound is:

Decision rule and probability space. Given a disjoint calibration set, the certifier prediction and conformal upper bound are computed at the policy level; the decision is:

Since q1-α ≥ 0, the cases are mutually exclusive. The guarantee is policy-level: labels yπ = I[˜ pπ > ϵ] are computed fromM i.i.d. contract scenarios.

Proposition 2 (Marginal False-Go Control). Let C = {π1, . . . , πn} be calibration policies and πn+1 a test policy, all drawn exchangeably from a policy distribution P. Define yπ = I[˜ pπ > ϵ] where ˜ pπ is the empirical failure rate under Dcontract (Eq. 7). The following assumptions are sufficient:

• (A1) Policy exchangeability: (π1, yπ1 ), . . . , (πn+1, yπn+1 ) are exchangeable. (Violated by: policy selection bias, nonstationary training.)
• (A2) Data separation: Calibration labels {yπi } use scenarios disjoint from boundary discovery (D1) and certifier training (D2). (Violated by: reusing boundary search samples.)
• (A3) Fixed certifier: gθ is trained on data disjoint from C ∪ {πn+1}. (Violated by: adaptive certifier updates.)
• (A4) Meaningful threshold: ϵ < 0.5 (satisfied by all practical safety contracts). (Required for the contradiction argument.)

Under (A1)-(A4), the split-conformal decision rule satisfies:

Interpretation: This is a marginal policy-level guarantee; conditional guarantees require larger |C| or density assumptions [Vovk et al., 2005]. The bound applies to empirical labels fromM contract scenarios (Eq. 7); with M ≥ 500, label noise is small relative to α.

Proof sketch. Under (A1), policy-label pairs are exchangeable, and split conformal gives P[sπn+1 > q1-α] ≤ α. Deploy implies ˆ pπn+1 ≤ ϵ - q1-α. If yπn+1 = 1, then sπn+1 = 1-ˆ pπn+1 ≥ 1-ϵ+q1-α > q1-α, contradicting the conformal event. Thus false-go is bounded by the conformal tail probability; (A2)-(A4) ensure the required independence and threshold conditions.

Procedure. Compute calibration scores {sπ′}, extract q1-α, set ˆ pU = ˆ p + q1-α, and decide by Eq. 10. Data splits are disjoint: (D1) boundary discovery, (D2) certifier training, (D3) calibration.

Lemma 1 (Calibration Validity under Adaptive Boundary Discovery). For each policy πi in calibration set C, let zπi = ϕ(D(i) 1 ) be features from adaptive boundary discovery, and yπi = I[˜ pπi > ϵ] be the label fromM i.i.d. contract scenarios. If:

• (L1) Policy independence: Policies {πi} are drawn i.i.d. from P.
• (L3) Fixed feature map: ϕ(·) is a deterministic function (e.g., summary statistics: boundary volume, CVaR, coverage). (L4) Fixed certifier: gθ is trained on policies Ptrain ∩ C = ∅. Then the nonconformity scores {sπi }n+1 i=1 are exchangeable, ensuring assumptions (A1)-(A3) of Proposition 2; assumption (A4), ϵ < 0.5, is independently required and is satisfied for all practical safety contracts (ϵ = 0.05 here).
• (L2) Label independence: Each yπi is computed from M fresh scenarios {x(i) j }Mj=1 ∼ DM contract , disjoint from D(i) 1 .

Proof. Labels yπi depend only on fresh scenarios (L2), not adaptive D(i) 1 . By (L1), policies are i.i.d.; by (L3)-(L4), features zπi and predictions ˆ pπi are i.i.d. Thus scores {sπi } are exchangeable.

Key insight. Adaptive search within each policy does not break exchangeability across policies.

Deployment contract: formal definition. We propose a standardized contract specification for industrial RL certification:

Definition 1 (Deployment Contract). A deployment contract K = (Dcontract, ϵ, α, πfallback) specifies:

• ϵ: maximum acceptable failure probability (default: 0.05)
• Dcontract: stress distribution over operational scenarios (x ∼ D)
• α: conformal significance level (default: 0.10)
• πfallback: fallback controller for abstention (e.g., PID, Robust MPC)

The certifier must satisfy: P[Deploy ∧ fπ(x) < h] ≤ α under x ∼ Dcontract.

Scope and validity. Dcontract mixes nominal (70%) and stress (30%). Guarantees are contract-relative; validity requires i.i.d. calibration, data separation (D1-D3), and n ≥ 50.

6 Experimental Setup

6.1 Domains

Papermaking Digital Twin

We use a proprietary papermaking digital twin [Tao et al., 2018] with 18 sensors, 3 actuators, H = 480 one-minute steps, and basis-weight/moisture constraints; fidelity was validated on held-out LOTs. Each episode corresponds to one LOT cycle, and safety constraints enforce basis weight within

±2% and moisture within ±1.5% of target setpoints. Simulator fidelity was checked by matching marginal distributions, autocorrelation structure, and step-response behavior against held-out production traces.

Tennessee Eastman Process (TEP)

We use the standard pyTEP benchmark [Downs and Vogel, 1993; Ricker, 1996]: 52 measured variables, 12 manipulated variables, 21 IDV disturbances, and 960 three-minute steps per episode.

Data splits. Policies are split 60/20/20; certifier training uses 200 policies from a separate pool, distinct from the 500 calibration policies, and stress scenarios are split separately.

6.2 Baselines

We compare 14 baselines: classical control (PID, MPC, Robust MPC [Rawlings et al., 2017], Chance-Constrained MPC), safe RL (PPO, CPO [Achiam et al., 2017], CVPO [Liu et al., 2022], PPO-Lagrangian, Recovery RL [Thananjeyan et al., 2021]), and certification gates (Grid+Rule, AST [Corso et al., 2021], Deep Ensemble, BO-Worst, Straddle [Gotovos et al., 2013]). Fallback uses Robust MPC with 15% constraint tightening.

6.3 Implementation

Stress space: Papermaking uses noise σ ∈ [0, 0.15], delay τ ∈ [0, 5], drift δ ∈ [0, 0.2]; TEP uses IDV 1-21 with noise and valve stiction. Boundary search: n0=50, B=200, GP (Mat´ern 5/2, lengthscale ℓ=0.3, variance σ2 f =1.0, noise σ2n=0.01), β=2.0; hyperparameters re-optimized via marginal likelihood every 25 iterations. Certifier: 3-layer MLP (128-64-32, ReLU), trained with Adam (lr= 10-3 , weight decay= 10-4 ), batch size 64, 100 epochs, early stopping (patience=10); |C|=500, α=0.1, ϵ=0.05. Calibration policies: 18 hyperparameter configurations (learning rate, hidden size, entropy) across 28 seeds yield ≈ 500 independent policies. Reproducibility uses seeds {42, 123, 456, 789, 1024}; TEP is public, while papermaking is proprietary with a synthetic 24dim surrogate reproducing 87% of boundary characteristics (R2=0.87). This grid-based policy generation is a practical approximation to an exchangeable calibration pool; validity relies on independent training runs and disjoint evaluation scenarios rather than identical hyperparameter settings.

6.4 Metrics

Metrics: False-Go (FG), Coverage, Recovery Time, FG@85% coverage, Certification Cost, and Fallback Burden.

7 Results

DEPLOY-RL reduces unsafe deployments by 49% while retaining 88.6% coverage (Table 1).

7.1 Main Deployment Safety Table

Table 1 summarizes deployment safety metrics.

∗∗∗ p < .001, ∗∗ p < .01 (bootstrap). FG: -4.2pp [CI: -5.1, -3.3].

Table 1: Deployment safety (avg. Papermaking+TEP). FG: false-go (%). Cov: coverage (%). Rec: recovery time. AUC: risk-coverage curve. Improvement vs. Deep Ensemble Gate.

Per-domain breakdown. Papermaking achieves FG 3.7±1.1%, Cov. 89.8±1.2%, AUC 0.95; TEP achieves FG

↓33% ↓18% ↓30%

∗∗∗ p < 0.001, ∗∗ p < 0.01 vs. best baseline.

Table 2: Additional deployability metrics for fair comparison. FG@C: false-go at matched 85% coverage (via threshold adjustment). CC: certification cost (simulation rollouts during boundary search). FB: fallback burden (% episodes triggering PID/MPC). "- ": methods without explicit certification stage use all training rollouts.

5.1±1.2%, Cov. 87.2±1.3%, AUC 0.93. Both meet the target (< 6% FG, > 85% coverage), and gains over Deep Ensemble are consistent across domains. Papermaking has lower absolute false-go because its dynamics are smoother and boundary features are more informative; TEP remains harder due to heterogeneous IDV disturbances.

7.2 Additional DeployabilityMetrics

Table 2 reports matched-coverage false-go, certification cost, and fallback burden.

7.3 Robustness, Trade-offs, and Efficiency

Figure 3 shows conservative shift behavior, the only FG<5%/coverage>85% frontier point, and a 5% target reached at B≈150 vs. B>300 for random sampling.

Ablation. Removing contract weighting, active sampling, or abstention raises false-go to 8.2%, 12.0%, and 16.4%.

Statistical significance. Paired bootstrap vs. Deep Ensemble confirms false-go -4.2pp (95% CI [-5.1,-3.3], p < 0.001) and coverage +8.8pp (95% CI [7.2, 10.4], p < 0.001); exploratory comparisons use Holm-Bonferroni.

Transfer/robustness. Cross-domain false-go rises to 1215%; 30% contract mismatch gives 19.4% without recalibration and 9.2% with rolling recalibration; block conformal reduces temporally correlated false-go from 8.2% to 6.1%.

8 Discussion and Limitations

Fair comparison and sim-to-real gap. At matched 85% coverage, DEPLOY-RL attains 4.8% false-go vs. 8.2-14.2% for baselines. Guarantees are simulator-relative, so deployment should pair twin-fidelity validation, rolling recalibration, and shadow mode; DEPLOY-RL is the certification gate, not a standalone sim-to-real solution. Abstention switches to PID/MPC within one control cycle. Under moderate shift, false-go degrades from 4.4% to 7.2% at 10% shift and 19.4% at 30% shift, motivating recalibration when contract mismatch is detected.

Limitations. (A1) block conformal handles temporal correlation; (A2) episode-level labels only; (A3) heuristic weights show ±1.2pp sensitivity; (A4) domain-specific retraining is required; (A5) GP scales to d≈5-10 (sparse GP for higher d); boundary discovery uses simulation only. These limitations are explicit contract assumptions rather than hidden guarantees: deployment outside the validated contract should trigger abstention or recalibration.

Open problems. Conditional guarantees, active calibration-policy selection, multi-objective contracts, and human-AI override.

9 Conclusion

DEPLOY-RL treats deployment certification as decisionoriented boundary discovery. Its contract-coupled acquisition, conformal false-go control, and fallback decision yield 4.4% false-go with 88.6% coverage on papermaking and TEP, the only result below 5% false-go and above 85% coverage. Future work includes online recalibration, multi-objective contracts, and non-i.i.d. martingale conformal guarantees for temporally coupled plants.

Ethical Statement

DEPLOY-RL is intended for safety-critical industrial process control where trained RL policies require deploy/nodeploy/abstain certification before operation. Its guarantees are contract-relative and require assumptions (A1)-(A4), data separation, and validation of the deployment contract; under uncertainty, Abstain should trigger the PID/MPC fallback rather than RL execution. The papermaking study reports only aggregate metrics from a proprietary digital twin, TEP uses public benchmark data [Downs and Vogel, 1993], and no personal or human-subject data are used. Reported experiments used approximately 200 GPU-hours on one RTX 3090 (about 30 kg CO2e under typical grid intensity). Certified RL deployment may reduce industrial waste, but miscalibrated use outside the verified contract could degrade safety; users remain responsible for operational validation. DEPLOY-RL should not replace domain-specific certification regimes in regulated sectors such as aviation or medical devices; it is intended as an additional statistical deployment gate for industrial control settings with validated simulators and fallback controllers.