Uncertainty-Guided Adaptive Conservative Offline Reinforcement Learning for Safer Mechanical Ventilation

1 Introduction

Mechanical ventilation (MV) is a life-saving intervention widely used in intensive care units (ICUs) for patients with respiratory failure, including acute respiratory distress syndrome (ARDS) and chronic obstructive pulmonary disease (COPD) [Al-Husinat et al., 2024]. Despite its clinical importance, determining optimal ventilator settings remains challenging due to the substantial heterogeneity across critically ill patients and the rapidly evolving nature of their physiological states. Current practice largely relies on clinician expertise or population-level protocols, which often lack the ability to adapt in a personalized and temporally consistent manner [Silva et al., 2022; Umerenkov et al., 2025]. These limitations motivate the need for data-driven approaches to individualized MV management.

∗Corresponding author

Offline reinforcement learning (ORL) has recently emerged as a promising paradigm for learning treatment policies from retrospective electronic health record (EHR) data without unsafe online interaction [Yu et al., 2021; Jayaraman et al., 2024; Li et al., 2025; Zhang et al., 2025]. However, ORL in healthcare faces two critical challenges: (1) distributional shift, where learned policies select actions rarely observed in historical clinician data; and (2) Q-value overestimation for out-of-distribution (OOD) actions, which can lead to unsafe clinical recommendations.

Conservative ORL algorithms such as Conservative QLearning (CQL) [Yuan et al., 2023] mitigate OOD overestimation by penalizing unseen actions, effectively constraining policies to the support of the dataset. While this improves safety, fixed conservative penalties often overly suppress useful policy improvement and lack flexibility across heterogeneous patient states. More recent uncertainty-aware approaches (e.g., ConformalDQN [Eghbali et al., 2025]) incorporate calibrated risk control, but provide limited granularity in modeling epistemic uncertainty and its interaction with conservative value estimation.

To address these limitations, we propose UBER-CQL (Uncertainty-Balanced Exploration and Robust Conservative Q-Learning), a novel ORL framework for safe MV optimization (Figure 1). UBER-CQL introduces a heteroscedastic Bayesian neural network (HBNN) [Immer et al., 2023] to model the posterior distribution over Q-values, enabling finegrained estimation of epistemic uncertainty. This uncertainty is incorporated into an adaptive penalty that dynamically adjusts conservatism according to state-specific confidence, allowing safer generalization beyond the behavior policy while avoiding unnecessary over-restriction. We further design numerically robust objectives to stabilize Bayesian conservative value learning. Then, we evaluate UBER-CQL on both MIMIC-IV and eICU cohorts, demonstrating consistent improvements over state-of-the-art ORL baselines and clinician policies across safety, policy value, and outcome-related metrics.

In summary, the main contributions of this work are as follows:

• We propose UBER-CQL, an uncertainty-aware conservative ORL framework that leverages HBNNs to model Q-value posteriors for reliable decision-making under distributional shift.
• We introduce an adaptive uncertainty-guided penalty that balances conservatism and policy improvement, reducing unsafe OOD actions while preserving performance.
• We develop numerically stable Bayesian conservative objectives and validate the framework on large-scale ICU datasets, where UBER-CQL consistently outperforms strong ORL and clinical baselines.

2 RelatedWork

Mechanical ventilation (MV) is a cornerstone therapy for critically ill patients with severe respiratory failure [Rubulotta et al., 2024]. However, inappropriate ventilator settings can induce harmful complications, including ventilator-induced lung injury and diaphragm dysfunction [Powers, 2024]. Conventional MV management largely relies on standardized clinical guidelines or physician experience, which often fail to capture complex, patient-specific, and time-varying physiological dynamics [Liu et al., 2024]. These limitations have motivated increasing interest in data-driven decision frameworks, particularly ORL, for personalized ventilator control.

VentAI [Peine et al., 2021] represents one of the earliest RL-based systems for ventilator management, employing Q-learning to derive adaptive MV strategies from retrospective ICU data. This work demonstrated the feasibility of sequential decision-making for personalized ventilation. Subsequently, Yin et al. [Yin et al., 2022] proposed the Deconfounding Actor-Critic (DAC) network, which integrates short- and long-term reward modeling with deconfounding mechanisms to mitigate bias and improve treatment policy accuracy. To address extrapolation errors under offline settings, DeepVent [Kondrup et al., 2023] adopted CQL with intermediate reward shaping to constrain learned policies within the empirical data support. While conservative methods improve safety by discouraging unsupported actions, their fixed penalty design often limits policy flexibility and reduces adaptability across heterogeneous patient states. ConformalDQN [Eghbali et al., 2025] introduced conformal prediction into deep Q-learning to quantify uncertainty and detect OOD actions. This approach provides distributionfree calibration and improves robustness; however, uncertainty is incorporated only post hoc, preventing dynamic regulation of conservative penalties during training and limiting fine-grained action-level risk control [Yue et al., 2023; Zhang et al., 2024].

Despite recent progress, two key limitations persist: (1) most ORL approaches rely on static conservatism that cannot adapt to varying uncertainty across patient states; and (2) existing methods lack precise state-action uncertainty modeling to guide risk-sensitive policy learning in safetycritical clinical environments. To overcome these challenges, we propose UBER-CQL, a unified ORL framework that integrates HBNNs with conservative Q-learning. By jointly modeling Q-value means and variances, UBER-CQL enables fine-grained epistemic uncertainty estimation and adaptively penalizes high-risk actions during training, resulting in improved robustness, safer OOD handling, and more personalized MV decision-making.

3 ProblemFormulation

Personalized MV involves dynamically adjusting ventilator parameters over the course of treatment to maintain physiological stability-e.g., optimizing blood oxygen saturation and arterial carbon dioxide levels-while reducing risks such as ventilator-induced lung injury. The ultimate goal is to improve patient outcomes and survival.

We formulate MV management as a continuous-time control problem. The patient's physiological state at time t is represented as x(t) ∈ Rn , and the ventilator control variables (e.g., tidal volume, PEEP) as u(t) ∈ Rm. The state evolves according to nonlinear dynamics:

where f(·) denotes unknown patient-specific transition dynamics. The objective is to determine a control policy u(t) that minimizes the cumulative treatment cost:

where L(·) captures instantaneous clinical risk and physiological deviation, and Φ(·) represents the terminal outcomerelated cost.

Since the transition function f (·) is unknown and online interaction is infeasible in safety-critical clinical environments, we reformulate personalized MV management as an ORL problem under a Markov Decision Process (MDP) framework [Ernst and Louette, 2024]. The MDP is defined as ⟨S, A, P, r, γ⟩, where:

• S: state space representing patient physiology. We construct a 42-dimensional state vector comprising demographic information, vital signs, laboratory measurements, and fluid-related indicators, collectively capturing the patient's clinical status.
• A: action space corresponding to ventilator settings. Actions consist of three controllable parameters: tidal volume adjusted by ideal body weight (Vt), positive

end-expiratory pressure (PEEP), and fraction of inspired oxygen (FiO2). Each parameter is discretized into seven levels, yielding a combinatorial action space:A = V t × PEEP × F iO2.

• P(s′ |s, a): transition probability from state s to s′ under action a. As direct interaction with patients is impossible, transitions are implicitly learned from the offline dataset D = {(s, a, r, s′ )}, enabling value estimation based on empirical clinical dynamics.
• r(s, a): reward function measuring treatment effectiveness. We define rewards using both short-term physiological improvement and long-term survival outcomes. Intermediate rewards are derived from normalized changes in the Apache II score, while terminal rewards encode 90-day mortality:

where At denotes the Apache II score at time t, and Amax, Amin are used for normalization. This design ensures alignment between immediate physiological stabilization and long-term prognosis [Eghbali et al., 2025].

• γ ∈ (0, 1): discount factor controlling the relative importance of immediate versus future rewards. Smaller values emphasize short-term physiological stabilization, while values closer to 1 prioritize long-term clinical outcomes.

Given an offline dataset D = {(si, ai, ri, s′i )}Ni=1 , ORL aims to learn a policy π : S → A that maximizes the expected discounted return:

where ξ = (s0, a0, s1, a1, . . . ) denotes a trajectory induced by policy π.

4 ProposedMethod: UBER-CQL

In this section, we present the proposed UBER-CQL framework for personalized mechanical ventilation optimization. As illustrated in Figure 2, the framework consists of three stages: 1) Data Extraction, which preprocesses 42 physiological variables to construct an offline MDP dataset; 2) UBER-CQL, an uncertainty-aware conservative RL algorithm for safe ventilator control; and 3) Evaluation, where offline policy evaluation estimates policy value on held-out cohorts. We detail the UBER-CQL algorithm below and describe Data Extraction and Evaluation in the following section.

4.1 Uncertainty-Aware Q-Value Estimation via HBNNs

Offline RL for MV optimization faces substantial challenges due to limited data coverage and the prevalence of OOD actions. These issues can result in unsafe control decisions, as the learned policy may query state-action pairs that are underrepresented in the dataset.

To address this, we adopt a HBNN to model uncertainty in Q-value estimation. Unlike standard Bayesian neural networks that assume homoscedastic noise, HBNNs capture input-dependent predictive variance, allowing for the estimation of both the predictive mean and aleatoric uncertainty, which reflects irreducible variability in clinical data [Skafte et al., 2019].

Formally, let θ denote the parameters of the HBNN, which models the Q-value for each state-action pair (s, a) as a Gaussian distribution:

where both the predictive mean µθ : S×A → R and variance σ2 θ : S ×A → R+ are learned. This probabilistic formulation contrasts with deterministic Q-networks Qϕ(s, a), which yield only point estimates.

Training proceeds by minimizing the negative loglikelihood:

where the target value y = r + γmaxa′ µθ-(s′ , a′ ) is computed using a target network θ-. This loss formulation naturally discourages high variance in data-scarce regions, encouraging conservative and risk-aware action selection-critical for ensuring safety in high-stakes clinical settings like MV therapy.

4.2 Adaptive UncertaintyManagement and Action Penalization

To further enhance robustness against OOD actions and distribution shift in offline MV datasets, we propose a dynamic uncertainty-aware learning framework. By leveraging model uncertainty to modulate Q-value targets, the framework mitigates overestimation bias while preventing overly pessimistic updates. As a result, it enables safer and more personalized policy optimization tailored for high-stakes MV settings, where decision reliability is paramount. This framework synergistically combines three key components: uncertaintyguided action penalization, adaptive Bellman updates, and conservative Q-learning regularization.

Uncertainty-Guided Action Penalization

In safety-critical MV tasks, actions exhibiting high epistemic uncertainty-stemming from sparse data coverage in offline datasets-can precipitate hazardous clinical outcomes, such as barotrauma induced by excessive airway pressures or volutrauma from overdistension. To counteract this, we utilize a deep ensemble of K = 5 HBNNs for robust uncertainty quantification [Seitzer et al., 2022], where the ensemble mean ¯Qθ(s, a) = 1K ∑Kk=1 µθk (s, a) and epistemic uncertainty σep θ (s, a) = stdk{µθk (s, a)} are computed across members. Each individual HBNN produces a predictive mean Q-value and an output variance that models aleatoric uncertainty (i.e., inherent stochasticity in observations, such as variable patient responses [Simone et al., 2026]), while epistemic uncertainty reflects model ignorance in underrepresented regions. This ensemble-based approach serves as an efficient approximation to the posterior distribution over network parameters, enabling reliable inference without the computational overhead of full Bayesian methods.

To penalize high-uncertainty actions, we compute a dynamic threshold τσ as the pu-th percentile of {σep θ (s, a) | (s, a) ∈ B}, where B is a mini-batch from dataset D and pu = 0.8:

The penalized Q-function is then:

where ¯Qθ(s, a) is the ensemble mean and β = 0.5.

This selective penalization mechanism reduces the agent's tendency to exploit poorly supported regions of the stateaction space, such as extreme ventilation parameters (e.g., abnormally high tidal volumes or PEEP levels), thereby lowering the risk of adverse outcomes such as barotrauma or volutrauma. By incorporating action-specific epistemic uncertainty into the value estimation pipeline, this approach provides a principled mechanism to encourage cautious decisionmaking under uncertainty-an essential property for realworld clinical deployment.

Adaptive Bellman Updates

To balance exploration and conservatism in offline reinforcement learning, we introduce an adaptive Bellman update mechanism that interpolates between optimistic and pessimistic targets. This dynamic weighting enables the agent to benefit from exploratory value propagation in well-supported regions while remaining conservative under high uncertainty or distribution shift.

The optimistic Bellman loss encourages standard value iteration by following the classical Bellman optimality operator:

where r is the observed reward, γ is the discount factor, and µθ- denotes the target network updated with a delay to stabilize learning.

In contrast, the pessimistic Bellman loss incorporates uncertainty-aware penalization to discourage overestimation in regions with limited support. It replaces the Q-value target with the penalized value Qp(s′ , a′ ) defined earlier:

The adaptive Bellman update mechanism is formalized by combining the optimistic and pessimistic losses into a unified adaptive loss function, weighted dynamically based on uncertainty or policy confidence:

where ω dynamically adjusts the trade-off between optimistic and pessimistic Bellman updates. Specifically, ω decays over training to gradually shift focus from exploration to conservatism:

where ωmax = 0.9, ωmin = 0.1, and p = min(iterations/30000, 1) represents normalized training progress. This schedule ensures the policy is exploratory early on and progressively conservative for safe deployment in clinical settings.

Conservative Q-Learning Regularization

To further reduce overestimation bias prevalent in offline RL, we incorporate a CQL penalty that discourages assigning high Q-values to unsupported actions:

where α = 0.1 is the regularization coefficient. This term penalizes overconfident Q-values for out-of-distribution actions, enhancing safety and robustness under limited data coverage.

4.3 Training

The overall training objective integrates all components to guide uncertainty-aware and conservative policy learning:

where λ = 0.1 balances the heteroscedastic Bayesian Qvalue loss.

5 Experiments and Results

5.1 Experimental Setup

We extracted mechanically ventilated patient cohorts from two large-scale ICU databases: MIMIC-IV [Johnson et al., 2023] and eICU [Pollard et al., 2018]. To rigorously evaluate cross-database generalization, we use MIMIC-IV as the primary training source and eICU as an independent crossdatabase test cohort. Demographic and physiological statistics for the resulting Markov Decision Processes (MDPs) are summarized in Table 1.

Following prior work [Eghbali et al., 2025], we also curated an out-of-distribution (OOD) cohort representing physiologically atypical patients. A patient is labeled as OOD if any initial state feature-including demographics, vital signs, laboratory results, or fluid status-falls within the top or bottom 1% of the population distribution. This procedure yields an OOD subset comprising approximately 25% of the cohort. Both in-distribution (ID) and OOD datasets were split into training (60%), validation (20%), and testing (20%) sets. Training and validation data were used for model fitting and hyperparameter tuning, while the held-out test set was reserved for final evaluation.

5.2 Evaluation

To assess the performance of learned policies in our offline RL framework for mechanical ventilation, we employ two established off-policy evaluation (OPE) methods: Fitted QEvaluation (FQE) [Hao et al., 2021] and the Doubly Robust (DR) estimator [Tang and Wiens, 2021]. These approaches enable reliable estimation of expected returns using historical data without requiring online interaction, which is critical in clinical settings.

FQE approximates the optimal Q-function Q∗ by minimizing the mean squared Bellman error over the offline dataset:

where F denotes the Q-function hypothesis space, M is the number of samples, and Qt,i is the Bellman target computed using the evaluation policy π.

Let δ(i) t = r(i) t +γ ˆ V (s(i) t+1 )- ˆQ(s(i) t , a(i) t ) denote the singlestep Bellman residual under policy π. The step-wise DR estimator is then:

where w′t,i = min(∏tk=0 π(a(i) k |s(i) k ) µ(a(i) k |s(i) k ) ,C) is the clipped cumulative importance weight, and C > 0 is the clipping threshold.

5.3 Baselines and Implementation Details

To evaluate the effectiveness of UBER-CQL for mechanical ventilation optimization, we compare it against several established offline RL baselines relevant to clinical decisionmaking: DDQN [Feng et al., 2023], BCQ [Liu et al., 2024], CQL [Kondrup et al., 2023], and ConformalDQN [Eghbali et al., 2025].

All models were implemented in Python 3.8 using PyTorch 1.13. We adopt a shared architecture with two hidden layers of 256 units and ReLU activations. Training proceeded for up to 30,000 timesteps, with target networks updated every 5,000 steps for stability. Optimal hyperparameters, including learning rates [10-3 , 10-4 , 10-5 , 10-6 ] and discount factors γ ∈ [0.05, 0.1, 0.5, 0.75, 0.9], were selected via grid search on the validation set, yielding γ = 0.75. Experiments were performed on Ubuntu 20.04.5 LTS with an Intel Core i710700 CPU, NVIDIA RTX 3090 Ti GPU, and 32 GB RAM. All models were trained over five independent runs, and reported results correspond to the mean performance to ensure robustness and reproducibility.

For UBER-CQL, the best performance was achieved with a learning rate of 5 × 10-4 . Baseline hyperparameters were configured as follows: BCQ (lr = 1 × 10-4 ); CQL (lr = 1 × 10-4 ); ConformalDQN (lr = 1 × 10-3 , α = 0.15); and DDQN (lr = 1 × 10-3 ).

5.4 Results and Analysis

Quantitative Results

We evaluate model effectiveness in optimizing MV treatment by examining the Pearson correlation coefficient [Checkley et al., 2008] between predicted Q-values and patient mortality. For each initial state-action pair (s, a) in the test set, we compute the predicted Q-value Q(s, a) and the corresponding binary mortality outcome M(s, a) (1 for death, 0 for survival).

As reported in Table 2, all models exhibit negative correlations, indicating that higher predicted Q-values are associated with lower mortality. The proposed UBER-CQL achieves the strongest correlations of -0.538 ± 0.007 on MIMIC-IV and -0.414 ± 0.009 on eICU, outperforming the strongest baseline (ConformalDQN) by approximately 19% due to its uncertainty-aware regularization. These results demonstrate the superior robustness and clinical reliability of UBER-CQL across heterogeneous ICU datasets.

Prior work [Luo et al., 2024] has highlighted limitations in existing RL methods regarding evaluation consistency and baseline selection, which can lead to misleading performance claims. To contextualize this, we include simple baseline policies such as random, zero-drug, and max-drug strategies. As shown in Table 3, these na¨ıve baselines achieve nearzero initial FQE scores on both MIMIC-IV and eICU, reflecting their limited ability to anticipate long-term outcomes from the onset of mechanical ventilation. Among offline RL methods, CQL outperforms BCQ, while the uncertaintyaware ConformalDQN further improves policy value estimation. The proposed UBER-CQL achieves higher estimated value estimates across both datasets, with FQE scores of 0.815±0.021 on MIMIC-IV and 0.725±0.022 on eICU, exceeding standard baselines, uncertainty-aware methods, and even the physician policy. These results demonstrate that incorporating fine-grained uncertainty modeling within conservative Q-learning produces more reliable and robust value estimates in offline clinical settings.

Robustness Evaluation

To evaluate robustness under distribution shifts, we compare mean initial Q-values for ID and OOD states, providing insight into each model's susceptibility to Q-value overestimation-a critical factor for safe and reliable decision-making in MV.

Figure 3 presents the mean estimated Q-values for ID and OOD states, with the red dashed line indicating the overestimation threshold at Q = 1.0. Traditional methods such as DDQN exhibit substantial overestimation, particularly in OOD settings, with means of 1.112 (ID) and 1.237 (OOD), both exceeding the threshold. CQL yields more conservative estimates, with means of 0.789 (ID) and 0.840 (OOD), remaining below 1.0. BCQ produces consistently low and stable values (0.710 ID, 0.762 OOD), suggesting slight underestimation. ConformalDQN, leveraging distribution-free uncertainty quantification, achieves reasonable calibration with 0.728 (ID) and 0.777 (OOD), mitigating overestimation while retaining responsiveness. Notably, UBER-CQL achieves the most balanced and stable estimates, with 0.899 (ID) and 0.894 (OOD), staying well below the threshold and showing minimal degradation under distribution shifts. In OOD scenarios, UBER-CQL's Q-values are approximately 27.8% lower than DDQN's and 15.1% higher than ConformalDQN's, demonstrating its effective trade-off between avoiding overestimation and preventing undue conservatism.

These results highlight the efficacy of UBER-CQL's uncertainty-aware penalization in managing distributional shifts and promoting cautious decision-making under high epistemic uncertainty, ensuring reliable value estimation for high-stakes ICU interventions.

Ablation Study

We conducted ablation experiments to systematically assess the contribution of each core component in the UBER-CQL framework.

Impact of Bellman Update Strategies. Table 4 reports the ablation results on MIMIC-IV and eICU datasets under both FQE and DR evaluation. The full UBER-CQL consistently achieves the highest performance across all metrics, confirming the effectiveness of our uncertainty-aware and conservative learning design.

First, we examine the role of the Bellman update strategy. The variant w/o Opt/Pess, which removes both optimistic and pessimistic Bellman objectives and relies on a standard Bellman update, exhibits clear performance degradation on both datasets. This highlights that explicitly incorporating optimistic and pessimistic value targets is crucial for robust Qvalue estimation in offline RL.

Next, disabling the adaptive interpolation between optimistic and pessimistic updates (w/o Adaptive Bellman) also reduces performance relative to the full model. This indicates that not only are the dual objectives important, but dynamically balancing them is essential for effectively reconciling exploration with conservatism under distributional shift.

Finally, we evaluate the contribution of conservative Qlearning regularization. Removing the CQL penalty (w/o CQL) leads to the largest performance drop, particularly in FQE scores, underscoring its critical role in mitigating overestimation and improving robustness against OOD actions. The magnitude of this drop emphasizes that CQL complements the adaptive Bellman updates to stabilize policy learning.

Overall, these results demonstrate that each component-optimistic/pessimistic Bellman targets, adaptive interpolation, and conservative Q-learning-contributes meaningfully to UBER-CQL's performance. Their joint integration is key to achieving reliable, safe, and effective offline policy learning for high-stakes mechanical ventilation control.

Sensitivity Analysis of Key Hyperparameters. We evaluate the sensitivity of UBER-CQL to two key hyperparameters: the uncertainty penalization coefficient β and the uncertainty quantile threshold pu, under both ID and OOD settings (Figure 4).

As shown in Figure 4(a), increasing β from 0.1 to 0.5 improves the mean initial Q-values for both ID and OOD actions, suggesting that a moderate penalty on highuncertainty actions effectively suppresses overestimation in under-supported regions. However, larger values of β lead to performance degradation, particularly for OOD actions, as overly aggressive penalization may incorrectly suppress Qvalues for actions that are clinically valid but epistemically uncertain due to limited data coverage.

Figure 4(b) shows that increasing pu from 60% to 80% enhances performance by better penalizing uncertain actions, but further increases yield diminishing returns, potentially restricting optimal ventilation strategies in ICU scenarios. Overall, these findings demonstrate UBER-CQL's robustness across reasonable hyperparameter ranges and offer practical tuning guidance.

Figure 4: Sensitivity analysis of key hyperparameters in UBERCQL.The results illustrate how varying these hyperparameters affects policy value estimates, highlighting the robustness of the proposed framework across reasonable parameter ranges.

6 Conclusion

We proposed UBER-CQL, a novel ORL framework tailored for safety-critical clinical decision-making, with a focus on personalized MV. UBER-CQL unifies HBNNs with conservative Q-learning to jointly model action-value uncertainty and adaptive conservatism. By explicitly estimating both the mean and variance of Q-values, the method provides finegrained epistemic uncertainty that is leveraged to dynamically penalize high-risk and poorly supported actions, thereby improving robustness under distribution shift and mitigating extrapolation errors common in offline settings. Extensive experiments on real-world MV cohorts from MIMIC-IV and eICU demonstrate that UBER-CQL consistently outperforms state-of-the-art offline RL methods and clinical baselines in terms of value estimation, robustness to OOD states, and alignment with patient survival outcomes. These results highlight the promise of uncertainty-aware conservative learning for safe and reliable policy optimization in high-stakes healthcare applications. Future work will investigate extensions to multi-modal clinical data, as well as the integration of causal inference to further improve interpretability and decision reliability across broader clinical domains.

Acknowledgments

This paper is supported by the National Key Research and Development Program for Young Scientists (No. 2025YFB4711500), National Natural Science Foundation of China (No. 42474027, No. 62403085), Xiaomi Young Talents Program and Graduate Research and Innovation Foundation of Chongqing, China (No. CYB240066).