Deep Reinforcement Learning for Efficient and Fair Allocation of Healthcare Resources

1 Introduction

The Institute of Medicine (IOM) defines Crisis Standards of Care as "a substantial change in usual health care operations and the level of care it is possible to deliver, which is made necessary by a pervasive (e.g., pandemic influenza) or catastrophic (e.g., earthquake) disaster" [Gostin et al., 2012]. These guidelines recognize that pandemics can strain health systems into an absolute scarcity of health care resources and could result in the unavoidable consequence of rationing.

Following the IOM framework, state governments throughout the U.S. have developed allocation protocols for critical care resources during the pandemic [Piscitello et al., 2020]. Consistent with the broad consensus of ethicists and stakeholders [Emanuel et al., 2020], these protocols aim to triage patients via a pre-specified and transparent policy.

Societal Challenges: Despite adhering to a general framework, critical details of these protocols vary widely across the U.S. [Piscitello et al., 2020]. The protocols also differ in whether they prioritize younger patients or those without preexisting medical conditions. For example, the SOFA protocol (used in New York [VEN, 2015]) focuses on maximizing short-term survival and does not consider age or pre-existing conditions; multiprinciple protocols (used in Maryland and Pennsylvania) give preference to younger patients and those without comorbidities [Biddison et al., 2019]. However, all these protocols assign a static priority score, lacking the flexibility to account for the severity of scarcity and the specific patient pool they are competing against, which leaves room for improvement in patient outcomes.

Beyond the lack of a universally accepted protocol, there is also a pressing societal challenge regarding health equity in resource allocation. Empirical assessments indicate that existing allocation protocols disproportionately disadvantaged Black patients, who were significantly less likely to receive ventilators [Bhavani et al., 2021]. Furthermore, the SOFA score, widely used in triage decisions, has been shown to be only moderately effective in predicting mortality for mechanically ventilated patients [Raschke et al., 2021]. Additionally, evidence suggests that SOFA scores may inadvertently assign higher severity scores to Black patients, which could lead to their lower prioritization for life-saving interventions. [Ashana et al., 2021] These disparities highlight the urgent need to improve both the effectiveness and fairness of allocation protocols to ensure that critical resources are allocated efficiently and equitably among all patients.

Multidisciplinary Collaboration Given that resource allocation decisions are made sequentially and must adapt to evolving patient conditions, reinforcement learning (RL) provides a natural framework for optimizing allocation protocols. RL can leverage large-scale patient data and has the potential to learn optimal allocation strategies that maximize lives saved while ensuring fairness across demographic groups. Developing such an approach requires a multidisciplinary effort that brings together health informaticians and AI researchers. By integrating expertise in clinical decisionmaking, health data science, and AI, we can design an RL

model tailored to the complexities of real-world health crises.

In this study, we investigate the use of RL for optimizing critical care resource allocation policies. Our goal is to develop an adaptive, data-driven allocation policy that maximizes survival rates while promoting fairness in rationing across different racial populations. Our main contributions are summarized as follows:

• We formulate fair health care resource allocation as a multi-objective deep reinforcement learning problem, by integrating the utilitarian and egalitarian objectives into the RL rewards.
• We propose a Transformer-based parametrization of the deep Q-network that significantly reduces the complexity of the classical deep-Q network while making allocation decisions based on individual patient disease progression and interactions among patients.
• We apply our approach to a large, diverse, multi-hospital real-world clinical datasets. Experiments show that our approach leads to fair allocation of critical care resources among different races, while maintaining the overall utility with respect to patient survival.

2 RelatedWork

Health care resource scarcity, especially during pandemics and in intensive care settings, requires decision strategies that balance outcomes, equity, and crisis management [Emanuel and Wertheimer, 2006; Truog et al., 2006; Persad et al.,

2009]. Despite these goals, implicit and explicit discrimination has long permeated healthcare, resulting in numerous instances of biased outcomes [Dresser, 1992; Tamayo-Sarver et al., 2003; Chen et al., 2008].

RL, given its inherent capacity for goal-oriented learning and sequential decision-making, holds promise for developing optimized allocation strategies. However, like other machine learning methods, RL can inadvertently reinforce disparities by favoring majority groups and failing to incorporate fairness objectives [Mehrabi et al., 2021]. Indeed, RL may further exacerbate inequities across patient populations if fairness considerations are neglected [Liu et al., 2018; Ahmad et al., 2020; Rajkomar et al., 2018; Pfohl et al., 2021; Wang et al., 2022; Li et al., 2022].

Existing work on resource allocation has primarily focused on one-time or rollout-based distribution, such as vaccine allocation [Awasthi et al., 2022; Rey et al., 2023; Cimpean et al., 2023], often using multi-armed bandit approaches. These methods, however, are ill-suited for complex scenarios requiring repeated, daily allocation decisions.

RL has also been applied to simulate pandemic trajectories, guide lockdown strategies [Zong and Luo, 2022], project ventilator needs [Bednarski et al., 2021], and allocate PCR tests for COVID-19 screening [Bastani et al., 2021]. For a comprehensive summary of RL applications in healthcare, see the survey by [Yu et al., 2021]. Nonetheless, many of these efforts prioritize efficiency over long-term fairness, leaving critical gaps in addressing disparities in high-stakes resource allocation.

3 ProblemFormulation

We formulate the ventilator allocation problem as a day-today sequential decision problem, as illustrated in Figure 1. Our objective is to optimize the triage protocol regarding who should receive health resources under scarcity, with the goal of saving more lives during a health crisis. This optimization by no means influences physicians' medical decisions regarding whether a patient should be ventilated or not. Therefore, in our formulation and experiments, all patients are prescribed ventilation by physicians but might not receive one due to resource limitation.

3.1 State Space

The state space S describes the current clinical conditions of all patients in the hospital, as well as their ventilation status. Each state is represented by s = [x1, x2, · · · , xN, I1, I2, · · · , IN] ∈ S ⊆ RkN+N , where xi ∈ [0, 1]k ⊆ Rk denotes the current medical condition of the patient on bed i, and indicator Ii ∈ {0, 1} denotes whether the bed i has been ventilated. Apart from normal medical conditions of patients, we consider three special conditions: Survived, Dead, and Vacant, corresponding to the cases where the patient in this bed is recovered or dead after ventilation, as well as currently no patient in this bed. They act as terminators for patients or separators between patients in the same bed. Such designs separate different patients on the same bed explicitly, so that the tasks of learning the progression of medical conditions given ventilation and the task of recognizing the end of each patient can be decoupled. The intensive care units have up to N beds. Here we only consider ventilator scarcity, but bed scarcity can be analogously modeled. In the following, we give a rigorous formulation of the RL model for both cases of without and with the consideration of the fairness in distributing ventilators.

With fairness in consideration, we further record the cumulative numbers of total and ventilated patients of different ethnoracial groups in the state vector, denoted by nk, mk ∈ R respectively, where k ∈ {B,W, A,H} denotes 4 ethnoracial group in the dataset: non-Hispanic Black, non-Hispanic White, non-Hispanic Asian, Hispanic. Thus, each state s = [x1, · · · , xN, I1, · · · , IN, nB, nW, nA, nH,mB,mW,mA, mH] ∈ S ⊆ RkN+N+8 describes the medical and ventilation status of all current patients, as well as the number of cumulative total and ventilated patients of different groups.

3.2 Action Space

For both cases, the action space A is a discrete space denoting whether each bed is on ventilation or not. Let us use 1 to denote ventilate and 0 otherwise, so the action space A ⊆ {0, 1}N . Note that we have two constraints on actions:

• Capacity Constraint: A ⊆ {a ∈ {0, 1}N : ∑Ni=1 ai ≤ C}, where C is the ventilator capacity of the hospital.
• Withdrawal Constraint (optional): a patient who has been ventilated must not be withdrawn until they no longer need it or are discharged.: A ⊆ {a ∈ {0, 1}N : ai = 1 if Ii = 1, i = 1, 2, · · · ,N}, where Ii in the state information denotes whether current patient on bed i has been ventilated before. We apply this constraint

following [Bhavani et al., 2021]. However, in real clinical practice, the withdrawal of a ventilator from one patient to save another raises ethical issues and has not reached a consensus (e.g., ranging from no mechanism in the Maryland protocol to an explicit SOFA-based approach in the New York protocol). Therefore, we also include the results without this constraint in Appendix A.5 1 to provide a comprehensive picture across the spectrum, considering both extremes where withdrawal is either considered or not at all.

Therefore, the action space is shrunk to A = {a ∈ {0, 1}N : ∑Ni=1 ai ≤ C and ai = 1 if Ii = 1, ∀i = 1, 2, · · · , N}.

3.3 TransitionModel

For ease of notation, we denote the three special conditions Survived, Dead, and Vacant as 1,-1, 0 ∈ Rk respectively. In the case without fairness consideration, given current state s = [x1, x2, · · · , xN, I1, I2, · · · , IN] and action a = [a1, a2, · · · , aN], it will transit to the next state s′ = [x′1 , x′2 , · · · , x′N , I ′1 , I ′2 , · · · , I ′N ] in the following coordinatewise way:

• If xi ̸= 0, 1,-1, then the patient will transit to Dead condition if not ventilated:

where pon (x′i |xi) denotes the probability of a patient transiting from medical condition xi to x′i given ventilation. The ventilation status is naively transited P vent i (I ′ i |s, a) = 1 if I ′ i = ai and 0 otherwise. We did not use computational methods to simulate pon (x′i |xi), as the progression of patients' conditions is highdimensional and difficult to model. Instead, we only used real clinical trajectories by sampling from realworld clinical databases.

Following existing clinical literature (e.g., [Bhavani et al., 2021]), we assume that patients who needed ventilators but did not receive one will die. Thus, there will be no patients waiting for a ventilator, as the inability to receive ventilation will result in their immediate deceased and removal from the dataset. However, such an assumption might be over-pessimistic. Therefore, in Appendix A.6, we also explore scenarios where patients not being allocated ventilators does not lead to immediate fatalities, which validates the broader applicability of our proposed methods.

• If xi = 0, 1 or -1, the action does not influence the transition dynamic:

1 https://arxiv.org/pdf/2309.08560

where ξ(·) denotes the distribution of the initial medical condition of a patient when admitted to the critical care units, and qi(s) denotes the probability of a new incoming patient staying in bed i. Note qi(s) depends on how new patients are distributed to empty beds. It satisfies qi(s) = 0 if the bed i is already occupied, and ∑Ni=1 qi(s) = E ∼ Poisson(Λ) assuming the number of incoming patients E obeys an Poisson distribution with parameter Λ. The ventilation status does not matter at this point, we can set P vent i (I ′ i |s, a) = 1 if I ′ i = 0 and 0 otherwise.

Given above discussion for transition dynamics of each individual patients/bed, the overall transition probability can be written as

We further consider the progress of the number of cumulative patients and ventilated patients of each ethnic group, whose deterministic transitions are naive by their definitions.

3.4 Reward

The reward function consists of the following three parts:

• Terminal condition Rt: If a patient is discharged alive after being on a ventilator, a positive reward of 1 is given. If a patient requires a ventilator but is not able to receive one, or dies after being on a ventilator, a penalty of -1 is given: Rt(s, a) = ∑Ni=1 1[xi = 1]-∑Ni=1 1[xi = -1].
• Ventilation cost Rv: if a ventilator is used, it occurs a small negative reward:Rv(s, a) = ∑Ni=1 ai.
• Fairness penalty Rf : in the case with fairness consideration, we consider the cumulative total and ventilated patients of different ethnoracial groups. We expect the distribution of ventilators is equitable in terms of the proportion of ventilated patients of all ethnoracial groups. Therefore, a penalty of KLdivergence between the frequency distributions of incoming patients of different ethnoracial groups Dn d∼ [nB, nW, nA, nH]/(nB + nW + nA + nH) and ventilated patients of different ethnoracial groups Dm d∼ [mB, mW , mA, mH ]/(mB +mW +mA +mH) is considered: Rf (s, a) = KL (Dn∥Dm).

Thus, the reward function is given by

where the parameter λ ≥ 0 balances the trade-off between ventilation effectiveness and fairness. In the case without fairness consideration, we set λ = 0. µ is a small scalar that controls ventilation costs and can be selected through parameter fine-tuning or clinical expertise.

4 Method

In our formulation, due to the resource constraints imposed through the restricted action sets and the fairness requirements via reward penalties, the interaction effects among the patients need to be considered. However, modeling all patients jointly poses computational and memory challenges due to the combinatorial nature of the action space. Addressing such a formulation necessitates a Q network with dimensions proportional to the size of the state space |S| and action space |A|, resulting in a complexity up to O(N ×2N ), where N represents the ICU bed capacity. This complexity makes it impractical to naively construct and train such a Q network, given the challenges in collecting a sufficient number of data points. In practical terms, the intensive care units of health systems typically have hundreds of beds (N ≥ 100), making the computational demands and data requirements for training such a network unattainable. Also, naively concatenating all patients' state vectors introduces an order among patients, which may cause the model to learn artificial factors on that order and potentially bias the model.

To circumvent the computational intractability without losing the consideration of the interaction effects among patients, we therefore propose Transformer-based Q-network parametrization, which inherits the classical Q-learning framework with new parametrization and greedy action selection tailored to our problem structure. In our formulation, the transition of states is partially decomposable because a single patient's clinical conditions depend only on their preceding clinical conditions and ventilation status. Therefore, we can reshape the state from a one-dimensional long vector to a two-dimensional matrix, with the i-th row as [xi, Ii] for patient i. In case the fairness features nk,mk are also considered, we can replicate them N times and append them to the end of each row. The Q-network is parametrized as Tθ : Rdim(S) → RN×2 , whose input is the reshaped state matrix. The i-th row of the output Tθ(s)i ∈ R2 corresponds to the Q-value contributed by the i-th patient given the current state s and the action ai on the i-th patient. We adopt an additive form of the joint Q-value, which estimates the trade-off between effectiveness and fairness when allocating under the constraints and considering the clinical conditions of all patients: Q(s, a) = ∑Ni=1 Tθ(s)i,ai . We leverage the Transformer architecture [Vaswani et al., 2017] for Tθ. Each row in the reshaped s are considered as an input patient token. As our input to the model is essentially an orderless set of patients' conditions with indefinite size, Transformers are a natural fit for our problem because Transformers without positional encodings are permutationally-invariant and can handle inputs with arbitrary sizes. Each Transformer layer is composed of a feed-forward layer and a multi-head attention layer. The feed-forward layer acts independently on each element of the input as a powerful feature extractor, while the attention layer is able to capture the interaction effects between all the elements. This quadratic complexity of Transformer architecture also results in a significant reduction in the network's complexity, decreasing it fromO(N×2N ) to a more manageable O(N 2 ). We also provide robust empirical results to demonstrate the effectiveness and advantages of our proposed transformer-based parametrization compared to the classical deep Q-network in Appendix A.4.

In this parametrization, the greedy action shall be searched within the valid action space A to accommodate the withdrawal and capacity constraints: πθ(a|s) : a∗ = argmaxa∈A ∑Ni=1 Tθ(s)i,ai . Under our parametrization and the additive form of the Q-value, this constrained greedy search can be efficiently solved by first allocating ventilators to those who have already been allocated (withdrawal constraint), where a∗i = 1 ⇔ Ii = 1. Subsequently, the remaining (C - ∑Ni=1 Ii) ventilators are allocated to the newly admitted patients for whom the ventilation improvement di = Tθ(s)i,1 - Tθ(s)i,0 ranks at top-(C - ∑Ni=1 Ii) among all their competitors (capacity constraints).

We also develop a simulator, Simu(X;C,Λ), constructed from real-world clinical trajectories to create the replay buffer for training the proposed model. This simulator maintains the clinical trajectory of each patient while randomizing their relative admission order. X is a set of clinical trajectories from all patients in the training cohort, with each trajectory being of various lengths covering all the clinical conditions of a patient in the critical care unit. At each time step in the simulator, we sample the initial conditions of E patients from X without replacement. E is determined by a Poisson distribution with parameter Λ, inferred from the distribution of ventilator requests in the training cohort. We apply a protocol Π to allocate the available C ventilators when the sum of existing patients and newly admitted E patients exceeds the capacity C. Patients who are not allocated ventilators or those who are discharged (either alive or deceased) are removed from the ongoing simulator and returned to the sampling pool. Patients who are allocated ventilators progress to the next time step of their own trajectory based on the transition function pon (x′i |xi). With this simulator setup, we can generate MDP tuples of indefinite length.

We combine the Transformer q-network parametrization with Double-DQN (TxDDQN) [Van Hasselt et al., 2016], a variant to the original DQN [Mnih et al., 2013] capable of to reduce overestimation. We summarize our proposed model and simulator in Algorithm 1.

5 Experiments

5.1 Dataset

The dataset is sourced from Northwestern Medicine Enterprise Data Warehouse (NMEDW), including electronic health records, pathology data from multiple real-world hospitals and research laboratories. We collected 11,773 ICU admissions that have been allocated mechanical ventilators between March 15, 2020 and January 15, 2023. We filtered the patients with age between 18 and 95 for consideration. We removed admissions with a ventilator allocation duration exceeding 30 days to eliminate anomalies in the data. We extracted 38 features for each admission, including SOFA components, vital signs, demographics, comorbidities (see Appendix A.1 for the list of features and their statistics).

We split our data into 3 splits, 5,455 admissions between March 15, 2020 and July 14, 2021 were used as training data; 1,047 admissions between July 15, 2021 and October 14 2021 were used as validation data on which we selected the best hyper-parameters for testing; 5,271 admissions between October 15, 2021 and January 15, 2023 were used as test data. The patient distribution of different races and the ventilator demands on each day are shown in Appendix A.1. We assume that the original dataset obtained from health systems reflects Algorithm 1 Transformer-based Double Deep Q Network Input: Simu(X;C,Λ), discount factor γ, learning rate α, batch size |B|, loss function Lc,network update frequency h, network update parameter τ

Initialize: Primary network Tθ0 ; Target network Tθ′ , θ′ ← θ0; Allocation protocol Π ← πθ0 ; D ← Φ; s(0) by sampling initial medical conditions of Pois(Λ) patients from X for e = 0 to E ▷ Training loop in epochs

for t = 0 to T - 1 ▷ Construct a ring replay buffer a ∼ πθe (s) ▷ πθ(a|s) : a∗ = argmaxa∈A ∑Ni=1 Tθ(s)i,ai s′ ∼ P(s, a) ▷ Eq. (1) for P(s, a) D ← D ∪ {(s, a, s′ , R(s, a)} ▷ Eq. (2) for R(s, a) θe,0 ← θe for g = 0 to G - 1 ▷ Gradient steps Sample mini-batch B ⊂ D ▷ B = {(s, a, s′ , r}|B| for i = 0 to N ▷ Search next action a′∗ di = (Tθe,g (s′ )i,1 -Tθe,g (s′ )i,0) ▷ Vent. improv. if I ′ i = 1 then a′i ← 1 ▷ Withdrawal constraint elif di ranks top- (C - |I ′ |) in {di | I ′ i = 0}then a′i ← 1 ▷ Capacity constraint else a′i ← 0 θe,g+1 ← θe,g-α·∇θLc[∑Ni=1 Tθe,g (s)i,ai , R(s, a)+ γ · ∑Ni=1 Tθ′ (s′ )i,a′i ] ▷ Policy update if g mod h = 0 then θ′ ← τ · θe,g+1 + (1 - τ) · θ′ θe ← θe,G

a scenario where there was an abundant supply of ventilators available. We aim to investigate the effectiveness of various protocols in mitigating excess deaths in ventilator scarcity.

This setup differed from conventional RL settings, which are trained and evaluated solely on a simulator. We had separate validation and testing sets. This design allowed us to compare the proposed method with existing protocols in realworld hospital operational settings, enhancing the effectiveness and generalizability of our model. Additionally, ventilator demands may fluctuate due to seasonality and outbreaks. A test set spanning a whole year can assess our method's vulnerability to these seasonal request surges.

5.2 Baseline Protocols

Our proposed method was compared with the following existing triage protocols: Lottery: Ventilators are randomly assigned to patients who are in need. Youngest First: The highest priority is given to the youngest patients. SOFA: Patients' prioritization is discretized into three levels (0-7: high, 8-11: medium, and 11+: low) with the lottery serving as the tiebreaker. Multiprinciple (MP): Each patient is assigned a priority point based on their SOFA score (0-8: 1, 9-11: 2, 12-14: 3, 14+: 4). Patients with severe comorbidities receive an additional 3 points. In case of ties, priority is given to patients in a younger age group (Age groups: 0-49, 50-69, 70-84, and 85+). If ties still exist, a lottery is conducted to determine the final allocation. Decision Tree (DT): GrandCl´ement et al. [Grand-Cl´ement et al., 2021] introduced a data-driven decision-tree-based approach for optimizing ventilator allocation. Their protocol factors in the BMI, age, and SOFA score, classifying patients into two priority levels. Admission time is the tie-breaker within the same priority level.

5.3 Off-policy and offline training

The maximum daily demand for ventilators in the validation and testing sets is 85. Therefore, we trained 85 capacityspecific protocol models C = 1, 2, · · · , 85. Ventilator capacity is normalized from [0, 85] to [0%, 100%]. We used Λ = 12 because the number of daily newly admitted patients to the critical care units in our dataset follows a Poisson distribution with Λ = 12. Our simulator supports both offpolicy and offline RL training settings [Levine et al., 2020]. In the off-policy RL setting, we iteratively update Π with πθ when constructing the replay buffer. We refer to the offpolicy trained model without and with fairness consideration as TxDDQN and TxDDQN-fair. In the offline setting, we use existing heuristic-based protocol, MP, as Π to create a large training set and do not update the training data during the training process. This offline training process mimics early-stage health crisis operations where resources are allocated using existing protocols, and improvements are sought thereafter. We refer the offline trained model with fairness consideration as TxDDQN-fair-off. Our experiments were conducted on firewall-protected servers. The training time for each capacity-specific model was less than 30 minutes using a single GPU. For hyper-parameter selections and the survivalfairness Pareto frontier, please see Appendix A.2.

5.4 Evaluation

We evaluated the performance of our proposed protocol based on normalized survival rates, with the survival number at no shortage in ventilators as 100% and no patients alive when no ventilator is available as 0%. Fairness was quantified by comparing the allocation rates across four ethnoracial groups. The allocation rate was calculated by dividing the total number of ventilators allocated by the sum of the ventilators requested. The demographic parity ratio (DPR) [Bird et al., 2020] served as the group metric of fairness, and it is defined as the ratio between the smallest and largest group-level allocation rate. A DPR value close to 1 signifies an equitable allocation, indicating a non-discriminatory distribution among the different groups. Additionally, to gain insights into the impact of protocols under varying shortage levels, we visualized the survival-capacity curve (SCC) and allocationcapacity curve (ACC). The Area Under the Survival-Capacity Curve (AUSCC) serves as an indicator of the overall performance in terms of life-saving abilities under different levels of ventilator shortage. Similarly, the Area Under the Allocation-Capacity Curve (AUACC) reflects the overall performance of ventilator utilization rates under varying levels of shortage.

5.5 Results

Our findings regarding the survival rates for different triage protocols under various levels of ventilator shortage are illustrated in Figure 2. Across all triage protocols, higher ventilator capacities are associated with increased allocation rates, thereby resulting in saving more lives. In the left Panel of Figure 2, our proposed TxDDQN models exhibit higher AUSCC compared to all other baseline protocols, demonstrating the superiority of our models in terms of life-saving efficacy. Likewise, in the right Panel, TxDDQN models also demonstrate higher AUACC compared to other baselines, indicating their superiority in terms of ventilator utilization.

Figure 3 showcases the allocation-capacity curves for different ethnoracial groups and triage protocols in Panels A through H. Only Lottery, TxDDQN-fair and TxDDQN-fairoff model exhibit minimal disparities, but TxDDQN-fair and TxDDQN-fair-off surpasses Lottery in terms of allocation rates and life-saving efficacy. Conversely, all other triage protocols display a preference for specific ethnoracial groups. For example, Youngest and MP favor Hispanic, while SOFA and TxDDQN favor White.

We conducted a detailed analysis in Table 1 under the scenario where approximately 50% of ventilators are unavailable. Our results show that TxDDQN-fair achieves the highest survival rate and allocation rate among the triage protocols. It ranks second in terms of DPR, following closely behind the Lottery protocol. These findings confirm the effectiveness of our TxDDQN-fair approach in improving both survival rates and fairness simultaneously. Importantly, the inclusion of fairness rewards does not compromise its lifesaving capabilities compared to TxDDQN. We also did not observe differences between TxDDQN-fair and TxDDQNfair-off in their life-saving abilities and fairness. This confirms that our proposed methods are adaptable to both offline and off-policy training, and provide the foundation for safe deployment in the early stages of health crises. Our proposed model also demonstrates enhanced fairness and life-saving outcomes in two additional settings: i) where withdrawal constraint is removed (Appendix A.5), and ii) the shortage of ventilators does not lead to immediate deaths (Appendix A.6). We provide ablation studies in Appendix A.3.

6 Summary

In this study, we formulated fair health care resource allocation as a multi-objective deep reinforcement learning problem. We developed a transformer-based deep Q network to integrate individual patient disease progression and interaction effects among patients to optimize for an efficient and fair allocation policy. Our proposed model outperformed existing protocols used by different states in the U.S., by saving more lives and achieving a more equitable allocation of health resources. We refer the reader to Appendix A.7 for the limitations and future directions of this study.

Ethical Statement

The study used real patient data, which underwent Institutional Review Board (IRB) approval to ensure compliance with ethical research principles and regulatory requirements. All experiments and data processing were performed on HIPAA-compliant internal servers, ensuring the confidentiality and security of sensitive health information. The deidentified dataset used in this study will be available as a subset of a consortium-level federated dataset. Throughout the study, efforts were made to minimize bias and ensure that allocation protocols align with established ethical frameworks, promoting fairness, equity, and maximization of benefits in the allocation of health resources.

Contribution Statement

Authors Y.Li, C.M., K.H., H.W., and Z.Y. contributed equally to this work and are designated as co-first authors (marked with an asterisk * in the author list). Authors M.W. and Y.Luo, who serve as PhD advisors to the co-first authors, jointly and equally supervised this work and are recognized as co-senior authors (marked with a dagger †in the author list).