Hierarchical Apprenticeship Learning for Disease Progression Modeling

1 Introduction

Disease progression modeling (DPM) aims to characterize patients' longitudinal medical records, identify disease progressive stages, and evaluate factors affecting the progression pathways [Cook and Bies, 2016]. Empowered by the increasing availability of large medical datasets, e.g., electronic health records (EHRs), various deep learning models [Choi et al., 2016; Zhang et al., 2017] have been developed for DPM based on the recorded observations in EHRs, e.g., vital signs and laboratory test results. However, relying solely on monitored observations may not fully capture the complexity of disease progression [Tintinalli et al., 1985]. For example, different patient groups may exhibit varying observations for the same disease, where a normal sign in one group may turn out to be abnormal in another [Baytas et al., 2017]. Some recent DPM works have used subsequence clustering for deriving progressive stages automatically in a data-driven manner from various observations [Yang et al., 2021]. Other than using observations for DPM, interventions also have a significant impact on disease progression [Komorowski et al., 2018; Azizsoltani et al., 2019], and some prior works have taken clinicians' interventions as additional features for DPM [Esteban et al., 2016; Goh et al., 2021]. Clinical interventions are usually carried out based on prior medical knowledge, which is difficult to be reflected by observations alone. In addition, interventions tend to take hours or days to manifest in observations [Dulac-Arnold et al., 2020]. Furthermore, individual clinical interventions on DPM cannot be fully explained without assessing their effectiveness, since different patient groups may respond to the same treatment differently, and patients at different stages of DPM may respond differently to the same treatment [Clifford et al., 2016]. For this reason, when learning progressive patterns in DPM, it is essential to incorporate the effectiveness of interventions.

∗This research was conducted during the author's affiliation with North Carolina State University.

Reinforcement learning (RL) has shown considerable potential in learning effective interventional strategies for clinical decision-making [Wang et al., 2018; Yu et al., 2021]. A common challenge when applying RL to healthcare is the design of reward function, which serves as an incentive to induce effective policies [Azizsoltani et al., 2019]. Apprenticeship learning (AL) was proposed to address this issue [Abbeel and Ng, 2004]: instead of taking an explicitly delineated reward function as input, AL learns the policy by imitating the demonstrated behaviors of clinical experts. Recently, an AL method named energy-based distribution matching (EDM), has shown great success in inducing effective clinical interventional policies directly from EHRs [Jarrett et al., 2020]. In EDM, the demonstrations provided by clinicians are assumed to be generated via a uniform policy, latently driven by a single reward function. However, while managing patients under various progressive stages, such as common fevers or severe organ failures, clinicians may adopt varying policies with multiple reward functions [Wang et al., 2021].

In this paper, we leverage an AL named Time-aware Hierarchical EM Energy-based Subsequence (THEMES) clustering [Yang et al., 2023] for DPM. THEMES is designed to handle the multiple evolving reward functions through two key components: a) subsequence partitioning, which clusters patients' sequential records into progressive stages, and b) policy induction, which induces interventional policy for each stage. The two components are performed iteratively: referring to the progressive stages learned by subsequence partitioning, fine-grained evolving interventional policies can be induced; meanwhile, these derived interventional patterns will, in turn, refine the partitioned subsequences to indicate more accurate progressive stages. The two components are modeled by a Reward-regulated Multivariate Time-aware Toeplitz Inverse Covariance-based Clustering (RMT-TICC) and an Expectation Maximization EDM (EM-EDM), respectively. RMT-TICC encodes the interventional patterns by introducing a reward regulator during the subsequence partitioning, motivated by the fact that interventions are latently driven by rewards; while EM-EDM captures the progressive patterns by using an EM to simultaneously cluster the RMTTICC learned subsequences and induce their respective policies. We incorporate both progressive patterns from RMTTICC and interventional patterns from EM-EDM for DPM.

The effectiveness of incorporating THEMES-learned patterns for DPM is evaluated by modeling sepsis, an extremely challenging and life-threatening organ dysfunction that is a leading cause of death worldwide [Singer et al., 2016]. Without timely interventions, patients can progress to the most severe condition of septic shock with a high mortality rate of 50% [Martin et al., 2003], while 80% of sepsis deaths can be prevented with timely diagnosis and treatment [Kumar et al., 2006]. Therefore, modeling sepsis progression is crucial for more accurate early prediction of septic shock. To achieve this goal, we leverage THEMES by incorporating latent progressive patterns extracted by RMT-TICC and interventional patterns derived by EM-EDM as supplemental features for original observations. Our results demonstrate that incorporating such additional features leads to improved DPM accuracy. The major contributions of this work are three-fold:

• To our best knowledge, this is the first work incorporating AL-derived progressive and interventional patterns for DPM.
• Our empirical results reveal that the success of incorporating AL for DPM relies on the fact that THEMES can handle time-awareness and evolving reward functions.
• We utilize AL for modeling a challenging disease, i.e., sepsis, and predicting its most severe condition, i.e., septic shock, in early stages. This indicates the potential to enhance personalized and timely treatments for patients.

2 RelatedWork

2.1 Disease ProgressionModeling

The importance of DPM has been recognized in previous studies [Cook and Bies, 2016; Severson et al., 2020], leading to the usage of deep learning models such as recurrent neural networks [Lipton et al., 2015; Saqib et al., 2018]. Among these models, long short-term memory (LSTM) has shown remarkable success due to its ability to capture sequential patterns from historical records. Typically, these deep learning models take monitored observations as input and use neural networks to extract latent progressive patterns for DPM.

An alternative approach to build DPM is through subsequence clustering, which captures disease progressive stages by partitioning and clustering subsequences. In general, subsequence clustering can be categorized into distance-based (e.g., dynamic time warping [Giannoula et al., 2018]) and model-based (e.g., Gaussian mixture models [Faruqui et al., 2021] and hidden Markov models [Kwon et al., 2020]). Model-based methods are usually more reliable for better handling noise and outliers. Recently, a model-based approach called Toeplitz inverse covariance-based clustering [Hallac et al., 2017] has gained attention for accurately partitioning subsequences in various applications, such as analyzing physical activities for Alzheimer's patients [Li et al., 2018] and segmenting critical stages for sepsis patients [Gao et al., 2022]. Building upon this work, a multi-series timeaware Toeplitz inverse covariance-based clustering approach (MT-TICC) [Yang et al., 2021] was proposed, which takes multiple series as input and uses a time-awareness mechanism to handle irregular time intervals in EHRs. The MTTICC-derived progressive patterns, when used as additional information of observations, enable more accurate early prediction for septic shock, outperforming competitive baselines.

It is important to recognize that clinicians' interventions play a significant role in patient's disease progression [Komorowski et al., 2018; Azizsoltani et al., 2019], yet this is not sufficiently addressed in the majority of existing works. Though some prior studies have included interventions as additional features in DPM [Esteban et al., 2016; Goh et al., 2021], they do not consider the effectiveness of interventions.

2.2 RL&AL for EHRs

Reinforcement learning (RL) has been widely applied in EHRs for dynamic treatment regimes, which aims at inducing a decision-making policy to dictate how the interventions should be executed so that the patients can gain improved outcomes [Yu et al., 2021]. As an input of RL, the reward function plays a critical role in praising/punishing the learning model to derive an optimal policy. However, manually specifying an appropriate reward function is usually expertiseintensive and time-consuming, posing a significant barrier to the broader applicability of RL [Abbeel and Ng, 2004]. Apprenticeship learning (AL) [Ng et al., 2000] tackles such problems by directly learning the reward function from experts' demonstrations. Behavior cloning [Raza et al., 2012] is a classic AL method, which directly learns a mapping from states to actions to greedily imitate experts' demonstrated behaviors [Ross et al., 2011]. Later, various Inverse RL (IRL) [Abbeel and Ng, 2004; Ziebart et al., 2008] and adversarial imitation learning [Ho and Ermon, 2016; Finn et al., 2016] based AL approaches have been proposed, but they are often online, requiring iteratively executing the latest policy to collect data for updating the model. The execution of a bad policy in healthcare is unethical [Levine et al., 2020], making offline AL methods desired. Though some online approaches have been adapted to offline [Chan and van der Schaar, 2021; Kostrikov et al., 2018], they usually rely on off-policy evaluation, which itself is nontrivial with imperfect solutions.

More recently, [Jarrett et al., 2020] introduced an AL approach named energy-based distribution matching (EDM) and evaluated it on various benchmarks, e.g., Acrobot, LunarLander, and BeamRider. Their results demonstrated that EDM outperformed both IRL-based and adversarial imitation learning-based methods [Brockman et al., 2016]. Therefore, we employ EDM as a baseline in this paper. When applying EDM to EHRs for modeling clinicians' interventions, it makes a strong assumption that all demonstrations are generated with a unified policy, following a single reward function. However, clinicians commonly execute different policies with multiple reward functions when treating patients under different progressive stages [Wang et al., 2021; Wang et al., 2022]. To handle the multiple reward functions, some improved AL have been proposed [Dimitrakakis and Rothkopf, 2011; Babes et al., 2011]. For example, [Babes et al., 2011] developed an EM-based IRL, which assumes reward functions are diverse across different demonstrations, while within each demonstrated sequence, the reward function is assumed to be unified. To model the multiple reward functions evolving over time, several more recent works have been proposed [Krishnan et al., 2016; Hausman et al., 2017; Wang et al., 2021; Wang et al., 2022]. However, these methods generally partition the demonstrations into fixed-length subsequences to learn their respective reward functions, without considering the irregular intervals during the partitioning.

Furthermore, existing AL methods typically concentrate on learning interventional patterns for either inducing a decisionmaking policy or evaluating whether interventions are carried out as expected. However, none of these methods have been incorporated to learn the progressive stages during the DPM.

3 THEMES for DPM

Figure 1 provides an illustration of how THEMES-derived patterns can be utilized for DPM. The entire input data DEntire consists of L sequences represented as {Dl } = {(xlt , alt )|t = 1, ..., T l ; l = 1, ...,L}, where xlt ∈ Rm is the t-th multivariate observed state with m features, and alt is the corresponding action in the l-th sequence with the length of T l . In the context of AL, it is typically assumed that the experts' demonstrations provided as input are optimal or near-optimal following a latent reward function [Abbeel and Ng, 2004]. The quality of these demonstrations plays an important role in inducing more accurate policies. Therefore, we have designed a procedure for determining the experts' demonstrations (detailed in Section 4.1), based on which a subset of N (N ≤ L) sequences DExpert are selected, denoted as: {Dn } = {(xnt , ant )|t = 1, ..., T n ; n = 1, ..., N}.

Taking DExpert as input, THEMES aims at learning the multiple underlying reward functions evolving over time. As illustrated in Figure 1 (Left), THEMES has a hierarchical structure, with two major components at its low-level, i.e., RMT-TICC and EM-EDM. Specifically, RMT-TICC operates on the states in DExpert as input. It partitions and clusters the subsequences, such that each subsequence cluster exhibits the same time-invariant patterns, which is regarded as a high-level state. Then focusing on the state-action pairs over the partitioned subsequences, EM-EDM will cluster and induce their policies, such that each cluster exhibits consistent decision-making patterns, which is referred to as a high-level action. Subsequently, taking the high-level state-action pairs as input, a high-level reward regulator will be learned by IRL and fed back to refine the RMT-TICC. This iterative procedure continues until convergence is achieved.

Once the THEMES model has been learned, we apply it over DEntire for extracting progressive patterns from RMTTICC and interventional patterns from EM-EDM. These patterns are then combined with the observations from the original data, which will further serve as an input of LSTM for septic shock early prediction, as depicted in Figure 1 (Right).

3.1 Subsequence Partitioning by RMT-TICC Preliminaries

Given experts' demonstrations, denoting the states in DExpert as {xnt |t = 1, ..., T n }, the objective of subsequence clustering is partitioning and clustering the {xnt } into subsequences based on their latent time-invariant patterns. It can be achieved by learning a mapping from each state to a specific cluster {k|k = 1, ...,K}. To capture the interdependence among neighboring states, a sliding window of length ω ≪ T n is employed to incorporate the context information. When assigning a state xnt into a cluster k, we also consider its preceding states within the sliding window, i.e., Xnt = {xnt-ω+1 , ..., xnt }, where Xnt is a random variable of dimension mω, concatenating the m-dim states in ω.

To learn the subsequence clusters, we fit Xnt into K Gaussian distributions, with each distribution indicating a specific cluster k. It can be modeled by Toeplitz inverse covariancebased clustering [Hallac et al., 2017] to learn the mean and inverse covariance matrix for each cluster. More specifically, the mean vectors, {µk|k = 1, ...,K}, are determined by assigning each state to an optimal cluster, resulting in clustering assignments P = {Pk|k = 1, ..., K}, where Pk ⊂ {1, ..., T } is the indices of states (sliding windows) belonging to cluster k. The inverse covariance matrices, Θ = {Θk|k = 1, ..., K}, are estimated to characterize the time-invariant structural patterns for each cluster. Herein, Θk is constrained to be blockwise Toeplitz, composed of ω sub-blocks A(i) ∈ Rm×m, i ∈ [0, ω - 1], where the sub-block A(i) represents the partial correlations among m features between timestamp t and t+i. For instance, the (p, q)-th element in A(i) represents the partial correlation between the p-th feature at t and the q-th feature at t + i, where p, q ∈ {1, ..., m}.

A multi-series time-aware Toeplitz inverse covariancebased clustering (MT-TICC) [Yang et al., 2021] was proposed recently, improving the subsequence clustering from two perspectives: First, MT-TICC operates on multi-series data, allowing for improved estimation of the mean and inverse covariance matrix by considering shared patterns across different sequences. This capability is particularly advantageous in real-world scenarios like EHRs, where the data is collected from various patients. Second, MT-TICC incorporates time-awareness to handle irregular intervals by using a decay function to constrain the consistency within each cluster. It is highly preferred when dealing with data collected irregularly, such as in healthcare settings. However, when applied to DPM, MT-TICC primarily focuses on the observed states, and it does not fully account for the interventional patterns conveyed by state-action pairs, which can significantly impact the disease progression [Komorowski et al., 2018].

RMT-TICC

To incorporate the interventional patterns conveyed by stateaction pairs, a Reward-regulated MT-TICC (RMT-TICC) is leveraged in THEMES, considering the fact that the interventions are driven by underlying reward functions. The objective function of RMT-TICC is as follows:

• Log-likelihood termmeasures the probability that Xnt belongs to the cluster k. Specifically, assuming Xnt is a Gaussian distribution with the mean of µk and the inverse covariance matrix of Θ-1 k , the log-likelihood term is defined as:

• Reward-regulated Time-aware Consistency term encourages the consecutive states {Xt-1,Xt} to be assigned into the same cluster, by taking account of both the time intervals and the corresponding rewards. It penalizes the neighbored states belonging to different clusters by minimizing Eq.(3):

In above equation, β is a weight parameter. 1{t-1 /∈ Pk} is an indicator function, with the value of 1 if Xnt-1 and Xnt does not belong to the same cluster; otherwise its value is 0. 1/log(e + ∆T n t ) is a decay function, which adaptively relax the penalization in Eq.(3) when the interval ∆T n t between consecutive states becomes larger [Baytas et al., 2017].

To incorporate decision-making patterns for refining the subsequence clustering, a hierarchical structure is utilized, as illustrated in Figure 1 (Left). Rather than directly learning rewards from observed state-action pairs, this hierarchical approach is employed for two reasons. First, decision-making patterns across different subsequences have varying degrees of importance contributing the patients outcomes, while treating them equally without considering the hierarchy would overlook their individual significance. Second, learning from state-action pairs in a flattened structure might not fully capture the transitional patterns across different policies.

Based on the high-level states {Pk|k = 1, .., K} and highlevel actions {Πg, g = 1, ...,G}, a high-level reward regulator R can be learned. It is initialized as R = 1. In each iteration, we employ a maximum likelihood inverse reinforcement learning (IRL) [Babes et al., 2011] to update the R, which has demonstrated efficiency in inferring reward functions [Yang et al., 2020]. Based upon R, the reward {rn t |t = 1, ..., T n } for each state-action pair (xnt , ant ) can be calculated as:

In Eq.(4), 1{t ∈ Pk} has the value of 1 if xnt belongs to the subsequence cluster k; otherwise its value is 0. Πg(xnt , ant ) denotes the probability of taking ant at xnt with the g-th policy. Based upon the learned reward rn t , we calculate the ∆rn t for consecutive state-action pairs to regulate the consistency constraint. To balance the effects of time-awareness patterns, i.e., log(e +∆T n t ), and decision-making patterns, i.e., ∆rn t , we employ a bivariate Gaussian distribution Φ to model their interactional regularizations, as shown in Eq.(3).

The introduced reward regulator can capture the interventional patterns and implicitly reflect the effectiveness of treatments. When two consecutive timestamps have similar states but significantly different rewards, it may indicate that the patient's condition has progressed into another stage.

• Sparsity term is responsible for controlling the sparseness of the model using an l1-norm penalty, denoted as λ||Θk||1, where λ is a coefficient. This term encourages the selection of the most significant variables, effectively preventing overfitting by reducing the complexity of the model.

To solve Eq.(1), we employed EM to learn the cluster assignments P and the patterns Θ iteratively until convergence. Specifically, in E-step, we fix Θ to learn P, then Eq.(1) simplifies to include only the log-likelihood term and the consistency term. It can be solved by dynamic programming to find a minimum cost Viterbi path [Viterbi, 1967]; In M-step, by fixing P to learn the Θ, Eq.(1) reduces to include only the log-likelihood term and the sparsity term. It can be formulated as a typical graphical lasso [Friedman et al., 2008] with a Toeplitz constraint over Θ and be solved by an alternating direction method of multipliers [Boyd et al., 2011]. We iteratively perform the E-step and M-step until convergence.

3.2 Policy Induction by EM-EDM

Preliminaries

Energy-based distribution matching (EDM) [Jarrett et al., 2020] is a strictly offline AL method that learns a policy solely from expert demonstrations {Dn } without requiring knowledge about model transitions or off-policy evaluations. It assumes that the {Dn } are carried out with a policy Πθ parameterized by θ, driven by a single reward function.

For simplicity, we will denote a state-action pair as (x, a), omitting their indexes when it does not cause ambiguity. The occupancy measures for the demonstrations and for the learned policy are denoted as ρD and ρΠθ , respectively. The probability density for each state-action pair can be measured as ρΠθ (x, a) = EΠθ [∑∞t=0 γ t1{xt = x, at = a}], where γ is a discount factor. Then, the probability density for each state can be measured by: ρΠθ(x) = ∑a ρΠθ(x, a). The goal of inducing the policy Πθ can be achieved by minimizing the KL divergence between ρD and ρΠθ :

Since Πθ (a|x) = ρΠθ(x, a)/ρΠθ(x), the objective function can be reformulated as:

When it is not possible to execute the policy Πθ in an online manner, estimating ρΠθ (x) in the first term of Eq.(6) becomes challenging. EDM addresses this issue by employing an energy-based model [Grathwohl et al., 2019].

According to energy-based model, the probability density ρΠθ(x) is proportional to e-E(x) , where E(x) is an energy function. The occupancy measure for state-action pairs can be represented as ρΠθ(x, a) = efΠθ (x)[a] /ZΠθ , and the occupancy measure for states can be obtained by marginalizing out the actions: ρΠθ(x) = ∑a efΠθ (x)[a] /ZΠθ . Herein, ZΠθ is a partition function, and fΠθ : R|X| →R|A| is a parametric function that maps each state to |A| real-valued numbers.

The parameterization of Πθ implicitly defines an energybased model over the states distribution, where the energy function can be defined as: EΠθ(x) = -log ∑a efΠθ(x)[a] . Within the scope of the energy-based model, the first term in Eq.(6) can be reformulated as an occupancy loss:

where ∇θLρ(θ) = -Ex∼ρD ∇θ log ρΠθ (x) can be solved by existing optimizers, e.g., stochastic gradient Langevin dynamics [Welling and Teh, 2011]. Thus, by substituting the first term in Eq.(6) with Eq.(7) via energy-based model, we can derive a surrogate objective function to get the optimal solution without the need for online policy rollouts.

EM-EDM

To handle multiple reward functions varying across demonstrations, an EM-based inverse reinforcement learning approach was proposed in [Babes et al., 2011]. It iteratively clusters the demonstrations in E-step and induces policies for each cluster by IRL in M-step. However, in M-step, it relies on IRL methods with discrete states, which may not be scalable for large continuous state spaces like EHRs. To handle the continuous states, motivated by the EM framework and the success of EDM, THEMES employs an extended method-EM-EDM, with EDM in the M-step of EM.

Taking subsequences { ˆDˆn |ˆn = 1, ..., ˆN} learned by RMTTICC as input, where ˆN represents the number of subsequences, the goal of EM-EDM is to cluster these subsequences and learn cluster-specific policies {Πg|g = 1, ...G}, with G being the number of clusters. The prior probability for each cluster is denoted as νg and the policy parameter is denoted as θg. Both νg and θg are randomly initialized. The objective function of EM-EDM is shown in Eq.(8):

where uˆng denotes the probability that subsequence ˆDˆn follows the policy of the g-th cluster. It is defined in Eq.(9), with U being a normalization factor.

During the EM, in the E-step, the probability that subsequence ˆDˆn belonging to the cluster g is calculated by Eq.(9). Then, in the M-step, the prior probabilities are updated as νg = ∑ˆn uˆng/ ˆN, and the policy parameters θg are learned by EDM. The E-step and M-step are iteratively executed until convergence. Finally, the output of EM-EDM consists of the clustered subsequences along with their respective policies.

Based on the subsequence clusters learned by RMT-TICC and the decision-making policies learned by EM-EDM in THEMES, we extract progressive patterns and interventional patterns from the entire input data DEntire . For each timestamp (xt, at), based on Eq.(2), we calculate the probabilities that xt belonging to each progressive stage. Additionally, we calculate the probabilities that (xt, at) follows each policy. These probabilities serve as additional features that are concatenated with the original observations xt, resulting in an augmented input for the downstream early prediction task. In this paper, we fix the early predictor as LSTM [Zhang et al., 2017] and focus on evaluating the effectiveness of feature engineering. We believe that by combining the patterns derived from THEMES with more advanced prediction models (e.g., [Baytas et al., 2017; Zhang et al., 2019], the performance of early prediction can be further enhanced.

4 Experiments

To assess the effectiveness of THEMES-derived patterns for DPM, we applied it to an EHRs dataset obtained from the Christiana Care Health System. This dataset spans over a period of two and a half years, where each sequence represents a patient's visit consisting of a series of observations and corresponding clinicians' interventions.

4.1 Data Preprocessing

Our sepsis-related study cohort comprised 52,919 visits (sequences) with suspected infection, consisting of 4,224,567 timestamps. We conducted data prepossessing as follows:

• Feature selection: We consulted clinicians and selected 14 sepsis progression-related features: 1) Vital signs: systolic blood pressure, mean arterial pressure, respiratory rate, oxygen saturation, heart rate, temperature, fraction of inspired Oxygen; and 2) Lab results: white blood cell, bilirubin, blood urea nitrogen, lactate, creatinine, platelet, neutrophils.
• Missingness handling: We address the missing data by an expert-suggested forward-filling (8 hours for vital signs and 24 hours for lab tests), with the remaining missing values imputed as the mean. This method has shown robustness, especially for septic shock early prediction [Zhang et al., 2019].
• Tagging septic shock visits: Clinical labeling commonly relies on diagnosis codes, such as ICD-9, which are primarily intended for administrative and billing purposes, leading to constrained reliability [Zhang et al., 2019]. To address this issue, our clinicians referred to the Sepsis-3 guidelines [Singer et al., 2016] and established specific rules for identifying septic shock. By combing the ICD-9 codes and clinicians' rules, we identified 1,869 shock and 23,901 non-shock visits. To handle the highly imbalanced ratio, we employed a stratified random sampling technique over the non-shock visits, which

ensured that the dataset maintained the original distribution of age, sex, ethnicity, and stay duration, while achieving a balanced amount. Thus, the non-shock visits were refined to include 1,869 visits, equaling to the number of shock visits.

• States & Actions: a) States are defined based on the 14 continuous features that are relevant to the progression of sepsis. In addition to the original features, we calculate the maximum and minimum values observed within the past 1 hour for each feature. This allows us to capture temporal changes and trends in the data. As a result, the states are represented as 42dimensional vectors. b) Actions are binary values indicating whether specific antibiotics (e.g., clindamycin, daptomycin) have been administered or not. It is well established, as suggested in [Gauer, 2013], that antibiotic therapy plays a crucial role in enhancing the clinical outcomes of sepsis patients.
• Determining Experts' Demonstrations: To enhance the selection of high-quality demonstrations for inducing more accurate policies, we developed a strategy to evaluate the effectiveness of interventions. This strategy involved comparing the early prediction of septic shock before the first intervention to the actual outcome at the onset of septic shock, as depicted in Figure 2. To assess the effectiveness of interventions, we trained multiple LSTMs (100 times) using different hyperparameter settings over the data prior to the first intervention for randomly selected visits. For a given visit, if more than 80% of the LSTMs produce the same early prediction result, we compare this prediction with the actual onset of septic shock: If a patient transitions from shock (prior to the first intervention) to non-shock (onset of septic shock), it indicates the interventions were effective, suggesting the visit to be treated an experts' demonstration for policy induction by THEMES; Otherwise, the visit would not be included during the THEMES modeling process. Using this approach, we identified 195 sequences as experts' demonstrations.

4.2 Experimental Settings

We compared the following methods for evaluating the effectiveness of progressive and interventional patterns in DPM:

• Original: Except for the Original observed state features, none of other additional information will be incorporated;
• MT-TICC: Observed state features are supplemented with additional progressive patterns learned by MT-TICC [Yang et al., 2021]. The additional features are extracted as the probabilities belonging to each subsequence cluster.
• Action: Interventional Actions are directly taken as additional features for the observed state features.
• EDM: An additional interventional feature is extracted by EDM-learned policy [Jarrett et al., 2020], represented as the probability of following the learned policy.
• THEMES 0: Additional progressive and interventional patterns are derived from the probabilities associated with the subsequence clusters learned by MT-TICC and the probabilities adherent to the policies learned by EM-EDM. It is a simplified version of THEMES, where the interventional patterns will not feed back to refine the subsequence clustering.
• THEMES: Based on THEMES, the probabilities belonging to RMT-TICC-learned subsequence clusters and the probabilities following EM-EDM-learned policies are respectively taken as additional progressive and interventional features.

As demonstrated in prior works, EDM outperformed competitive AL methods with a single reward function [Jarrett et al., 2020]. Meanwhile, THEMES 0 and THEMES could induce more accurate policies compared to competitive AL baselines with multiple reward functions (e.g., hierarchical IRL [Krishnan et al., 2016] and multi-modal imitation learning [Hausman et al., 2017]) as well as their ablations (e.g., EM-EDM, and learning progressive stages by MT-TICC then inducing cluster-wise policy by EDM) [Yang et al., 2023]. Therefore, in this paper, we employed EDM, THEMES 0, and TMEMES for extracting AL-derived patterns for DPM.

We utilized Keras to implement the LSTM and performed parameter tuning through grid search. The results were measured when: τ = 12 and τ ∈ [12, 36], for evaluating if septic shock prediction could be achieved 12 hours prior to the onset or even earlier before 36 hours. Each method was repeated 10 times, with the data randomly divided into 80% for training and 20% for testing. The evaluation metrics include Accuracy (Acc), Recall (Rec), Precision (Prec), F1-score (F1), and AUC. All parameters in THEMES are determined by 5-fold cross-validation. In RMT-TICC, the cluster number K is set to 11 based on Bayesian information criteria (BIC) [Friedman et al., 2001], the window size ω is set to 2, and the sparsity λ and consistency β coefficients are set to 1e-5 and 4, respectively. In EM-EDM, the cluster number is determined heuristically as 3, by iteratively applying the EM algorithm until empty clusters are generated or the log-likelihood varies less than a predefined threshold. The THEMES approach uses a threshold of 10 iterations, as our observed that the clustering likelihood for both MT-TICC and EM-EDM converges within 10 iterations. To ensure fair comparisons, optimal parameters for other baselines are also determined by cross-validation.

4.3 Results

The experimental results are reported in Table 1. The best results are in bold highlighted with * and the second-best results are in bold only. Additionally, we provide Critical Difference diagrams with Wilcoxon signed-rank tests over F1 and AUC in Figure 3. In the diagrams, unconnected models indicate pairwise significance at a confidence level of 0.05.

According to the results: a) For progressive patterns: Comparing MT-TICC to Original, the incorporation of progressive patterns in MT-TICC yields significant improvements. Moreover, among the four methods incorporating interventions (i.e., Action, EDM, THEMES 0, and THEMES), the two with progressive patterns (i.e., THEMES 0 and THEMES) outperform the others (i.e., Action and EDM). As a result, the inclusion of progressive patterns enables a better capture of progressive stages, leading to improved early prediction. b) For interventional patterns: Comparing Action and EDM vs. Original, the Action yields similar performance to Original, while EDM shows slightly better performance. Additionally, comparing THEMES 0 and THEMES vs. MTTICC, THEMES 0 is marginally better than MT-TICC, while THEMES further improves the performance. Thus, directly taking actions as additional features via Action cannot fully capture the clinicians' decision-making patterns, while incorporating AL-learned patterns can better reflect the effectiveness of interventions; c) For both progressive and interventional patterns: THEMES performs better than THEMES 0, which indicates the effectiveness of using interventional patterns to refine the learned progressive stages.

Figure 4 shows the F1 and AUC when τ ∈ [1, 36] hours before the onset of septic shock. As τ increases, it becomes harder to early predict across all models. The figures show the advantage of incorporating progressive patterns, since MT-TICC, THEMES 0, and THEMES outperform Original, Action, and EDM. Meanwhile, incorporating interventions is important, and how they are incorporated also matters, as Action and EDM get similar results with Original, while THEMES 0 and THEMES significantly improve the results.

Figure 5 demonstrates the projection of THEMES-derived features for 100 patients whose interventions best match the learned policies (High Match) and 100 patients with the lowest matching rate (Low Match) on a 2D scatter plot utilizing t-SNE. One shock (red) and one non-shock (green) patient with the same length are randomly sampled. Though the start points for the two patients are close, they drift apart as the non-shock patient's treatments (green) align with the High Match group, while the shock patient's treatments (red) align with the Low Match group. It demonstrates that THEMES can capture the effectiveness of treatments for progression to distinguish shock patients from non-shock patients.

5 Conclusions

In this paper, we explore incorporating AL-derived patterns for DPM. Building upon the success of subsequence clustering in extracting progressive patterns for DPM, we leverage an AL approach named THEMES to capture both progressive and interventional patterns. Taking advantage of these patterns, we aim to handle a challenging task for septic shock early prediction. The experimental results demonstrate that the inclusion of THEMES-derived patterns leads to improved accuracy in predicting septic shock at earlier stages. This advancement holds significant potential for assisting clinicians in delivering timely and personalized treatments.

Acknowledgements

This research was supported by the NSF Grants: #2013502, #1726550, #1660878, and #1651909.