MINA: Multilevel Knowledge-Guided Attention for Modeling Electrocardiography Signals

1 Introduction

Heart diseases are among the leading causes of death of the world [Benjamin et al., 2018]. The routine monitoring of physiological signals is deemed important in heart disease prevention. Among existing monitoring technologies, electrocardiography (ECG) is a commonly used non-invasive and convenient diagnostic tool that records physiological activities of heart over a period of time. Deciphering ECG signals can help detect many heart diseases such as atrial fibrillation (AF), myocardial infarction (MI), and heart failure (HF) [Turakhia, 2018; Yanowitz, 2012].

An example of real world ECG signal is shown in Figure1. ECG signals from cases and controls of heart diseases show different patterns at 1) beat level, 2) rhythm level, and 3) frequency level, each representing different anomalous activities of the heart. For example, beat level morphology such as P wave (atrial depolarization) and QRS complex (ventricular depolarization) can reflect conditions related to heart electric conduction. Rhythm level patterns capture rhythm features across beats and reflect cardiac arrhythmia conditions (abnormal heart rhythms). Frequency level is about frequency variations and sheds light on the diagnosis of ventricular flutter and ventricular fibrillation. Learning these patterns to support diagnoses has been an important research area in ECG analysis [Roopa and Harish, 2017; Lake and Moorman, 2010; Linker, 2016; Tateno and Glass, 2001].

In real clinical settings, in addition to the demand of an accurate classification, the interpretability of the results is equally important [Tsai et al., 2003]. Cardiologists need to provide both diagnosis and detailed explanations to support diagnosis [EC57, 2012]. Also, many heart diseases do not pose abnormal ECG diagram constantly [Benjamin et al., 2018; Yanowitz, 2012], especially during the early stage of the diseases. Therefore, interpretability of the results, particularly highlighting diagnosis related parts of the data, is crucial for early diagnosis and better clinical decisions.

Traditional machine learning methods either learn time domain patterns including beat level [Ladavich and Ghoraani, 2015; Pürerfellner et al., 2014] and rhythm level [Huang et al., 2011], or extract frequency patterns using signal processing techniques such as discrete wavelet transform [García et al., 2016]. However, time domain approaches are easily affected by noise or signal distortion [Rodríguez et al., 2015]; while frequency domain methods cannot model rare events or some temporal dynamics that occur in time domain. Besides, they all require laborious feature engineering, and their performance also relies on the quality of the constructed features.

Recently, deep learning models showed initial success in modeling ECG data. Convolutional neural networks (CNN) were used to learn beat level patterns [Kiranyaz et al., 2016; Rajpurkar et al., 2017; Hannun et al., 2019]. Recurrent neural networks (RNN) are suitable for capturing rhythm features [Schwab et al., 2017; Hong et al., 2017; Zihlmann et al., 2017]. Moreover, attention mechanism is employed to extract interpretable rhythm features [Schwab et al., 2017]. Despite their progress, these models were either black-box or only highlighted one aspect of patterns (such as rhythm features as in [Schwab et al., 2017]), thus lack the comprehensive interpretability of the results for real clinical usage.

In this work, we propose MultIlevel kNowledge-guide Attention model (MINA) to learn and integrate different levels of features from ECG which are aligned with clinical knowledge. For each level MINA extracts level-specific domain knowledge features and uses them to guide the attention, including beat morphology knowledge that guides attentive CNN and rhythm knowledge that guides attentive RNN. MINA also performs attention fusion across time- and frequency domains. We proposed new evaluation approaches by interfering ECG signals with noise and signal distortion. We evaluated interpretability and robustness of the model by tracking intermediate reactions across layers frommultilevel attentions to the final predictions.

Experimental results show MINA can correctly identify critical beat location, significant rhythm variation, important frequency component and remain robust in prediction under signal distortion or noise contamination. Tested on the atrial fibrillation prediction, MINA achieved PR-AUC 0.9436 (outperforming the best baseline by 5.51%). Finally, MINA also showed strong result interpretability and more robust performance than baselines.

2 RelatedWork

Traditional methods include time domain methods such as beat level methods [Ladavich and Ghoraani, 2015; Pürerfellner et al., 2014] and rhythm level ones [Tateno and Glass, 2001; Huang et al., 2011; Oster and Clifford, 2015], both depending on segmentation by detecting QRS complex. However, time domain methods rely on the accuracy of QRS detection, thus are easily affected by noise or signal distortion. Frequency domain approaches, on the other hand, cannot model rare events and other time-domain patterns and thus lack interpretability. Moreover, both types of features are subjective.

Recently, deep neural networks (DNNs) have been used in ECG diagnosis [Kiranyaz et al., 2016; Rajpurkar et al., 2017; Hannun et al., 2019; Zihlmann et al., 2017; Hong et al., 2017; Schwab et al., 2017]. Many of them have demonstrated stateof-the-art performance due to their ability in extracting effective features [Rajpurkar et al., 2017; Hong et al., 2017]. Some of them build an end-to-end classifier [Kiranyaz et al., 2016; Rajpurkar et al., 2017; Zihlmann et al., 2017], others build a mixture model which combines traditional feature engineering methods and deep models [Hong et al., 2017; Schwab et al., 2017; Hong et al., 2019]. However, existing deep models are insufficient in three aspects. First, they neglect the characteristics of ECG signals when design model architecture, namely, beat morphological, rhythm variations. Second, they only analyze ECG signals in time domain. Last, they are "black-box" and thus not interpretable. In real world medical applications, interpretability is critical for clinicians to accept machine recommendations and implement intervention.

3 Method

In this section, we will introduce the model design of MINA. Section 3.1 provides an overview and introduces all notations. Section 3.2 describes the basic framework, including each layer of MINA. Section 3.3 proposes our new attention mechanism which is integrated in MINA. Section 3.4 describes how we evaluate interpretability and robustness. Figure2 depicts the architecture of MINA.

3.1 Overview of MINA

Here we briefly describe the framework and introduce notations used in this paper. Assume we are given a single lead ECG signal x ∈ Rn and use it to predict class probability. We firstly transform it into multi-channel signals with F channels across different frequency bands where ith signal is denoted as x(i) ∈ Rn . We then split each x(i) into M segments s(k) . Next we apply CNN and RNN consecutively on s(k) to obtain beat level attention o(k) , 1 ≤ k ≤ M and rhythm level attention c(i) . This follows by a fully connected layer that transforms c(i) into q(i) . We then take weighted average to integrate Q = [q(1) , ..., q(F) ] across all channels to output frequency attention d, which will be used in prediction. To improve model accuracy and interpretability, we propose a knowledge guided attention to learn attention vectors from beat-, rhythm-, and frequency levels, denoted as α, β, and γ respectively. More details will be described in Section 3.2. The notations are summarized in Table 1. Detailed configurations of MINA are shown in the Experiments section.

3.2 Description of MINA

Signal Transformation and Segmentation

In order to utilize the frequency-domain information, we employ an efficient strategy by decomposing original ECG signals into different frequency bands (where each band is regarded as a channel). Then we can concurrently model signals of each channel.

Specifically, we propose a new time-frequency transformation layer to transform a single lead ECG signal into multichannel ones. Here we use Finite Impulse Response (FIR)

bandpass filter [Oppenheim et al., 1996] to transform single lead ECG signal x into F multi-channel ECG signals X = [x(1) , x(2) , ..., x(F ) ].

Then for each channel, we split x(i) ∈ Rn into a sequence of M equal length segments. Unlike previous deep models [Schwab et al., 2017; Kiranyaz et al., 2016] that perform segmentation using QRS complex detection, which is easily affected by signal quality, we simply use sliding window segmentation. By cutting each of ith segment is indexed by (i - 1) × T and i × T - 1, we receive M equal length segments s ∈ RT (without the loss of generality, we assume that n = M ∗ K, otherwise we can cut off last remain part which is shorter than T). In general, segment length T needs to be shorter than the length of one heart beat, so that we can extract patterns in beat level. Detailed configurations can be found in Implementation Details section.

Beat Level Attentive Convolutional Layer

For beat level patterns, we mainly consider the abnormal wave shapes or edges. To locate them from signals, we design an attentive convolutional layer. Formally, given M segments s ∈ RT , we perform 1-D convolution on each of them and output convolved features: L = Conv(s), L ∈ RK×N , K is the number of filters, N is the output length of segments after convolution, which is determined by hyperparameters like stride of CNN. Conv operations are shared weights of M segments. Then instead of traditional global average pooling which treats all features homogeneously, we propose a knowledge-guided attention to aggregate these features and get beat level attention o = ∑Nj=1 αjl(j) , where αj represents the weight for convolved features, l(j) ∈ RK is the jth column in L, 1 ≤ j ≤ N. Thus the model can focus more on significant signal locations and have better beat level interpretation. Details of knowledge-guided attention will be introduced in Section 3.3.

Rhythm Level Attentive Recurrent Layer

For rhythm level patterns, we mainly consider the abnormal rhythm variation. To capture them from beat sequences, RNNs are a natural choice due to their abilities to learn on data with temporal dependencies. Again to improve interpretability and accuracies, we use knowledge guided attention with rhythm knowledge.

Specifically, we use a bidirectional Long Short-Term Memory network [Schuster and Paliwal, 1997] (Bi-LSTM) to get rhythm level annotations of segments. The bidirectional LSTM is denoted here as h(k) = BiLSTM(o(1) , ..., o(k) ). We concatenate the forward and backward outputs of BiLSTM and receive the rhythm level feature H ∈ RJ×M , H = [h(1) , ..., h(M) ], 1 ≤ k ≤ M. Here we use knowledgeguided attention with rhythm knowledge to output the rhythm level attention c = ∑Mk=1 βkh(k) , where βk represents the weight of kth rhythm level hidden state h(k) .

Fusion and Prediction

At the beginning we decompose ECG signals into multiples channels (i.e., frequency bands) and learn rhythm level features {c(i) } from each channel i. Now we will perform attention fusion across all channels to have a more comprehensive view about the signal.

We first perform fully connected transformation: Q = W T c C ⊕ bc, where C ∈ RJ×F , C = [c(1) , ..., c(F) ], Wc ∈ RJ×H , bc ∈ RH and Q ∈ RH×F . ⊕ means broadcasting bc to all N column vectors inW T c C and applies addition. Then, since the importance of these channels may not be homogeneous, we will take weighted average of q(i) to calculate frequency level attention d = ∑Fi=1 γiq(i) where γi is the weight of q(i) , Q = [q(1) , ..., q(F) ], 1 ≤ i ≤ F. We use frequency knowledge, signals with greater energy are more informative, to determine the weight γ. Here we use power spectral density to measure energy.

Last, given integrated features d we make prediction using p = softmax(W T d + b), where W ∈ RH×C , d ∈ RH , b ∈ RC , and optimize the weighted cross entropy loss CE(p) = - ∑Cc=1 I{zc = 1}wc log pc, where C is the number of classes, z is the ground truth , w is the weight vector with the same shape as z, I is the indication function. w is adjusted to handle with class imbalance problem which is common in medical area.

3.3 Knowledge Guided Attention of MINA

We now describe how to compute multilevel attention weights α, β, γ. The attention mechanism can be regarded as a twolayer neural network: the 1st fully connected layer calculates the scores for computing weights; the 2nd fully connected layer computes the weights with via softmax activation.

In the first layer, the scores are computed based on the following features. (1) Multilevel outputs L ∈ RK×N , H ∈ RJ×M , Q ∈ RH×F extracted by MINA. (2) Domain knowledge features including beat level Kα ∈ REα×N , rhythm level Kβ ∈ REβ×M , and frequency level Kγ ∈ REγ×F . Concretely, three levels of domain knowledge features can be represented as below.

• Beat Level Kα: For beat level knowledge we mainly consider the abnormal wave shapes or sharply changed points such as QRS complex [Kashani and Barold, 2005]. To represent it we compute first-order difference ∆ and a convolutional operation Convα on each segment s to extract the beat level knowledge feature Kα = Convα(∆(s)), ∆(s)i = si - si-1 and ∆(s)0 = s0, si is the ith value in s. Detailed configurations of Convα are introduced in Implementation Details section.
• Rhythm Level Kβ: Attention weights β focus on rhythm level variation, such as severe fluctuation in ventricular fibrillation disease [Yanowitz, 2012]. To characterize it we compute standard deviation on each segment in S to extract the rhythm level knowledge feature vector Kβ = D(S), where D calculate standard deviation of each s in S, D(s) = 1 T ∑(si - ¯ s)2
• Frequency Level Kγ: On frequency level, signals with greater energy contain more information and thus need more attention [Yanowitz, 2012]. So we use power spectral density (PSD), a popular measure of energy, to extract the frequency level knowledge feature vector Kγ = F(X), where F calculate PSD [Oppenheim et al., 1996] using a periodogram of each x(i) inX.

Then, we concatenate model outputs and knowledge features to compute scores and attention weights.

where, Wα ∈ R(K+Eα)×Dα ,Wβ ∈ R(J+Eβ)×Dβ ,Wγ ∈ R(H+Eγ)×Dγ , bα ∈ RDα , bβ ∈ RDβ , bγ ∈ RDγ represent weights and biases in the first layer, Vα ∈ RDα×1 , Vβ ∈ RDβ×1 , Vγ ∈ RDγ×1 represent weights in the second layer. ⊕ is addition with broadcasting.

3.4 Method for Evaluating Interpretability and Robustness

To evaluate the interpretability and robustness of MINA, we perturb the signals and observe attention weights and prediction results. The evaluation method is illustrated in Figure 3.

Concretely, we add signal distortion (low frequency interferer) or noise (high frequency interferer) to the original ECG signal x and get x′ , here we choose baseline signal distortion and white noise. For the perturbed signals x′ , we applied MINA to generate prediction p′ and output multilevel attention weights α′ , β′ , γ ′ . We compare them with the original results p and α, β, γ from unperturbed data.

To evaluate the interpretability of MINA, we visually check whether attention weights are in line with medical evidences. For beat level attention weights of M segments A = [α(1) , ...,α(M) ] ∈ RM∗N and A′ = [α′(1) , ...,α′(M) ] ∈ RM∗N , we align them to input ECG signals x ∈ Rn , where the ith attention weight Aj approximately corresponds from x⌊ n∗j M∗N ⌋ to x⌈ n∗(j+1) M∗N ⌉ . Then we visualize the values and verify whether high Aj relates to beat level medical evidence. For rhythm level attention weight β and β′ , we align them to M segments S = [s(1) , ..., s(M) ], where βk corresponds to s(k) . Then we verify whether high βk relates to rhythm level medical evidence. For frequency level attention weight γ and γ ′ , we align them to F channels X = [x(1) , ..., x(F) ], where γi corresponds to x(i) . Likewise, we check whether high γi relates to frequency level medical evidence.

We evaluate the robustness of MINA based on the two tasks: (1) we visually compare whether the new attention weights after perturbation are still in line with medical evidences, using the same way above, (2) we gradually change the interfered amplitude and evaluate the overall performance changes. The more robust model will be less impacted. Moreover, these results can also be used to evaluate interpretability, since interpretable model can highlight meaningful information, while also suppress unrelated parts.

4 Experiments

In this section, we first describe the dataset used for the experiments, followed by the description of the baseline models. Then we discuss the model performance.

4.1 Source of Data

We conducted all experiments using real world ECG data from PhysioNet Challenge 2017 databases [Clifford et al., 2017]. The dataset contains 8,528 de-identified ECG recordings lasting from 9s to just over 60s and sampled at 300Hz by the AliveCor device, 738 from AF patients and 7790 from controls as predefined by the challenge. We first divided the data into a training set (75%), a validation set (10%) and a test set (15%) to train and evaluate in all tasks. Then, we preprocess them to get equal length data, where n = 3000. The summary statistics of the data is described in Table 2. In this study, the objective is to discriminate records of AF patients from those of controls.

4.2 Baseline Models

We will compare MINA with the following models:

• Expert: A combination of extracted features used in AF diagnosis including: rhythm features like sample entropy on QRS interval [Lake and Moorman, 2010]; cumulative distribution functions [Tateno and Glass, 2001]; thresholding on the median absolute deviation (MAD) of RR intervals [Linker, 2016]; heart rate variability in Poincare plot [Park et al., 2009]; morphological features like location, area, duration, interval, amplitude and slope of related P wave, QRS complex, ST segment and T wave; frequency features like frequency band power. We used QRS segmentation method in [Pan and Tompkins, 1985] and trained an LR classifier using these features. Then, we build both logistic regression (ExpertLR) and random forest (ExpertRF) on above extracted features.
• CNN: Convolutional layers are performed on ECG segments with shared weights. We use global average pooling to combine features, and fully connect (FC) layer and softmax layer for prediction. The model architecture is modified based on [Kiranyaz et al., 2016] to handle ECG segments. The hyper-parameters in CNN, FC and softmax are the same as MINA to match the model complexity.
• CRNN: We used shared weights convolutional layers on ECG segments, and replaced the global average pooling with bi-directional LSTM. Then FC and softmax are applied to the top hidden layer. The architecture is modified based on [Zihlmann et al., 2017], but only keep one convolutional layer. Other hyper-parameters in CNN, RNN, FC and softmax are the same as MINA.
• ACRNN: Based on CRNN, with additional beat level attentions and rhythm level attentions. Other hyperparameters are the same as MINA.

4.3 Implementation Details

In convolutional layers of CNN, CRNN, ACRNN and MINA, we use one layer for each model. The number of filters is set to 64, the filter size is set to 32 and strider is set to 2. Pooling is replaced by attention mechanism. Convα of Kα has one filter with size set to 32, the strider is also 2. In recurrent layers of CRNN, ACRNN and MINA, we also use one single layer for each model, the number of hidden units in each LSTM is set to 32. The dropout rate in the fully connected prediction layer is set to 0.5. In sliding window segmentation, we use non-overlapping stride with T = 50. Deep models are trained with the mini-batch of 128 samples for 50 iterations, which was a sufficient number of iterations for achieving the best performance for the classification task. The final model was selected using early stopping criteria on validation set. We then tested each model for 5 times using different random seeds, and report their mean values with standard deviation. All models were implemented in PyTorch version 0.3.1, and trained with a system equipped with 64GB RAM, 12 Intel Core i7-6850K 3.60GHz CPUs and Nvidia GeForce GTX 1080. All models were optimized using Adam [Kingma and Ba, 2014], with the learning rate set to 0.003. Our code is publicly available at https://github.com/hsd1503/MINA.

4.4 Performance Comparison

Performance was measured by the Area under the Receiver Operating Characteristic (ROC-AUC), Area under the PrecisionRecall Curve (PR-AUC) and the F1 score. The PR-AUC is considered a better measure for imbalanced data like ours [Davis and Goadrich, 2006]. Table 3 shows MINA outperforms all baselines, and shows 5.51% higher PR-AUC than the second best models.

5 Interpretability and Robustness Analysis 5.1 MINA Automatically Extracts Clinically Meaningful Patterns

When reading an ECG record (upper left in Figure 4), cardiologists will make AF diagnosis based on following clinical evidences: 1) the absence of P wave: a small upward wave before QRS complex; 2) the irregular RR interval such as the much wider one between the 4th and the 5th QRS complex.

MINA learns these patterns automatically via beat-, rhythm, and frequency level attention weights. From Figure 4, the beat level attentions point to where QRS complex or absent P waves occur. The rhythm level attentions indicate the location of abnormal RR interval, which precisely matches the clinical evidence. Besides, from the frequency level attentions, we notice channel 10Hz-50Hz receives the highest attention weight so MINA pays more attention to it. In fact, QRS complex, the most significant clinical evidence in ECG diagnosis, is known to be dominant in 10Hz-50Hz [Tateno and Glass, 2001; Linker, 2016; Lake and Moorman, 2010].

5.2 MINA Remains Interpretable and Robust Against Baseline Signal Distortion

The baseline wander distortion is a low frequency noise with slow but large changes of the signal offset. It is a common issue that drops ECG analysis performance. In this experiment, we mimic the real world setting by distorting data and observe whether MINA can still make robust and interpretable predictions.

For the experiment we interfered the signal in Figure 4 with baseline wander distortion. The interfered signal is plotted in Figure 5(a). From the original frequency attention in Figure 4, it is easy to see Channel 1 (<0.5Hz) has the lowest weights, while Channel 2 (0.5Hz-50Hz) weights much higher. Thus Channel 1 can be interpreted as baseline component while Channel 2 as clean signal component. MINA pays more attention to Channel 2 than Channel 1. After signal distortion, the importance of both channels remain the same, which is also reflected from their beat level and rhythm level attentions. Channel 1 shows no significant patterns, but the more informative Channel 2 have similar beat- and rhythm level patterns as unperturbed data, which indicates the interpretability of MINA will be less impacted by data distortion.

To evaluate model robustness, we compare the performance change along the increase of distortion amplitude on the entire test set. As shown in Figure 5(d), MINA still has much lower performance drop even after distortion by large amplitude. While all baselines start to have large performance drop even with little distortion. This is mainly thanks to frequency attention fusion. In training process, the model already identified Channel 1 a baseline signal. Thus baseline distortion will have less impact on important signals in clean signal channel. Since baseline signal distortion occurs in real clinical setting, MINA will provide more accurate prediction in these scenarios.

5.3 MINA Remains Interpretable and Robust in the Presence of Noise

The high frequency noise contamination is another common issues. For this experiment, we perturbed the signal in Figure 4 with white noise. The perturbed signal is in Figure 6(a). Similar to last experiment, from original frequency attentions we know Channel 3 (>50Hz) has lower weights. It is a channel known for high noise. While Channel 2 (0.5Hz-50Hz) weights much higher and is known as a clean signal channel.

After noise contamination, the noise impacts more to the noise Channel which is less important in the prediction of MINA, but the more informative Channel 2 have similar beatand rhythm level patterns as unperturbed data, which indicates the interpretability of MINA will be less impacted by noise contamination. In Figure 6(d), we compare the PR-AUC change along the increase of noise amplitude on the entire test set. MINA is less impacted by noise than other methods, demonstrating more robust performance in the presence of noise thanks to frequency attention fusion.

Figure 6: (a) Signal in Figure 4 perturbed by noise. (b) Channel 3 (lower attention weights) shows no significant patterns. (c) Channel 2 (higher attention weights) remains meaningful patterns similar to original data. (d) MINA has much lower PR-AUC drop % than baselines.

6 Conclusion and FutureWork

In this paper, we propose MINA, a deep multilevel knowledgeguided attention networks that interpretatively predict heart disease from ECG signals. MINA outperformed baselines in heart disease prediction task. Experimental results also showed robustness and strong interpretability against signal distortion and noise contamination. In future, we can extend to a broad range of disease where ECG signals can be treated as additional information in the diagnosis, on top of other health data such as electronic health records. Then we will need to investigate interpretable prediction based on multimodal data, which is a possibly rewarding avenue of future research.

Acknowledgements

This work was supported by the National Science Foundation, award IIS-1418511, CCF-1533768 and IIS-1838042, the National Institute of Health award 1R01MD011682-01 and R56HL138415. We also thanks valuable discussions with Li Jiang from BOE.