K-margin-based Residual-Convolution-Recurrent Neural Network for Atrial Fibrillation Detection

1 Introduction

Atrial Fibrillation (AF) is an abnormal heart rhythm characterized by rapid and irregular beating of the atria, which can trigger cardiac arrest and sudden death as a consequence of pumping blood less effectively [Ye et al., 2010; Kass and Clancy, 2005]. The Electrocardiograph (ECG) has been a cornerstone for the detection and diagnosis of such condition for a long time. However, its interpretation is mostly done by medical experts due to the high error rates of computerized interpretation. It has been found that good performance was shown on carefully-selected often clean data [Clifford et al., 2017], and only about 66% of AF predictions were correctly recognized from noisy ECG data [Shah and Rubin, 2007]. This is especially problematic in developing countries, where the availability of clinics and medical experts is low [Gradl et al., 2012] [Silva et al., 2011]. Accordingly, there is a definite need for automatic, accurate physiological monitoring solutions for AF detection (without domain knowledge and features given by domain experts in advance), which can be used in home or ambulatory settings.

∗Contact Author

Despite the significance of this problem, it is challenging to reliably detect AF from noisy ECGs due to the following aspects. First, there is insufficient training data for AF detection, since well-labelled ECG recordings are hard to acquire. Second, the inevitable existence of class skewness in ECGs (as diseases happen rarely) may result in poor performance. Then, it is hard to capture both local heartbeat-level characters and long-term rhythm-level trend in an ECG record. Moreover, there may be semantical ambiguities caused by noisy segments in an ECG record as shown in Figure 1. Learning features from these noisy segments will lead to poor Deep Neural Network (DNN) models.

In this paper, we propose a K-margin-based ResidualConvolution-Recurrent neural network (K-margin-based RCR-net) for AF detection from noisy ECGs. We first employe a skewness-driven dynamic augmentation method to handle problems of data inadequacy, class imbalance and high computing cost in ECG analysis. Then, we propose a multi-view learning method based on a novel ResidualConvolution-Recurrent neural networks (RCR-net) to automatically extract both long-term rhythm-level and local heartbeat-level characters. Besides, a K-margin-based diagnosis model is presented to identify the classification of each ECG by selecting its most probable label of relevant top-K augmented segments with the least certainty margin, which can automatically focus on the most important segments associated with an ECG record and handle noise by naturally exploiting expected consistency among the segments associated for each record. We carry out thorough experiments on PhysioNet/Computing in Cardiology Challenge 2017 Dataset [Clifford et al., 2017]. The experimental results demonstrate that the proposed method with 0.8125 F1NAOP score outperforms all state-of-the-art deep learning methods for AF detection task by 6.8%

2 RelatedWork

AF detectors can be divided into two categories including feature engineered methods and deep learning based methods.

The feature engineered methods are highly related to cardiological knowledge which can achieve high accuracy if the recorded ECG signals are clean and high-resolution. In AF detection task, a process of using cardiological knowledge to extract informative values is essential to make classifiers able to distinguish AF patients from the population. Although engineering effective features is difficult and time-consuming, there are many excellent works that proposed various kinds of features and got good detection performance [Tateno and Glass, 2001; Linker, 2016; Carrara et al., 2015]. These methods help cardiologist to promote medical service quality in hospital. However, the biggest problem is that it's hard to extract accurate features from noisy contaminated ECGs.

Recently, deep learning-based methods like Convolutional Neural Network (CNN) [Kiranyaz et al., 2015; Xiong et al., 2017; Andreotti et al., 2017; Chandra et al., 2017; Kamaleswaran et al., 2018; Hannun et al., 2019], Recurrent Neural Network (RNN) [Schwab et al., 2017], Convolutional Recurrent Neural Network (CRNN) [Zihlmann et al., 2017], and combined methods [Hong et al., 2017; Hong et al., 2019] have achieved success in AF detection due to their powerful automatic feature learning ability. However, as far as we are concerned, these methods still have high error rates for ECG diagnosis when encountering insufficient training data, class skewness, as well as the semantical ambiguities of noisy ECGs.

3 Methods

3.1 ProblemDefinition and General Framework

AF detection is the task of automatically classifying an ECG into one of cardiac arrhythmia classes. Formally, we denote the training dataset as D = {X, Y } where X = [x(1) , x(2) , . . . , x(N) ] are ECG sequence inputs, Y = [y(1) , y(2) , . . . , y(N) ] the corresponding label set and N the total number of training data. Given the labelled training dataset D, our goal is to learn a predictive model which takes an unlabelled ECG sequence x(i) as input and outputs the prediction ˆ y(i) ∈ C where C = {c1, c2, . . . , cm} is a set of m different rhythm classes.

In this paper, we consider a DNN model as the basic sequence classifier. Recent deep learning works have demonstrated significant success on sequential classification tasks, that can naturally integrate and extract hierarchy features automatically. For the purpose of benefiting from both automatic feature extraction from CNN and capturing the longterm trend from RNN, we employ a Residual-ConvolutionalRecurrent Neural Network (RCR-net) as shown in Figure 2.

Nevertheless, a RCR-net model still can't be directly applied for reliably detecting AF from noisy ECG segments. On the one hand, ECGs are generated continuously with hundreds of millions of points of each patient, training a DNN model on such a long time period may result in high computing complexity. On the other hand, the class skewness and semantical ambiguities caused by noisy ECG segments may lead to poor, or even unacceptable quality of DNN models.

To solve the aforementioned problems, we adapt a Kmargin-based learning approach. The framework of Kmargin-based RCR-net is shown in Figure 2. In detail, we firstly preprocess raw data using Skewness-driven Dynamic Data Augmentation (Sec. 3.2) to relieve the class imbalance problem. Then, the RCR-net model (Sec. 3.3) is trained using the labelled augmented segments. It is noteworthy that only a portion of the segments in each record participate in the RCR-net learning process. Finally, a K-margin-based diagnosis model (Sec. 3.4) is employed to identify the classification of each ECG record by automatically focusing on the most important segments and handling noise.

3.2 Skewness-driven Dynamic Data Augmentation

As introduced before, to solve or relieve the problems of inadequate data, classes skewness and high computing complexity, we propose a skewness-driven dynamic augmentation method to handle these problems. Specifically, we deploy slide-and-cut methodology to generate more short-term ECG segments based on original long ECG records. Usually, slideand-cut needs two predefined parameters: 1) windows size (denoted as w): length of cut, 2) stride (denoted as s): length of slide. In our dynamic data augmentation process, we employ a skewness-driven dynamic stride. The setting of stride gets smaller for records whose labels are scarce, and gets larger for records whose labels are common, constrained by a maximum stride threshold. Formally, given a maximum stride threshold MS and m labels C = {c1, c2, . . . , cm}, the stride of record with label cj is given by:

Notice that if one ECG record has less than windows size length, we pad it with zeros at the end. Therefore, a Tsegments array X(i) = [x(i) 1 , x(i) 2 , . . . , x(i) T ] can be generated from an ECG record x(i) ∈ X , where t ∈ {1, 2, . . . , T}-th segment x(i) t is a continuous real-valued vector [x(i) t1 , x(i) t2 , . . . , x(i) tw ] ∈ Rw and w is the window size. The label y(i) of i-th record x(i) is assigned to all the segments in its augmented segments array X(i) . For clarity, we denote a segments labels set as Y (i) = [y(i) 1 , y(i) 2 , . . . , y(i) T ] for each ECG sequence x(i) where y(i) t is the corresponding label for t-th segment x(i) t .

3.3 Multi-viewResidual-Convolution-Recurrent Neural Network

Recently, DNN methods have achieved great success in AF detection [Wu et al., 2016; Majumdar and Ward, 2017; Kiranyaz et al., 2015; Hannun et al., 2019]. To build a more general deep learning method which could automatically capture anti-noise characteristics for a wider range of arrhythmias, we want to extract and integrate both local heartbeatlevel characters and long-term rhythm-level trend from an ECG record, namely multi-view deep features. From this point of view, we design a multi-view residual-convolutionrecurrent neural network architecture (RCR-net) to benefit from both the power of automatic feature extraction using CNN and the long-term trend captured using RNN. The highlevel architecture of the RCR-net is shown in Figure 3. The network takes augmented ECG segments as input, and outputs the predictions.

In detail, RCR-net consists of a 33-layer stacked residual block [He et al., 2016], 1-layer recurrent block and 1-layer fully connected block. The residual block layers aim to automatically extract more effective local heartbeat-level features by constructing a very deep model via residual connections between blocks. The recurrent layer is designed to capture underlying rhythm-level structure in ECG data. Here, we use Bi-directional Long-Short Term Memory (Bi-LSTM) cells which can capture long term trend by utilizing a gated architecture. Then, the predictions are made by a fully connected layer and a softmax layer. Finally, we compute crossentropy loss for objective function, and optimize the loss for training the neural network.

3.4 K-margin-based Diagnosis Model

Skewness-driven dynamic data augmentation is essential to boost the model performance by handling the problems of data inadequacy, class imbalance and high computing complexity in ECG analysis. However, as shown in Figure 4, it inevitably generates "hard" ECG segments due to noisy labels. Therefore, to enhance the robustness of RCR-net, we propose to compute cross-entropy objective function of only a portion of the selected ECG segments.

In order to select the proper segments for computing crossentropy, we first propose a multi-class uncertainty variant measurement called least uncertainty margin. Given a trained RCR-net model, one can get all the predictions of augmented ECG segments for each ECG record. Then, we define the uncertainty margin of t-th segment x(i) t for the given ECG record x(i) under trained RCR-net model as:

where ˙ˆ y(i) t and ¨ˆ y(i) t are the most and second-most probable predicted classes of x(i) t using the RCR-net model. Intuitively, segment with the least uncertainty margin is the most confident one for the given ECG record under the trained model, because the trained model has little doubt in differentiating between the two most probable classes. By contrast,

Algorithm 1 K-LABELS(X(i) , K, φ)

Input: A T-segments array X(i) = [x(i) 1 , x(i) 2 , . . . , x(i) T ] of an input ECG record x(i) , trained RCR-net model φ

Parameter: An integer K

Output: Top-K most confident segments X(i) 1:K and their label predictions ˆ Y (i) 1:K

• 2: while k > 0 do
• 3: x(i) ∗ ←Most confident segment predicting byφ(X(i) ) using Eq. 3
• 4: ˆ y(i) ∗ ← Most confident label predicting by φ(X(i) ) using Eq. 4
• 5: X(i) ← X(i) ) \ x(i) ∗ # Implementation: remove segment index of x(i) ∗ from segment indexes list of X(i)

• 8: end while
• 9: returnX(i) 1:K , ˆ Y (i) 1:K

segments with larger margins are more ambiguous. Therefore, we define the segment with least uncertainty margin of i-th input ECG record x(i) with T-segments array under trained RCR-net model as:

where x(i) t is the t-th segment of the input ECG record xi. Least margin diagnosis aims to find the predicted classes of an ECG record by applying the most confident strategy. In a similar way, the most confident label of an input ECG record x(i) under trained RCR-net model is defined as follows:

Based on the above discussion, the proposed K-marginbased diagnosis algorithm (K-LABELS) is described in Algorithm 1. It takes an ECG record x(i) as an input, and the top-Kmost confident segments and their labels under trained model φ can be identified by it. Given an ECG record x(i) and its augmented segments array X(i) = [x(i) 1 , x(i) 2 , . . . , x(i) T ], it requires K iterations where K is the number of most confident segments associated with it in Algorithm 1. The top-1 most confident segment and its label needs to be found out from the segment array in iteration, and then it will be removed from the segment array. Given m rhythm labels C = [c1, c2, ..., cm], after K iterations, the top-Kmost confident segments X(i) 1:K and their label ˆ Y (i) 1:K for the given ECG record x(i) under trained model φ can be identified.

To avoid learning features from the noisy segments, it is expected that the model can automatically focus on the most important segments which could represent the features of the given ECG record. Intuitively, the top-K most confident segments X(i) 1:K can be used for the training process, instead of the whole segments array X(i) . However, as they are selected by the RCR-net model, it is unreliable to use them for training when the model is unreliable. Therefore, for each ECG record X(i) , we can compute the average probabilistic prediction of all of its augmented segments:

where T is the number of augmented segments within candidate x(i) , P( ˙ˆ y(i) t |x(i) t ) is the prediction probability of t-th segment of x(i) . If αi > 0.5, we select the top-K most confident segments X(i) 1:K of given record x(i) for model finetuning process to achieve higher consistency among the augmented segments within a record and reduce the effects of noisy segments. Otherwise, we select the top-K segments of its inverted sequence. That is because a low average probabilistic prediction of all of its augmented segments indicate a poor performance of our model. Therefore, the prediction of our model is inaccurate and unreliable, and we need more "hard" segment samples for training. The selected αsegments X(i) α can be defined as follows:

Therefore, the task of AF detection using our K-marginbased RCR-net model can be expressed as optimizing the cross-entropy objective function:

We further transform the label predictions ˆ Y (i) 1:K of the topK most confident segments for the given ECG record x(i) under trained model φ to a matrix ˆ Y (i) 1:K ∈ RK×m as follows:

Thus, we propose a voting algorithm based on the top-K uncertainty variant measurement. To identify the most probable label of a given ECG record x(i) , we employ a K-marginbased diagnosis method to vote on the probability of rhythm class the ECG record x(i) might belong to. The most probable label ˆ y(i) of an ECG record x(i) can be defined as follows:

Our K-margin-based RCR-net model can automatically focus on the most important segments associated with an ECG record and handle noise, as only a portion of the segments in each record participate in the learning process, by naturally exploiting expected consistency among the segments associated for each record. In our experiment, we demonstrate that our K-margin-based diagnosis can further improve the robustness and accuracy in the AF detection task.

4 Experiments and Results

4.1 Experimental Setup

Dataset

We carry out experiments on PhysioNet/Computing in Cardiology Challenge 2017 Dataset [Clifford et al., 2017], which contains 8,528 single lead ECG records lasting from 9 s to just over 60 s sampling with 300 Hz. In this dataset, all nonAF abnormal rhythms are treated as a single class, namely Other rhythm. Thus these records are classified as 4 classes: 1) Normal sinus rhythm N, 2) Atrial Fibrillation A, 3) Other rhythm O, 4) Too noisy to classify P.

Evaluation Measurements

Commonly used multi-class metrics such as Precision, Recall and Hamming Loss are utilized to measure the performance by evaluating how close the predicting labels are to corresponding labels given by doctors:

• Precision = 1m ∑mc=1 ∑i∈{i|y(i)=c} 1(y(i)=ˆ y(i) ) |{i|y(i)=c}| .
• Recall = 1m ∑mc=1 ∑i∈{i|y(i)=c} 1(y(i)=ˆ y(i) ) |{i| ˆ y(i)=c}| .
• Hamming Loss = 1N ∑Ni=1 1(y(i) ̸=ˆ y(i) ) m .

We also use the following F1 evaluation measurements:

• F1 scores of each class: Denoted as F1N for normal sinus rhythm, F1A for AF, F1O for other rhythm, F1P for noise. Detailed definitions can be found in [Clifford et al., 2017].
• Averages of F1 scores: F1NAO = F1N+F1A+F1O 3 , F1NAOP = F1N+F1A+F1O+F1P 4 .

Implementation Details

In implementation, we randomly split 80% for model training, and evaluate on remaining 20% testing data. We evaluate the effects of various state-of-art DNN methods using the measurements given above by repeatedly running 20 times using 5-fold cross validation, and report the average results.

We set the parameter value of window size and max stride threshold MS to be 6000 and 500 for skewness-driven dynamic data augmentation respectively. Then we set the parameter value of K and N split to be 3 and 300 for training K-margin-based RCR-net using augmented data (see Section 3.2), and identifying the final predicting label of each ECG record by the trained model. Methods are implemented using Python 3.6.2 on TensorFlow version r1.4.

4.2 Results

Comparing with Other Methods

First of all, we compare our method with the following deep neural network methods:

• RNN is one of the most popular DNN architecture, which allows it to exhibit temporal dynamic behavior. It has been used as main classifier for detecting AF in [Warrick and Homsi, 2018].
• CNN use a variation of multilayer perceptrons to automatically extract features. Recently, [Sodmann et al., 2018] constructed CNN architecture for AF detection without manual extracted features.
• ResNet allows training of very deep networks by constructing the network through modules called residual models, which can avoid the higher training error caused by naively adding the layers [He et al., 2016]. In [Hannun et al., 2019], they have achieved accuracies comparable to or higher than cardiologists for classifying 14 different types of arrhythmia by employing the ResNet as the main classifier.

Results are shown in Figure 5. The left part shows precision, recall and F1NAOP results. We can see that the proposed K-margin-based RCR-net method with 0.8125 F1NAOP score outperforms all state-of-the-art deep learning methods for AF detection task by 6.8%. The method of AlexNet and VGG are not effective, because it is difficult to construct very deep networks which will cause information loss and feature loss. The ResNet method works poorly because feature mapping simply aggregates all the automatic extracted local characters without considering any context information of overall long-term trend, which is constrained by the expressiveness of the model.

From the view of generalization ability of predictor (Hamming Loss), our K-margin-based RCR-net also performs better than others in terms of 38% - 50% fraction of labels that are incorrectly predicted, as shown in Figure 5 (right).

Besides, Table 1 also demonstrates our F1 score and nine DNN methods employed in the Challenge 2017. We can see that automatically focusing on the most important segments (with a 6.8% - 21.3% higher F1NAOP score when comparing it with other DNN methods) is an effective way to improve prediction performance (especially on handling noisy segments and identifying Too Noisy To Classify records).

Moreover, Figure 6 shows detailed performance of our method. The left part of Figure 6 shows a confusion matrix of the model predictions on the test set. Often mistakes made by the model are understandable. For example, many records of Other Rhythm are confused with Normal Sinus Rhythm which makes sense given that it is very difficult to distinguish between the normal sinus rhythm and some arrhythmias in the ECG records. Confusing Other Rhythm and Too Noisy To Classify also makes sense, as this can be subtle to detect especially when their ECG morphologies are similar or when noise is present. The right part of Figure 6 shows deep features learnt using the K-margin-based model on the test set. This shows whether the learned deep features effectively capture the latent relationships among the given four classes. Obviously, we can come to the conclusion that the learnt deep features are highly distinguished and representative.

Hyper-parameters Analysis

Now we further evaluating the performance of our method in different hyper-parameter settings.

The effects of threshold K are shown in the left part of Figure 7. We display the accuracy results of our K-margin-based RCR-net under different threshold K in the range of [1, 3] and how the accuracy (Precision, Recall and F1 score) raises dramatically. However, it drops when the K-margin threshold K is larger than 3. The reason is that the accuracy of our K-margin-based RCR-net model depends on both the number of segments engaged in the algorithm and the confidence of their labels. As K determines how many segments of an ECG record are engaged in the algorithm. Therefore, a little bit larger value of threshold K may result in higher accuracy. While the confidence of labelling accuracy decreases when K gets larger, as we select top-K segments from the most confident one to a less confident one. It means that higher K may also result in smaller confidence of labels engaged in the learning and voting phase, and therefore it would decrease the performance of our K-margin-based RCR-net model.

In addition, we evaluate the effects of three main parameters (windows size, maximum stride threshold MS and N split). In the middle part of Figure 7, we display the accuracy results of our K-margin-based RCR-net model under different parameter N split and MS in the range of [50, 1000] and [300, 2000] respectively. We can see that the model performs better by splitting ECG segments into smaller fragments other than larger fragments, as it could provide more fragments for recurrent neural network layer to capture rhythm-level trend features. However, the length of a fragment decreases with the increasing of the number of fragments which would cause the loss of some critical local heartbeat-level features.

Furthermore, as shown in the right part of Figure 7, when the model parameters of window size and MS vary within the range of [3000, 8000] and [300, 2000] respectively, as smaller parameters increase the augmented training data, the K-margin-based RCR-net performs better. However, the performance decreases when using very small ones, as it can only provide a very narrow view of local beat-level features.

5 Conclusion

In this paper, we propose a K-margin-based RCR-net for AF detection. The experimental results demonstrate that the proposed method with 0.8125 F1NAOP score outperforms all state-of-the-art deep learning methods for AF detection task by 6.8%. A possibly rewarding avenue of future research is to consider multi-modality data input and more fine-grained output categories to improve the model performance and apply it in a more practical situation.