EGCN: An Ensemble-based Learning Framework for Exploring Effective Skeleton-based Rehabilitation Exercise Assessment

1 Introduction

It has been expected to have a growing worldwide burden caused by musculoskeletal disorders [Sebbag et al., 2019]. Physical therapists often conduct therapeutic exercises for different rehabilitation phases of musculoskeletal disorders such as back pain, sprains, and epicondylitis [Wyss and Patel, 2012]. However, regular rehabilitation therapy episodes in hospital settings are often unaffordable for patients due to their inadequate working ability [Machlin et al., 2011]. Accordingly, home- or office-based rehabilitation programs initiated with the supervision of a therapist becomes a costeffective alternative [Jessep et al., 2009]. Keeping exercise regimens in a home-based setting, however, is hard for patients to adhere to, which can even lead to higher healthcare expenditure [Bassett and Prapavessis, 2007]. In recent years, some home-based physical therapy systems that utilize the

∗Corresponding Author

3D skeleton data collected by motion sensors have been attempted to evaluate the quality or correctness of exercises performed by rehabilitation patients [Komatireddy et al., 2014; Saraee et al., 2017]. Such exercise assessment systems could provide patient-identified barriers for increasing adherence to home-based rehabilitation exercises, which works as a professional therapist to motivate patients to do the therapeutic exercises [Karmali et al., 2014].

Motion sensors such as Kinect and motion capture are used to collect skeleton data in existing solutions [Ahad et al., 2019]. The 3D skeleton data is a sequence of skeleton joints featured with 3D position and 3D orientation (i.e., the angle of a skeleton joint) as shown in Figure 1. By analyzing the joint movement patterns thereof, abnormalities in exercises can then be detected [Lei et al., 2019]. This tasks is a more challenging task, especially comparing with the action recognition task.Specifically, different rehabilitation exercises such as "deep squat" and "sit to stand" can be easily recognized, but evaluating the correctness or quality of a single recognized exercise is a more fine-grained task.

In addition to the challenges, exercise assessment in the 3D skeleton data remains not well tackled. First, existing methods [Williams et al., 2019; Liao et al., 2020a; Bruce et al., 2021b] did not make good use of the position and orientation features together as they mainly rely on single modal skeleton feature. Second, existing evaluation standards used in [Williams et al., 2019; Liao et al., 2020a] impede the exploration of effective algorithms as they do not rely on the prediction results and can be hardly further improved based on the results in [Bruce et al., 2021b].

We observe that the position and orientation streams of skeleton data (see Figure 1) are structurally homogeneous but they are heterogeneous in terms of their physical meanings. Specifically, the position feature represents the global structural movement of an exercise. While the orientation feature describes the local characteristic of skeleton joints, which is relatively more independent from one to another. These two skeleton features are not mutually convertible. To explore effective algorithms for skeleton-based exercise assessment with both the position and orientation features, we propose a learning framework called Ensemble-based Graph Convolutional Network (EGCN) that contains various ensemble strategies. Our main contributions are as follows:

• As far as we know, we are the first to explore the ef-

• fects of position and orientation features of skeleton data for rehabilitation exercise assessment. Our EGCN explores ensemble strategies at various levels including data level, feature level, decision level, and model level.
• We provide in-depth analysis regarding the properness of existing evaluation criteria [Liao et al., 2020a; Williams et al., 2019] for skeleton-based exercise assessment methods. Based on the analysis, we adopt cross-validation to validate the effectiveness of methods regarding their prediction abilities.
• We conduct extensive experiments on two latest public datasets: UI-PRMD [Vakanski et al., 2018] and KIMORE [Capecci et al., 2019], the proposed model-level ensemble in our EGCN significantly outperforms not only other ensemble strategies but also state-of-the-art GCN-based single modal methods.

2 RelatedWork

2.1 Skeleton-based Exercise Assessment

We briefly review the related work of skeleton-based exercise assessment in two perspectives: datasets and algorithms. Some exercise assessment datasets have been briefly reviewed in [Ahad et al., 2019], where only the UI-PRMD [Vakanski et al., 2018] dataset is relevant to this study since the other datasets are either focusing on action classification or not skeleton-based. Comparing with UI-PRMD, KIMORE is collected with real patient subjects and it also provides clinical evaluation. According to the surveyed datasets in [Liao et al., 2020b], we use UI-PRMD and KIMORE for our experiments as they are the two latest public datasets for skeletonbased exercise assessment.

In terms of algorithms, they can be roughly grouped to regression- and non-regression-based methods. For regression-based methods, early works using various Hidden Markov Model (HMM) models were compared in [Tao et al., 2016]. [Elkholy et al., 2019] proposed a similar HMMbased method that has less computational overhead than [Tao et al., 2016]. Recently, a deep learning framework [Liao et al., 2020a] was proposed to encode the skeleton data of the UI-PRMD dataset, which is supervised by a quality score function. The training process of [Elkholy et al., 2019] is supervised by the score of abnormality degree (on the scale of 1 to 5) evaluated by a professional specialist. Unlike the regression-based methods that require the supervision of clinical scores or a score function, some non-regression-based methods [Bruce et al., 2020; Bruce et al., 2021b] were proposed to deliver a numerical exercise evaluation score by utilizing the probability results of the SoftMax classifier or transforming outputs before the SoftMax layer via a sigmoid function. We follow the works of non-regression-based methods that treat the problem as abnormality prediction by considering the effects of different skeleton features.

2.2 Ensemble Learning

Ensemble learning is a hot research topic that aims to integrate data fusion, data modeling, and data mining into a unified framework [Dong et al., 2020]. Typical ensemble methods usually achieve better performance through a proper combination mechanism. Otherwise, simply combining ensemble members might jeopardize the overall performance. For the classification task, [Dong et al., 2020] categorized ensemble methods to data-level, feature-level, decision-level, and model-level. These different levels are also known as data fusion strategies [Du and Swamy, 2019]. Motivations behind ensemble learning can be forcing the diversity or independence of submodels, focusing on local information, and aggregation mechanism [Sagi and Rokach, 2018]. In the regime of deep learning, ensemble methods have seldom been surveyed although there are many ensemble-based methods being proposed. With deep learning, typical ensemble methods usually combine or fuse the feature-level representations of different data streams, or aggregate their results at the decision level [Baltruˇsaitis et al., 2019]. Meanwhile, it is also possible to follow the traditional motivation of ensemble methods by forcing the feature-level diversity or independence of small classifiers [Ross et al., 2020]. Otherwise, proper learning methods that fuse different data streams at the model level need to be proposed based on the comprehensive understanding of the data characteristics, which is also known as modalbased fusion [Bruce et al., 2021a] or co-learning [Baltruˇsaitis et al., 2019]. In our proposed learning framework, we design a model-level fusion method with an effective training strategy and compare it with various general ensemble strategies proposed in our EGCN.

3 ProposedMethod

In this section, we introduce our skeleton-based exercise assessment method. We first introduce the GCN model adopted to extract features from the position and orientation streams of the skeleton data. Then, we introduce various ensemble strategies in our EGCN.

3.1 Data Structure and Notation

With N data samples, the exercise repetitions of a dataset can be represented as S = {S(n) |n = 1, . . . ,N }. An exercise repetition S(n) that begins at time t = 1 and ends at time T with skeleton frames collected at regular intervals can, therefore, be represented as a set of T × J skeleton joints S(n) = {S(n) ti | t = 1, ...T, i = 1, . . . , J}, where J is the total number of skeleton joints. Here, a skeleton joint S(n) ti = (P (n) ti ,O(n) ti ) has two groups of features include the position feature P (n) ti and the orientation feature O(n) ti . The position feature P (n) ti = (x, y, z) has 3 attributes featured the 3D cartesian coordinates of the position. The orientation feature O(n) ti = (X, Y,Z) also has 3 attributes that could be transformed to pitch, roll, and yaw values of the joint.

3.2 Representing the Skeleton Data

The skeleton frame is streamed as an ordered list of skeleton joints that has the position and orientation attributes. A complete exercise repetition contains varied lengths of such skeleton frames. We adopt a graph to represent the spatially and temporally structured information among these joints. The structure and the traversal rules of the graph follows the works of 2T-GCN [Bruce et al., 2021b]. The skeleton graph at time t could be represented as ϑt = {υt, εt}, where υt = {υti|υti = S(n) ti , i = 1, . . . , J} denotes the graph vertexes containing all the skeleton joints. While εt denotes the spatial edges representing the skeleton bones. The attributes of each graph vertex are featured with the position and orientation streams of the corresponding skeleton joint.

To perform convolutional operations, the convolutional sampling area of a graph vertex υti is defined as a neighbor set N (υti). Specifically, the strategy empirically uses 3 spatial subsets: the vertex itself, the centripetal subset that contains the neighboring vertexes being closer to the center of gravity, and the centrifugal subset that contains the neighboring vertexes being farther from the gravity center. Assuming there are a fixed number of K subsets in N (υti), these subsets will be numerically indexed with a mapping lti : N (υti) → {0, . . . ,K -1}. For a graph vertex υti, the convolutional operation on the spatial dimension could be calculated as

where υtj is one graph vertex of the defined neighbor set, fin(υtj) is a mapping getting the attribute vector of υtj, W(l(υtj)) is a weight function W (υti, vtj) : N(υti) → Rc that could be implemented with a tensor of (c,K) dimensions. Here, c is the number of feature attributes. Zti (υtj) = |{υtk|lti (υtk) = lti (υtj)}| equals to the cardinality of the corresponding subset, which performs as a normalization term.

For a spatial skeleton frame, the spatial convolutional layer could be implemented by an adjacency matrix A with its elements indicating if a vertex υtj belongs to a subset of N(υti). The graph convolution is implemented by performing a 1×1 classical 2D convolution and multiplies the output tensor with a normalized adjacency matrix Λ- 1 2AΛ- 12 on the second dimension, where Λii = ∑j (Aij ) + α is a diagonal matrix with α set to 0.001 to avoid empty rows. With K sampling strategies ∑Kk=1 Ak, Equation 1 could be transformed as

where Wk is a weight tensor of the 1 × 1 convolutional operation with (Cin, Cout, 1, 1) dimensions, which represents the weighting function of Equation 1. Mk is an attention map with the same size of Ak, which indicates the importance of each vertex. ⊙ denotes the element-wise product between two matrixes.

For the temporal dimension, the convolutional operation is the same as 2T-GCN [Bruce et al., 2021b], i.e., performing a 1 × Γ convolution on the feature map fout, where Γ is the temporal kernel size. Both the spatial and temporal graph convolutional layers are followed by a batch normalization layer and a ReLU layer. A basic GCN block is the combination of a spatial convolution layer, a temporal convolution layer, and an additional dropout layer to avoid overfitting.

Our GCN model is a stack of 9 basic GCN blocks. The first three blocks have 64 output channels. The middle three blocks have 128 output channels. And the last three blocks have 256 output channels. The temporal kernel size Γ is set to 9. To stabilize the training, the residual mechanism is applied to each GCN block. The strides of the 4th and the 7th blocks are set to 2, while all the other blocks use a stride size of 1. A global average pooling layer is added at the last GCN block to pool the GCN feature map to a 256-dimensional feature vector. The last layer of the GCN model is a 1 × 1 2D convolutional layer, transforming the feature vector to our desired outputs (i.e., correct or incorrect).

3.3 Ensemble-based Learning Framework

As shown in Figure 1, we use the defined GCN model to extract features from the different skeleton input streams. In the middle of the two separate learning pipelines, different fusion strategies could be performed. We use g(P (n) , θg)

and h(O(n) , θh), where θg and θh are learnable parameters, to denote the GCN submodels for learning features from the skeleton position and orientation streams, respectively. The goal is to improve the abnormality prediction performance of our EGCN with an effective ensemble strategy. As surveyed in [Baltruˇsaitis et al., 2019], fusion methods for ensemble learning at the data level, feature level, and decision level are commonly adopted for ensemble-based methods. Otherwise, special data fusion design needs to be performed. In the following, we introduce four groups of ensemble-based methods proposed in our EGCN.

Data-Level Ensemble. The data-level ensemble method is also known as Sample-level Ensemble (SLE). Our SLE follows [Bruce et al., 2021b] that concatenated the position and orientation streams at the data level and feed them to a single GCN model, which could be represented as

where GAP is the global average pooling layer, Conv is the fully connected convolutional layer, and σ is the SoftMax classifier.

Feature-Level Ensemble. We investigate two FeatureLevel Ensemble (FLE) strategies. The first one, FLE-1, separately extracts features from different skeleton input streams and concatenate the extracted features at the feature level. The whole model could be optimized with an end-to-end learning process. FLE-1 can be formulated as

where Cat is the concatenation operation.

The second FLE, FLE-2, is based on FLE-1 by forcing the feature-level diversity of FLE-1. Since forcing the diversity of small classifiers is one of the main motivations for ensemble-based methods, we follow the Local Independence Training method [Ross et al., 2020] that was implemented by penalizing the cosine similarity between the features to approximate the feature-level diversity. The loss objective of Cosine Independence Error (ECI ) could be formulated as

To incorporate ECI to our EGCN, we use the ensemble strategy defined in Equation 4 and optimize the ECI together with the cross-entropy loss of FLE-1. The whole FLE-2 model is trained end-to-end.

Decision-Level Ensemble. Decision-Level Ensemble (DLE) could have different training strategies for optimizing the whole learning framework. We investigate two DLE strategies: DLE-1 and DLE-2. For DLE-1, we aggregate the prediction results at the decision level and train the submodels together with an end-to-end learning process, which could be written as

For DLE-2, the submodels are separately trained and then their prediction results are aggregated, which could be represented as

Model-Level Ensemble. Given the intuition that the position and orientation features respectively represent the global and local characteristics of an exercise, we utilize the neuron activation values of g that represents the position stream as the spatial and temporal importance of skeleton joints to regulate the training of the orientation stream. This is also inspired by existing model-based fusion methods [Baradel et al., 2018; Si et al., 2019; Bruce et al., 2021a] that utilize an attention mechanism by averaging the neuron activation values along specific dimensions for the action recognition task. We do not follow this operation as it tends to smooth out the joint importance if we average the neuron activation values along the spatial or temporal dimensions of the GCN feature map. In our Model-Level Ensemble (MLE), we fuse the joint importance derived from g, which is a Cout×Tout×Jout tensor, with the model representation of skeleton orientation stream h(O(n) , θh) by element-wise multiplication along their three dimensions. The MLE model can be written as

where g(P (n) , θg) is pretrained with the action recognition task that classifies different exercises of a whole dataset. During the training process of h(O(n) , θh), the pre-trained parameters θg of g(P (n) , θg) is fixed to maintain the mutual independence of submodels g and h.

4 Experiments

4.1 Datasets

UI-PRMD. The UI-PRMD dataset [Vakanski et al., 2018] is a popular dataset for exercise assessment, which consists of skeletal data collected from 10 healthy subjects with every subject performing 10 repetitions of 10 rehabilitation exercises (E1-10) like "deep squat", "hurdle step", and "sit to stand". The subjects perform every exercise in both correct and incorrect manners. The incorrect manner is simulating the performance of patients with musculoskeletal constraints. Two sensors namely Kinect v2 and Vicon motion capture are utilized to collect the dataset. Both sensors provide positions (i.e., 3D Cartesian coordinates) and orientation features of skeleton joints. According to the results in [Bruce et al., 2021b], Kienct v2 turns out to work better than Vicon motion capture. Hence, we use the data of Kinect v2 for experiments. As the dataset contains inconsistent samples caused by measurement errors and performing with incorrect limbs, we use the consistent version provided by [Liao et al., 2020a], which has 1, 326 exercise repetitions in total.

KIMORE. Another rehabilitation dataset called KIMORE [Capecci et al., 2019] is collected with Kinect v2 from 78 subjects that are categorized to three groups including Control Group Expert (CG-E), Control Group Non-Expert (CG-NE) and Group with Pain and Postural disorders (GPP). The GPP

group has 34 subjects that have different motor dysfunctions like stroke, Parkinson's disease and back pain. All subjects perform 5 exercises (Es1-5) like "lifting of the arms", "lateral tilt of the trunk with the arms in extension", "trunk rotation", "pelvis rotations on the transverse plane" and "squatting". It is verified in [Capecci et al., 2019] that the clinical total score distribution of groups CG-E and GPP are without overlapping, which means we could treat their exercise repetitions as correct and incorrect. Hence, to perform abnormality prediction based on each exercise repetition, we manually segment the datasetand group the repetitions of 17 experts and 34 patients as normal and abnormal, respectively.

4.2 EvaluationMetrics

In existing methods, Distance Metric (DM) and Separation Degree (SD) respectively defined in [Williams et al., 2019] and [Liao et al., 2020a] have been used to evaluate the representation ability of a model. On the one hand, DM and SD quantify the difference between the correct and incorrect evaluation results but cannot reflect the model's prediction ability. [Liao et al., 2020a] attempted to show the prediction ability of their method by dividing the data into training set and validation set, but only reported the results for the E1 of UI-PRMD. On the other hand, the results of SD in [Bruce et al., 2020] reached 0.808 (derived from the training accuracy of 99.59%) for UI-PRMD by using the orientation feature. [Bruce et al., 2021b] achieved an even higher SD of 0.933 for UI-PRMD by using the SoftMax to calculate the exercise evaluation score. The main difference of [Bruce et al., 2020] and [Bruce et al., 2021b] is the way for calculating the exercise evaluation score, where the former used the sigmoid function while the latter used SoftMax. This observation indicates SD can be irrelevant to a model's representation ability as it can be improved by just changing the calculation method of exercise evaluation score.

For DM, given two positive sequences x = (x1, . . . , xN) and y = (y1, . . . , yN), the DM could be calculated as

To analyze the properness of DM, we implement the methods in [Bruce et al., 2020] and [Bruce et al., 2021b]. Table 1 gives the DM results of existing methods, which shows a similar observation with SD. Hence, we do not continue to use the evaluation criteria DM and SD.

Given that both KIMORE and UI-PRMD are relatively small datasets, we extend the attempt of [Liao et al., 2020a]

by following the 5-fold cross-validation criterion applied in [Bruce et al., 2021b] to test the prediction ability of different ensemble strategies in our EGCN.

4.3 Implementation Details

For the cross-validation, we split both UI-PRMD and KIMORE based on the subject ID to five folds. The proposed MLE strategy requires to pretrain the GCN model g(P (n) , θg) with the position feature. The pretrained model could then be utilized to retrieve the joint importance from the position feature for model-level data fusion with the orientation feature. To do so, we use the position feature to pretrain a GCN model with the action classification task. In our implementation, we involve all the exercise classes of a dataset. The overall action classification accuracy for UI-PRMD and KIMORE are 96.91% and 98.04%, respectively (please refer to the supplementary for details). Like single modal methods, all the ensemble strategies proposed in EGCN are optimized with the cross-entropy loss using stochastic gradient descent with a base learning rate of 0.01. By training 50 epochs in total, we decay the learning rate by 0.1 at epochs 10 and 30. All experiments are conducted on a workstation with 2 GTX 1080 GPUs.

4.4 Comparison of Different Ensemble Strategies

Table 2 showes the correctness prediction results of all exercises in UI-PRMD and KIMORE. General ensemble methods such as SLE, FLEs and DLEs could not effectively take advantage of multiple skeleton input streams. Although ensemble strategies of SLE, FLE-2, and DLE-2 can gain some improvements when compare with single modal methods, the improvements cannot generalize well to different exercises. While FLE-1 and DLE-1 could not perform well as they just simply combine the features. In contrast, our MLE method achieves greatly better performance than both single modal and other ensemble methods for almost all exercises of UIPRMD and KIMORE datasets, which validates the effectiveness of our model design.

4.5 Ablations &Comparison with State-of-the-art

Ablations. Our MLE can be implemented with different training strategies. To validate the results of MLE in Table 2, we conduct ablation study with other five different implementations of our MLE. Their results are shown in Table 3. "Self Importance" means using the joint importance derived from h to replace that from g by fixing one θh and updating another θh. "Swap h and g" means calculating the joint importance from h and updating the θg . "No Pretraining" means optimize h and g together without pretraining g. While "Tune θg" and "Fix θg" respectively indicate optimizing θh with and without updating θg of the pretrained g. The results consistently show that "Fix θg" is the effective setting for our MLE. This verifies that the learned joint importance is effective to facilitate the exercise evaluation of the orientation stream. While tuning the θg will affect its physical meaning (i.e., joint importance).

Besides, we also compare our model design with averaging the joint importance along the Cout dimension (see Figure 2[right]), which is similar with [Baradel et al., 2018; Si et al., 2019]. The results of "Fix θg, Mean Along Cout" in Table 3 indicate that it is not as effective as "Fix θg", which might be caused by the smoothing out effect of channel-level importance. We can observe this from Figure 2(left), where the joint importance layers are actually changing along the Cout dimension. While the averaged joint importance in Figure 2(right) is less informative than the original one.

Comparison with State-of-the-art. We first compare with state-of-the-art GCN models such as AGCN [Shi et al., 2019] or MS-G3D [Liu et al., 2020] via single modal setting. Table 4 shows the prediction results of these baselines implemented with single modal settings (i.e., use either position or orientation). We also use these GCN baselines as the backbone of our EGCN to further explore different ensemble strategies proposed in our EGCN on the two datasets (see Table 4). We can observe that changing the backbone with other advanced GCN baselines does not lead to stable improvements. For example, MS-G3D can improve the single modal setting of orientation feature for UI-PRMD, but it does improve the performance of other single modal settings on two datasets. This might be due to the fact that these GCN baselines are originally designed for the action recognition task rather than for exercise evaluation. With our model-level fusion design, the MLE method using the basic GCN achieves better performance than the SLE using MS-G3D on both the UI-PRMD and KIMORE datasets. This further verifies our intuition that the joint importance learned from the position stream can regularize the training of the orientation stream.

5 Conclusion

In this paper, we propose the EGCN framework with various ensemble strategies for the exploration of effective skeletonbased exercise assessment. This is the first work that uses both position and orientation skeleton features for the task. With extensive experiments on two latest datasets: UI-PRMD and KIMORE, the proposed MLE outperforms other ensemble strategies in terms of the prediction accuracy. Meanwhile, we further verify the proper design of our MLE with ablations of different training strategies and other baselines. In the future, we will investigate if our EGCN can generate evaluation scores that are consistent with human evaluation and design online exercise evaluation methods.

Acknowledgements

This work is supported by The Hong Kong Polytechnic University for the project: Artificial Intelligence and Robotics (AIR) Group: Artifical Intelligence Art under grant No.: P0033738.