Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation

1 Introduction

Hand gestures are a flexible and intuitive means of communication for human beings. With the advancement of wearable sensors and machine learning, gesture recognition has become quite popular for communication, smart home appliances, interactive entertainment, etc. [Rafiq et al., 2023; Guo et al., 2021]. Gesture-based interactions with wearables depend on specific presumptions about users' motor abilities. As a consequence, people with motor impairments face challenges in performing gestures with wearables that are widely adopted for the general public [Siean and Vatavu, 2021]. The severity of motor impairments leads to a different pattern of motion gestures and creates individual differences among the users [Vatavu and Ungurean, 2022]. It is social discrimination for this underrepresented population as it deprives them of completely leveraging those wearable devices. As the United Nations Sustainable Development Principle is Leave no one behind, increasing independence and including people with disabilities aid in achieving UN Sustainable Development Goals Good health and well-being and Reduce inequalities [Yu et al., 2023].

To tackle the problem, a large-scale labeled dataset is expected to build a robust hand gesture recognition method. However, this is impractical and cumbersome for motorimpaired individuals to participate in vast data collection. The transfer learning approach has been used to solve the problem. In transfer learning, a model is trained on a source domain and then fine-tuned to a target domain by transferring knowledge from the prior learned task [Zhuang et al., 2020]. But fine-tuning shows worse performance in a target domain with out-of-distribution samples [Kumar et al., 2022]. Therefore, applying transfer learning alone cannot solve the problem, as the gesture data from the motorimpaired population are more variable and noisy than the control population (Figure 1), and limited data samples might not help the deep learning model capture the diverse patterns among each individual. To utilize limited training data, a unique approach has been proposed [Finn et al., 2017] and in our case, few-shot transfer learning is an applicable solution.

In our case, another real-world problem is that all the gesture classes may not be available initially. For example, standard pre-defined gestures can be difficult to perform for individuals lacking fine motor skills. New unseen gestures may become accessible incrementally if motorimpaired individuals want to input their flexible and custom gestures. This context is referred to as a continual learning setting as the model involves learning a disjoint set of classes incrementally [Parisi et al., 2019]. Continual learning posts two challenges, namely catastrophic forgetting [French, 1999] when the model's performance drops drastically on old classes and overfitting when the model is not capable of learning generalized features with a few training examples [Gidaris and Komodakis, 2018].

Many continual learning approaches including parameter regularization, functional regularization, replay strategy, etc. have become popular at present [Van de Ven and Tolias, 2019]. In this paper, we propose a novel method called Latent Embedding Exploitation (LEE) in our replay-based few-shot continual learning framework that can learn gesture classes incrementally from motor-impaired people. Specifically, in our framework, we utilize three latent embeddings from the feature extractor of a pre-trained model which is trained on the control subjects' gesture samples. The three embeddings are:

• a preserved latent embedding works as gesture prior knowledge,
• two additional latent embeddings known as temporary and learned embedding maintain a intra-gesture divergence.

They jointly aid a pre-trained model to be fine-tuned effectively with a few training examples from a motorimpaired individual. Ideally, the goal of LEE is to navigate the learned feature space toward a rich and diversified feature representation for variable and noisy data. Thus the finetuned model can capture the diverse pattern of unseen gesture classes with a few training examples. As a result, motorimpaired people can take full advantage of wearable devices with our proposed method. The major contributions of this paper are as follows:

• We explore wearable sensor-based hand gestures from the underrepresented population. In addition, we introduce the LEE mechanism in our replay-based fewshot continual learning framework that formulates the diverse gesture samples into a heterogeneous feature representation. Hence, the pre-trained model can be
• fine-tuned competently with a few training samples for each unseen class in a continual learning setup.
• We utilize two publicly available gesture datasets to demonstrate the performance of the proposed method. Our proposed method achieves competitive performance compared to existing methods.
• We experimentally show how latent embeddings can be leveraged to improve the performance of fine-tuning in the limited data of shifted distribution.

2 RelatedWork

A wide range of hand gesture recognition techniques has been explored by utilizing images and videos [Hu et al., 2018; Zhou et al., 2021], electromyography (EMG) [Caramiaux et al., 2015] and wearable-sensors [Laput and Harrison, 2019; Kunwar et al., 2022]. Among these techniques, visionbased approaches show poor performance due to complex backgrounds, varying light conditions, and the presence of another person in the background [Pisharady and Saerbeck, 2015; Mohamed et al., 2021]. Moreover, high computational power is required to analyze high-quality video sequences and individuals might not be comfortable sharing live video streams due to privacy. On the contrary, sensors including accelerometers and gyroscopes are low-cost and widely available in current wearable devices. Therefore, in our work, we utilize wearable sensor-based motion data to solve the problem.

Prior work has been done using hand-crafted features such as mean, variance, median, maximum, minimum, etc. for classifying the hand gestures [Xie and Cao, 2016]. However, domain expertise is needed to prepare the necessary hand-crafted features which is time-consuming and the methods that utilize those features show poor performance in practice. On the contrary, deep learning approaches such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Transformer architectures have demonstrated significant performance in hand gesture classification with automatically extracted features from training data examples [Kunwar et al., 2022; Nguyen-Trong et al., 2021; Li et al., 2019].

The goal of few-shot continual learning is to train new classes incrementally with few data instances. In order to tackle few-shot learning problem, metric-learning [Kaya and Bilge, 2019] , meta-learning [Finn et al., 2017] and multitask learning [Zhang and Yang, 2021] have been proposed. In the field of hand gesture recognition, camera data [Wu et al., 2012; Stewart et al., 2020] and EMG signals [Rahimian et al., 2021] have been used for few-shot learning. However, Xu et al. [2022] proposed a hand gesture customization framework that can learn novel hand gesture classes incrementally with a few training examples. Kimura [2022] also proposed a self-supervised method for few-shot hand gesture recognition using wearable sensor data. However, they used only control participants' data. In addition, these methods would not work for motor-impaired individuals as the data instances are varying and noisy. Although Malu et al. [2018] and Kim et al. [2019] explored the smartwatch interactions for people with upper body motor impairments, they experimented with the touch gestures only. The primary difference between our approach and existing works is that they did not consider the out-of-distribution samples for motion gestures. Directly fine-tuning a pre-trained model may cause a sudden performance decay. The objective of our method is to adapt a model that generates an enhanced feature representation via gesture prior knowledge exploitation and intra-gesture divergence exploration to incrementally learn novel gestures with few training examples from motor-impaired individuals.

3 ProposedMethod

3.1 Problem Statement

A domain is defined as a joint probability distribution Px,y on X × Y, where X and Y denote the instance space and label space, respectively [Ding and Fu, 2017; Qian et al., 2021]. In our setting, we have two domains including source domain, Ds = {(xi, yi)}ns i=1 and target domain, Dt = {(xi, yi)}nt i=1 where nt << ns. Each sample, x ∈ RL×3 denotes a signal of L length with three-axis motion values collected from wearable sensors at each timestamp. The two domains have the same feature space (X s = X t ) but different label spaces (Y s ̸= Y t ). In addition to it, they have different probability distributions i.e. P s (xi, yi) ̸= P t (xi, yi). The data distribution for Dt is harder to learn than Ds due to highlevel noise e.g. H[P t (x)] >> H[P s (x)] where H denotes the entropy. Therefore, any statistical learner needs more samples from Dt to converge to achieve the same level of accuracy as models trained on Ds . In Dt , the classes C1, C2, ..., Cn are incrementally accessible at the training time. While accessing a new class, Ci, a memory buffer stores training examples from old classes such as C1, C2, ....., Ci-1. Motorimpaired individuals may face challenges in providing many consistent data samples. As a result, it becomes more difficult to train the model with such limited samples. The goal is to build a pre-trained model in the source domain f : X s → Y s that can be fine-tuned to learn currently available classes from the target domain with few training examples. We design the method in such a way that should reduce the memory usage compared to traditional replay buffers.

3.2 Overall Framework

Sensor readings from motor-impaired individuals vary substantially due to factors such as health conditions, severity of disability, movement patterns of arms, etc. (Figure 1b). Therefore, it is critical for a pre-trained model to adapt to such data samples from sensor readings. Machine learning models without considering out-of-distribution data often result in large performance degradation. For example, a model trained on the data from control participants often fails to capture unique patterns and performs poorly on specific populations such as Parkinson's patients [Bin Rafiq et al., 2020]. It is difficult to collect a large volume of diverse labeled data frommotor-impaired individuals. Moreover, individuals may need their own, custom flexible gestures from time to time in human-computer interaction and social communication.

In this paper, we propose a novel technique where the model learns new gesture classes incrementally by utilizing multifunctional latent embeddings. Our method considers three latent embeddings instead of a single representation compared to Autoencoders [Bank et al., 2020]. As shown in Figure 2, a latent embedding from the control population is preserved by leveraging the feature extractor (deep encoder) of a pre-trained model. This preserved latent embedding works as gesture prior knowledge to assist the model to incrementally learn unseen out-of-distribution gesture samples and prevent overfitting. Two additional identical feature extractors are utilized to produce two latent embeddings with available gesture classes from the motorimpaired subjects, and one of them is being updated during training. As a consequence, the learned latent embedding has a strong capability in classifying highly variable data during inference. The total loss of the model in weighted summation is as follows:

where Lci is the loss of discrimination between preserved and learned embedding, Lii is the loss of discrimination between temporary and learned embedding and Lcls is the classification loss. α and β are trade-off hyper-parameters where α + β = 1. While minimizing the loss in the training stage, the model exhibits complementary learning behavior by adaptively adjusting its focus between exploitative and explorative representation learning.

3.3 Complementary Learning Paradigm

The complementary learning system plays an important role in the human brain where the hippocampus and the neocortex function in a complementary manner to learn complex behavior [Perrusqu´ıa, 2022; Blakeman and Mareschal, 2020]. Our learning strategy for new classes is inspired by this complementary learning paradigm. The representation space holds the same classes closer under the effect of classification objective while training a deep learning model. However, the model struggles to learn a robust latent space with fewer training examples from out-of-distribution data [Yang et al., 2021]. Therefore, we model the representation space generation process by utilizing both the classification and the embedding discrimination objectives. In our method, we introduce three multipurpose latent embeddings including preserved control embedding, temporary sample embedding, and learned embedding. The preserved control embedding contains the representation space of the expected pattern of gestures from the source domain. The learned embedding constructs a latent statistical structure with the help of preserved and temporary embedding. Thus, the constructed latent space sufficiently narrows the features of the sameclass data with limited training examples.

Latent Embedding Exploitation

The learned latent embedding exploits the preserved latent embedding (gesture prior knowledge) to enlarge and diversify the feature space. The network architecture is expected to increase the similarity of the feature space between the preserved and the learned embedding. We utilize the feature extractor, fθ (deep encoder), from the pre-trained model to preserve a latent embedding, zc = fθ(X s ) where X s is the feature space from the source domain i.e. control participants.

The input of our model is denoted as x. In our continual learning module, two additional identical feature extractors, fθ and f c θ draw out temporary latent embedding, zi= fθ(x) and learned latent embedding, zci = f c θ (x) respectively. To this end, we require to expand the similarity between zc and zci as the following loss:

where Sc is the cosine similarity between zc and zci and it can be defined as follows:

Latent Embedding Exploration

Simultaneously, the learned latent embedding aims to maximize the distance from the identical temporary sample embedding throughout the training which is called intragesture divergence. This action works as a way of exploration for a wide feature space. As a result, the learned latent embedding captures tailored and generalizable feature representation to learn novel gesture classes. To minimize the similarity between zi and zci is identical as follows:

Learning Objective

The learning objective of the model is to identify the gesture classes which is a transformation of the input sensor signals to a gesture category. Therefore, we utilize class labels in the final classification layer to guide the learned latent embedding during the training stage. We adopt standard cross-entropy loss for the classification task:

where C represents the number of classes, y is the true gesture label, δc(f c θ (x)) is the predicted probability and δc is the softmax function.

4 Experiments

4.1 Datasets

The SmartWatch Gesture Dataset [Porzi et al., 2013; Costante et al., 2014] was built for interacting with mobile applications using arm gestures. This dataset contains 20 distinct gestures from eight different subjects. A first-generation Sony smartwatch with a built-in 3-axis accelerometer was worn on the user's right wrist while performing 20 repetitions for each gesture. In total, 3200 sequences were collected and each sequence contains 3-axis acceleration data. We use this dataset as our source domain to build the pre-trained model.

The Motion Gesture Dataset [Vatavu and Ungurean, 2022] was built to understand the gesture articulation of people with upper-body motor impairments. Six different motion gestures were collected by a group of 12 people (six male and six female) with upper-body motor impairments, ranging ages from 27 to 65 years. The participants had a wide range of disabilities including Spinal cord injury, Traumatic brain injury, Multiple sclerosis, Parkinson's disease, etc. A Samsung Gear Fit 2 smartwatch was used by the participants to collect the wrist gesture's accelerometer data. Each participant repeated each gesture eight times. We utilize this dataset in our few-shot continual learning setting.

4.2 Implementation Details

The sequence length of the data samples varies extensively. As a result, we apply a linear interpolation technique to our source and target datasets so that the sequence length (L = 50) of each data sample is constant throughout the experiments. While working with the sensor data, outlier features can negatively influence the results. Therefore, dataset standardization is conducted by removing the mean and scaling it according to the interquartile range for each feature. We select 16 out of 20 gestures from the source domain to build a pre-trained model because we do not want any overlapped gestures between the source domain and the target domain. We follow the leave-one-subject-out strategy to pre-train the model. Throughout all experiments, we used the same subjects for a fair comparison. For our architecture, the feature extractor contains one LSTM layer with 64-dimensional hidden representation, one fully connected layer with 14 units, and one dropout layer with a value of 0.5 between them. A fully connected layer is used as the classification layer. The network architecture remains the same throughout all experiments. The Adam optimizer with learning rate 10-3 is used for the few-shot setup. In the few-shot continual learning setting, since each gesture class contains very few training examples, the epoch is set to 15 and the mini-batch contains all examples. We run each experiment 10 times with five different orders of the gesture classes. We report the average accuracy with one, three, and five training examples over all the encountered gesture classes. As this is a continual learning setup, we also report the class-wise macro F1 score and forgetting metric to understand each gesture's performance individually. Our memory buffer stores 60× fewer examples compared to traditional replay buffers [Rebuffi et al., 2017].

4.3 Baselines and ComparedMethods

We compare our methods with different closely related approaches in the few-shot continual learning setting. Since our method utilizes LSTM in the network architecture, we consider comparing it with an LSTM classifier that learns the gesture classes incrementally with a few training examples. Vanilla-Ft, MAML-Ft [Finn et al., 2017], and Prototypical Net-Ft [Snell et al., 2017] involve fine-tuning the pre-trained models on the few-shot classes. We also compare our method with iCaRL [Rebuffi et al., 2017]. We assume that the memory buffer exists in all methods for a fair comparison.

4.4 Experimental Results

We compare the average accuracy of the proposed method with other approaches. Figure 3 shows the test accuracies for three participants including a motor-impaired individual with Spinal cord injury, an individual with Parkinson's disease, and an individual with Multiple sclerosis. In most cases, our LEE method outperforms other techniques. We observe that the iCaRL classifier occasionally shows better accuracy than our method while learning two initial gestures. But for the rest of the incrementally added classes, LEE always performs better than the iCaRL classifier. The accuracy of the iCaRL classifier significantly drops for new gesture classes because it fails to capture the diverse and highly variable pattern of unseen gestures with few training examples. The performance increases with the sample size for all methods. Surprisingly, the fine-tuning approach fails compared to the basic LSTM classifier and our LEE.

In a continual learning setup, it is important to performwell in old classes while trained on a new class. Therefore, in Table 1, we report the class-wise F1 scores after six gestures are trained with five training examples. Our method always provides a higher macro F1 score than other methods. LEE has a 12.3% higher F1 score for all gesture classes. Figure 4 shows performance and forgetting score for Gesture 1 and Gesture 3 after six gestures are trained with five samples. Our LEE method has better performance with less forgetting.

5 Ablation Study

5.1 Loss Hyperparameter Sensitivity Analysis

We report the effect of loss hyperparameter sensitivity. We focus only on α as it complements the other hyperparameter (β) in our continual learning module. We choose α to exploit the gesture prior knowledge, selected from α ∈{0.01, 0.05, 0.5, 0.1, 0.9}. According to Figure 5 (left), LEE provides robust accuracy with a wide range of hyperparameters after learning six gestures using five training examples.

5.2 Number of Participants and Gestures in Source Domain

We experiment to produce the preserved latent embedding with a different number of participants from the source domain. We achieve higher accuracy with one and three samples using seven different participants (Figure 5 middle). Apart from this, the proposed method is invariant to the number of participants from the source domain. However, it is preferable to utilize a large number of control participants in the source domain to capture a more diversified representation space. We also conduct experiments with a different number of gestures from the source domain to generate the preserved latent embedding (Figure 5 right). Though we get the highest accuracy using 16 gestures, we observe that the accuracy does not change for other different numbers of gestures. Therefore, the observation illustrates that our method is robust to the number of gesture classes.

5.3 Significance of Embeddings

We conduct experiments to investigate how the preserved latent embedding and the temporary latent embedding contribute to our proposed method. We apply LEE without one of those embeddings, one at a time, and evaluate the performance. Figure 6 (left) and (middle) show that removing either component of interest results in a less tridiagonal-shaped confusion matrix, indicating a drop in model performance and robustness. We further confirm from Figure 6 (right) that both embeddings jointly contribute to robust performance, and thus these embeddings are the foundations for learning unseen gestures incrementally with limited training examples.

6 Use Cases and Social Impact

Wearable sensor-based gesture recognition is becoming popular in many areas including communication, controlling home appliances, and interactive entertainment. As most research doesn't include the motor-impaired population, those individuals face challenges using wearable devices for gesture recognition and communication. We believe our contribution can be impactful for those who need more than a set of pre-defined gestures. Gesture recognition exists for standard movements such as sign language for speechimpaired people but sign language can be difficult to perform for individuals lacking fine motor skills. Our work is part of a fast and flexible gesture-to-speech recognition system that we are developing in collaboration with Shirley Ryan AbilityLab 1 . The need for such solutions is underscored by the prevalence of motor impairments that also impact speech. 12.1% of the population has a motor disability [CDC, 2023] while 7.6% have a speech disorder [NIDCD, 2024] through Stroke (795, 000 cases each year), Parkinson's disease (1 million), Multiple sclerosis (727, 000), Spinal cord injury (294, 000) and Cerebral palsy (764, 000) [Wallin et al., 2019; White and Black, 2016]. The proposed method has the potential to transform the lives of these individuals by providing a more natural and efficient mode of communication, improving quality of life, enhancing interaction, and reducing the burden on caregivers.

7 Conclusion and FutureWork

Hand gestures are natural and flexible means of communication. Available wearable sensor-based hand gesture solutions are widely adopted for the normal population and these solutions fail to capture highly variable and inconsistent data samples from motor-impaired people. Moreover, in the real world, motor-impaired individuals face challenges in performing predefined gestures, and many valuable use cases rely on acquiring new gestures. However, a substantial amount of data samples is needed to develop a strong hand gesture recognition method. Therefore, we introduce a novel method called Latent Embedding Exploitation (LEE) to learn novel gesture classes incrementally using a few samples from motor-impaired individuals. We experimentally show that our method outperforms the existing baselines. Our method helps motorimpaired persons leverage wearable devices and their unique movement styles can be learned and applied in humancomputer interaction and social communication. By enabling meaningful interactions with motor-impaired individuals and seamlessly integrating wearable devices into their daily lives, we open the gate to collecting invaluable data from this underrepresented group in real-world scenarios. This data collection paradigm can play a central role in facilitating the advancements in other machine learning research, benefiting not only motor-impaired individuals but also contributing to broader technological innovation. In the future, we will integrate our method with a wearable application and online learning will be explored to assist motor-impaired individuals to input their custom, flexible gestures in realtime. Furthermore, we will survey to collect the opinions of the population and tailor our approach accordingly.

1 https://www.sralab.org/

Acknowledgments

Research reported in this publication was supported by the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the National Institutes of Health under Award Number P2CHD101899.