BeyondVision: An EMG-driven Micro Hand Gesture Recognition Based on Dynamic Segmentation

1 Introduction

Hand gesture recognition (HGR) is a longstanding task in machine learning [Mohamed et al., 2021][Guo et al., 2021][Rawat et al., 2023][Wu et al., 2023]. Existing HGR methods are mainly divided into two classes: vision-based methods (e.g., [Shamayleh et al., 2018]) and wearabledevice-based methods (e.g., [Das et al., 2017; Lauss et al., 2022]). Recently, most HGR literature focus on recognizing gestures from images or videos sampled by vision sensors, such as depth cameras [Zengeler et al., 2018] and RGB cameras [Zhu et al., 2023].

However, as shown in Figure 1, the vision-based HGR methods are typically limited in finger occlusion, and are insensitive to micro hand gestures (MHGs) (e.g., pinch within

∗The corresponding author.

1cm, fingers touching each other) [Ling et al., 2020]. In fact, MHGs are even more effective than large-amplitude gestures, in human-computer interactions. First, MHGs reduce user fatigue and are less physically demanding, which is vital in prolonged interactions. Second, MHGs are more natural and intuitive like human daily behaviors, which improves the user experiences.

To address the issues of finger occlusion and MHG insensitivity, combined with the electromyography (EMG) technique [Merletti and Farina, 2016; Eddy et al., 2023] that measures the muscle's electrical activities during movement, we propose BeyondVision, an EMG-driven micro HGR system based on deep learning. Compared with vision-based methods, our EMG-driven method directly captures the underlying muscle activities of MHGs, which is not affected by fingers' view angle and occlusion. Compared with EMG-based HGR methods, traditional literature (e.g., Ninapro [Atzori and et al., 2015]) typically employs the general EMG gesture databases, and focuses on large-amplitude gestures without involving MHG recognition. In contrast, our BeyondVision is a novel method proposed to implement EMG-driven MHG recognition. Although Meta shows a demo video of similar techniques [Facebook, 2021], they have not released any academic literature or completed product.

BeyondVision consists of a wristband-style EMG sample device, and a corresponding lightweight convolutional neural network (CNN) BV-Net. The device samples 16channel EMG signals in 2000Hz, and supports the effective implementation of MHGs. The BV-Net is designed to accurately translate EMG signals to control commands in realtime. Subjective and objective experimental results show that our approach achieves over 95% average recognition rate, 2000Hz sampling frequency, and real-time gesture recognition.

To summarize, our main contributions are three-fold:

• We propose BeyondVision, a novel EMG-driven hand gesture recognition approach that is sensitive to MHGs and without influence by occluded fingers.
• We propose a wristband-style device that samples EMG signals in 2000Hz, and propose a BV-Net that effectively translates EMG signals to control commands in real-time. Moreover, we propose a weight segmentation (WS) algorithm that significantly improves the recognition accuracy.
• Experiments show that BeyondVision can achieve an impressive average accuracy rate of over 95%, and is effective in MHGs recognition, which satisfies the basic requirements of human-computer interaction, and has the potential for further commercial application.

2 RelatedWork

Below we summarize the most related studies that involve three main topics, vision-based HGR, gesture interaction by wearable devices, and EMG-based gesture interaction.

2.1 Vision-based Hand Gesture Recognition

Machine learning techniques have been widely adopted for gesture and action recognition, such as support vector machines [Dardas and Georganas, 2011], artificial neural networks [Singha and et al., 2015], and hidden Markov models [Moni and Ali, 2009].

Recently, supervised learning has been the most commonly used technique that employs labeled data to train the models. The initial experiments involved 2D CNNs [Yu et al., 2021] for extracting spatial features from frames, and incorporating temporal elements through additional optical flow streams or temporal pooling layers. Subsequently, 3D CNNs [Tran et al., 2018; Hara et al., 2018; Ur Rehman et al., 2021], and two-stream models [Zhao et al., 2017; Zhu et al., 2019], were developed to capture both spatial and temporal aspects simultaneously, using three-dimensional filters. These 3D CNNs, compared to the 2D versions, can hierarchically process temporal information throughout the network. RNNs, particularly Long Short-Term Memories (LSTMs) [Tsironi et al., 2017; Obaid et al., 2020] have shown effectiveness in integrating temporal data, especially for sequential data. However, gesture recognition based on vision increasingly relies on more data with higher resolution and more complex algorithm architecture, requiring more computational resources.

Compared with vision-based methods, our EMG-driven method is more effective for recognizing MHGs and gestures with occluded fingers.

2.2 Gesture Interaction byWearable Devices

Some studies focus on using optical sensing to identify micro gesture recognition, such as ultrasonic [Zhang et al., 2018a], infrared [McIntosh and et al., 2017; Yeo et al., 2019], and various sensors [Esposito et al., 2020].

BeamBand [Iravantchi et al., 2019] is a wrist-worn system using ultrasonic beamforming for hand gesture sensing, achieving 94.6% accuracy in recognizing a six-class gesture set at 8 FPS. It employs an array of ultrasonic transducers for acoustic interrogation of the hand's surface geometry. DualRing [Liang et al., 2021], a novel ring-based input device, records hand and finger movements with 94.3% accuracy for 10 gestures, utilizing two inertial measurement unit rings and a circuit to measure thumb and index finger impedance. The Z-Ring [Waghmare et al., 2023], facilitates gesture input and object detection using bio-impedance sensing. However, these methods may encounter difficulties and general wearing is cumbersome, limiting the degree of freedom of fingers, for example when hand occlusion occurs due to holding objects.

As shown in Figure 2, compared with existing methods based on wearable devices, our device is more convenient. Moreover, our method achieves higher recognition accuracy and supports a greater number of gesture classes (as shown in Section 4.2 and Table 3).

2.3 EMG-based Gesture Interaction

EMG traditionally rooted in medical diagnostics, has seen its applications burgeon into a myriad of domains in recent decades. One of the most profound applications of EMG is its integration into Prosthetic Control. As highlighted by Englehart and Hudgins [Englehart and Hudgins, 2003], EMG signals have been pivotal in the real-time control of multifunctional prosthetic devices. These range from traditional machine learning algorithms such as support vector machines [Chen and Zhang, 2019] and hidden Markov models [Wen et al., 2021], to more recent deep learning techniques including CNNs [Zhai et al., 2017] and recurrent neural networks. Atzori et al. [Atzori et al., 2014] undertook a detailed comparison of these classifiers, emphasizing the superior performance of deep learning methodologies.

Following the first commercial EMG device proposed [Rawat et al., 2016], EMG-driven systems become a viable option in human-computer interaction. Subsequently, many studies apply EMG technique to various control scenarios, including prosthetic control [Parajuli et al., 2019], sign language recognition [Khomami and Shamekhi, 2021], gaming interactions [Karolus et al., 2022], and so on. With this framework [Eddy et al., 2023], EMG-based control is gaining increasing attention within the human-computer interaction community.

However, existing EMG-based literature does not involve MHG recognition. Although Meta Reality Lab shows a demo video of similar techniques [Facebook, 2021], they have not released any academic literature or completed product.

3 Method

3.1 Overview

Given a micro hand gesture G, our method BeyondVision is modeled as a function Φ to recognize G and predict a corresponding control command C = Φ(G).

Figure 3 shows the pipeline of our BeyondVision Φ. First, according to a user input MHG G [Figure 3(a)], a wristbandstyle device Φd samples the 16 channels EMG signals S = Φd (G) [Figure 3(b)]. Then, as shown in Figure 3(c), in a designed pre-processing mechanism Φpr , we eliminate the invalid signals while enhancing the valid signals, and output an optimized signal S′=Φpr (S). Next, as shown in Figure 3(d), in a designed WS algorithm, we segment EMG spectrum to a fragment sequence {S′n }(n=1, 2, ..., n), and predict two sequences of gesture classes {Cn} and probabilities {Pn} by a designed lightweight BV-Net Φn [Figure 3(e)], formulated as Cn, Pn=Φn (S′n ). Finally, utilizing a weight-based gesture evaluation [Figure 3(f)], we obtain the final control command C [Figure 3(g)].

We will detail the EMG sampling device Φd , the preprocessing mechanism Φpr , the prediction CNN Φn , and the WS algorithm in Section 3.2, Section 3.3, Section 3.4, and Section 3.5 respectively.

3.2 EMG Sampling Device

As shown in Figure 4, in BeyondVison, the EMG sampling device Φd is designed to collect 16 channels of EMG signals S, according to an input user MHG G.

For the device Φd , we employ two ADS1299 chips [Gao, 2023] to sample 16-channel EMG signals, and each chip manages 8 channels with a frequency of 2000Hz and a resolution of 24 bits. As shown in Figure 4(c), in Φd , the 16 channels correspond to a series of muscle tissues, including 14 muscles and 2 nerves[Liu et al., 1997]. The meaning of each abbreviation is shown in Table ??.

3.3 Pre-processingMechanism

In the pre-processing stage Φpr , following the sampled EMG signals S, our goal is to eliminate the noise and enhance the valid signals, and finally output the optimized signal S′=Φpr (S).

First, as shown in Figure 5 middle, we eliminate the noise in S by a 10Hz high-pass filter. Then, as shown in Figure 5 bottom, S′ is computed by a root mean square (RMG) algorithm, defined as

where n=16 and S0 to Sn is corresponding to the 16 channels EMG signals.

3.4 Network Architecture

We propose a BV-Net Φn to translate gesture EMG to control command. As shown in Figure 6(a), Φn is a lightweight CNN architecture that introduces two tailored modules, EMG Block and CBM [Figure 6(c)]. As shown in Figure 6(b) EMG Block is based on shuffleNet [Zhang et al., 2018b; Albawi et al., 2017]. The architecture adopts two new operations, pointwise group convolution, and channel shuffle, to reduce computation costs while maintaining accuracy. CBM consists of convolutional layers, batch normalization layer [Gustineli, 2022], and Mish operation represented as

where x is the input of the Mish operation Mish( ).

We adopt the focal loss L for our network, defined as

where n=13, yi is the label in a one-hot vector form. Pi is the predicted probability for class Ci. α is the focusing parameter of focal loss, λ is the regularization coefficient, and ∑k θ2 k represents the L2 norm of the model weights.

We modify the focal loss function to increase the contribution from hard-to-classify EMG signal, and improve the model robustness. With the inclusion of L2 regularization, the gradient of L with respect to the model's weights is modified. The gradient descent update rule for a weight θk is defined as

where η is the learning rate. The partial derivative of the regularized L for θk includes the additional term due to L2 regularization, defined as

where Eq.(5) ensures that the model not only focuses on the difficult examples but also preserves simplicity and generalization capability.

3.5 Weight Segmentation Algorithm

Since EMG signals are represented as long spectrums, accurately predicting gestures from the entire spectrum is difficult. Therefore, we propose a WS algorithm module to further improve the recognition accuracy. As shown in Figure 3(e), our proposed WS algorithm Φws consists of two modules, attention-based spectrum segmentation and weightbased gesture evaluation.

Attention-based spectrum segmentation. As shown in Algorithm 1, first, employing an attention window, we segment the EMG spectrum S′ into a fragment sequence S′={S′0 , S′1 , ..., S ′n }. Then, we input each fragment in S′ to Φn , and predict the corresponding gesture class sequence C, and classification probability sequence P .

Weight-based gesture evaluation. Following the predicted C and P , we compute the final MHG class C as shown in lines 7 to 16 of Algorithm 1. In Algorithm 1, S′ is the optimized EMG signals produced by Φpr , w is the attention window size, s is stride, and t is the count of spectrum fragments in a gesture group. In lines 10 to 12, we compose t spectrum fragments to a gesture group. In line 13, the weight score W(Gi) of gesture group Gi is computed by

where |G c i | indicates the cardinal of sequence G c i , and |G c i | = t. Let ϱi indicates the gesture class with the highest proportion in G c i , the function ξ(Cj) returns 1 (or -1) if Cj is in (or in not in) ϱi. Finally, we output the final MHG class C that corresponds to the class of gesture group with the maximum weight.

Algorithm 1 Weight segmentation algorithm

Input: S′ , w, s, t Output: C 1: // Attention-based Spectrum Segmentation 2: {S′0 , S′1 , ..., S ′n } ←S′ (w, s) ; 3: for each S′n in S′ (w, s) do 4: Cn, Pn ←Φn (S′n ) ; 5: Get C ← {C0, C1, ...Cn} ; // gesture class 6: Get P ← {P0, P1, ...Pn} ; // probability 7: // Weight-based Gesture Evaluation 8: for each ci in C do 9: G c i , G p i ,W ←∅ ; 10: G c i ← {Ci, Ci+1, ..., Ci+t} ; 11: G p i ← {Pi, Pi+1, ..., Pi+t} ; 12: Gi ← {G c i , G p i } ; // gesture group 13: Compute the weight scoreW(Gi) by Eq.(6). ; 14: AddW(Gi) intoW ; 15: Compute the maximumweight Wmax inW ; 16: C ←gesture class corresponding toWmax;

4 Experiments

Below we first introduce our dataset, and then evaluate our method in accuracy, application, and user study.

4.1 Dataset

Gesture definition. As shown in Figure 8, we selected several common gesture data sets [Valli and Lucas, 2000; Gong and et al., 2016; Zhang and Harrison, 2015; Iravantchi et al., 2019] with large action amplitude that are not natural micro gestures. Therefore, we put forward our MHGs, actions, and corresponding commands.

Collection. We collect data from 20 participants, including 12 males and 8 females. Each participant underwent roughly 500 data collection sessions, with each session lasting 10 seconds. Following data processing and filtering, we allocate one-ninth of the total data as our test set. Each category has at least 5,000 samples, with certain categories containing up to approximately 12,000 samples. The diverse samples ensure our dataset is both comprehensive and robust for our evaluation purposes.

Data augmentation. We develop a data augmentation technique designed for EMG datasets. First, to resist unexpected or atypical noises, white Noise Integration is employed to simulate the varied noise spectra commonly encountered in empirical EMG recordings, and the introduced white noise follows the Gaussian distribution [Pellegrino et al., 2022]. Second, we multiply signal amplitudes with random scaling factors in [0.8, 1.2]. This processing is designed to capture the natural variability in signal amplitudes that are caused by various intensities of muscle contractions among different individuals [Grebenyuk, 2022].

4.2 Accuracy of MHGRegonition

Different MHG regonition. We evaluate the accuracy of recognizing different MHGs using 9000 samples containing 14 classes of MHGs. Experimental results as shown in Table ??, our method can achieve considerable recognition rates for different MHGs. Specifically, our average accuracy is 95.66%, the high accuracy is achieved with the slide right gesture at 99.34%, index push at 99.14%, and slide left at 98.55%. The lowest accuracy is achieved with the index pinch gesture at 89.40%.

Influence of WS algorithm. Utilizing 500 MHG samples collected from 20 volunteers, we evaluate the influence of the WS algorithm on accuracy. Experimental results show that the WS algorithm can improve 2.67% average accuracy. Therefore, our designed WS algorithm is effective for improving recognition accuracy.

4.3 Comparison with Other Methods

First, we compare our BeyondVision with ViT-HGR [Montazerin et al., 2022] which is the state-of-the-art method for EMG data analysis. For fair comparison, we train ViT-HGR by our proposed dataset, and ViT-HGR achieves an accuracy of 76.4%. In contrast, our algorithm achieved an accuracy of 95.7%. Particularly for micro gestures such as slide up and slide down, ViT-HGR typically has recognition rates of only 30% to 50%, our results consistently reach a good level.

Then, we compare BeyondVison with other wearable methods, including SensIR [McIntosh and et al., 2017], ZRing [Waghmare et al., 2023], Tomo [Zhang and Harrison, 2015], and BeamBand [Iravantchi et al., 2019]. The summarized comparison results are shown in Table ??, and our method achieves the highest average accuracy. Moreover, BeyondVison outperforms other methods in three aspects. First, we can support more number of gesture classes. Second, as shown in Figure 7, compared with related methods using large-amplitude gestures, our used MHGs are more natural, comfortable, and effortless. Third, as shown in Figure 2, the wearable device of our method is the most convenient.

4.4 Real-world Application

As shown in Figure 9, we conduct an experiment using the widely played Subway Surfer game to evaluate our real-time recognition efficiency. Subway Surfer requires quick and accurate gesture inputs, which makes it a suitable platform to evaluate our technology. The game operations include four key commands: left swipe, right swipe, jump, and crouch. Our approach can effectively map these game commands to intuitive MHGs, that is, using the thumb and the first two joints of the index finger. The experiment shows that users can effectively operate the game by BeyondVision (we show the detailed experiment processing in the demo video of our supplementary materials).

Moreover, Figure 10 shows some application samples of our designed MHG control command of Human-Computer interaction, including clicking or releasing a button, moving and dragging the cursor, inputting special commands, and so on. Each of the above control functions has been tested and verified on a real machine.

5 User Study

To subjectively evaluate the performance of BeyondVision, we invited 20 volunteers to conduct a user study. Each volunteer is asked to score 1 to 10 for our designed items, and the scores are directly proportional to user satisfaction. We conduct an introductory session on BeyondVision's functions, a three-day field test of the device, and a post-use interview to gather feedback on usability, comfort, and functionality.

Figure 11 shows the box chat of our study results. The user study indicates a positive reception for BeyondVision, highlighted by its ease of use (7.85), effective gesture recognition (8.45), and competitive edge (7.85). While overall satisfaction is high (7.45), initial impressions (4.48) and learning cost (2.4) suggest areas for potential improvement. Experimental results show that our system is user-friendly and performs competitive results.

6 Discussion and Limitation

Device parameter. Our device weighs 70g, and has a 500mAh battery, 3-hour operational time, and 24-hour standby time. Our processing is performed on a computer, and the device transmits EMG to the computer via WiFi.

Application. Our BeyondVision supports 14 classes of MHSs, and satisfies the basic requirements of humancomputer interaction, which has the potential to be applied to AR, VR, and other related areas. Moreover, BeyondVision can also be applied in combination with vision-based methods to complement each other.

Generalization. An important area of future work is improving our system's generalization ability, especially in zero-shot learning scenarios. This involves developing methods that allow gesture recognition systems to accurately interpret and respond to unseen gestures or signals without direct training. This aims to elevate the system's adaptability and ensure its reliable performance across a diverse range of novel and challenging environments, broadening its applicability and effectiveness.

Limitation. Our EMG-driven method is significantly different from vision-based methods. Specifically, vision-based methods are static-oriented, that is, a hand command can be recognized when a hand gesture matches the predefined appearance. In contrast, the EMG-driven methods are dynamicoriented, that is, if hands do not move, muscles will not produce obvious EMG signals, and thus our method can only recognize moving hands or fingers.

7 Conclusion

In this paper, we propose BeyondVision, an EMG-driven HGR system that can accurately translate EMG signals to control commands, which is sensitive to MHGs and without influence by occluded fingers. BeyondVision, we propose a user-friendly wristband-style EMG sampling device and tailor a lightweight network BV-Net with a WS algorithm, which effectively improves the user experiences and recognition accuracy. Subjective and objective experimental results show that our approach achieves a 95% average recognition rate, 2000Hz sampling frequency, and real-time gesture recognition, which can support commercial applications.

Acknowledgements

This work was supported by the (Grant Numbers 62372027 and 62372028).