Fitness Activity Recognition Using a Novel Pressure Sensing Mat and Machine Learning for the Future of Accessible Training

1 Introduction

Physical inactivity is still a major problem despite a growing wellness market. According to the Center for Disease Control and Prevention (CDC), half of adults lack sufficient physical activity, contributing to chronic disease risk, with 28% of the global population remaining physically inactive [CDC, 2022; Guthold et al., 2018]. There is a well-documented link between this sedentary lifestyle and serious mental and physical health issues like heart disease, cancer, diabetes, and depression [Gonz´alez et al., 2017; Martinson et al., 2001]. The wellness market has seen significant growth recently, becoming a dominant sector within the 4.5 trillion wellness industry, and novel fitness technologies have garnered special attention in both academia and industry [Yeung and Johnston, 2019]. Yet, a widening disparity exists between the expanding body of wellness research, fueled by technological advancements, and the escalating global prevalence of physical inactivity, exacerbating one of the 21st century's most significant public health crises [Blair, 2009].

The inaccessibility of fitness training in particular and its associated technologies exacerbates this gap. Fitness and sports are still inaccessible to many due to socioeconomic factors, cultural and demographic barriers, and more recently the threat of global pandemics that disrupt disrupt traditional avenues for physical activity. Emerging technologies, such as Artificial Intelligence (AI) and sensing, offer promise in reducing these barriers, potentially revolutionizing human engagement in physical activity for improved health outcomes.

A significant body of experimental and theoretical studies over the last two decades has focused on the development of intelligent systems for empowering human activity and augmenting human performance in sports and fitness for the promotion of overall well-being. Enabled by reliable, affordable sensing technologies and increased computing power, understanding human activity, from posture recognition to skill modeling has received special attention. However, most of these works focus on sensing modalities that are challenging for deployment in real-world use cases, especially with regards to accessibility. For instance, many of the previous works use proof-of-concept solutions that rely on bulky, expensive and restrictive interfaces such as force plates, or inaccurate off-the-shelf wearable trackers that need to be attached to the user during the fitness activities.

We propose a novel system for accessible fitness using a novel pressure sensing mat called LifeMat and ML for gym activity recognition and guidance from real-world user data. In this paper we target four main gym exercises, squats, deadlifts, bicep curls, and shoulder presses. We demonstrate the feasibility of our system in practice in a live real-world gym case and report on feedback from its natural users which include an elderly group sensitive to issues of accessibility, and we note the challenges and direction of future research in this area. Our main contributions are highlighted as follows:

• We present a novel pressure sensing mat called LifeMat for the detection and recognition of four main gymbased fitness activities in a real-world gym scenario.
• We present the first detailed and previously unreported COP profile for each target activity in terms of several COP-related metrics for a comparative investigation of the activities as targets of ML-based recognition.
• Based on this, we identify salient features for the MLbased recognition of these gym-based activities for similar applications.
• We present and compare the performance of several ML models in the activity recognition task and achieve over 98.5% accuracy, outperforming previous works using universal, transferable and explainable COP metrics.
• We show that COP displacement and COP balance are particularly important for ML-based activity recognition for further research in this area.
• We deploy our system in a live real-world use case in a natural gym environment and discuss challenges and directions for future works in the area of ML-powered gym and sports activity recognition for accessible fitness

In Section 2, we review previous literature in fitness activity sensing and recognition; in Section 3, we detail our method of ML-based activity recognition using a novel pressure sensing mat and ML; in Section 4, we present and discuss experiments and results on the COP profile of each activity, ML model performance, and the real-world case study; we conclude in Section 5 with a summary of our approach and directions for future work.

2 RelatedWorks

The sensing technologies and computational methods applied in exercises activity recognition vary substantially from one application to another. In this section we detail the most common sensing technologies and data modalities related to our proposed method, i.e wearable sensing and pressure sensing.

2.1 Activity Recognition Based onWearable Inertial Sensors

Wearable inertial sensors are the most common wearable devices for measuring daily-life and fitness activities. They are generally composed of accelerometers, which measure acceleration, gyroscopes which measure angular velocity or turn rate, and magnetometers which measure magnetic fields. [Haider et al., 2020] used wearable inertial sensors for volleyball action modelling. Data from IMUs worn on both wrists by volleyball players during natural training was used to train a binary ML classifier to detect action and non-action instances. In a similar work, [Kautz et al., 2017] used wristworn wearable sensors to collect data on beach volleyball actions for the purpose of injury prevention. [Ma et al.,

2018] used player-worn IMUs sensor to collect data on common basketball postures. Other works explored using IMUs mounted on sports objects and tools rather than on the players themselves. For gym exercises activities, [Depari et al., 2019] used single wrist-worn wearable IMU and LDA to classify nine undefined exercises from a 7 user dataset and achieved an accuracy of 93%. [Yoshida and Yuda, 2023] used a single wrist-worn wearable sensor (Silmee W22) and Random Forest (RF) to detect yoga workouts from awake and sleeping activities with an accuracy of 96.2%. [Bian et al., 2022] also used a wrist-worn IMU and Convolutional Neural Networks (CNN) to detect 11 gym exercises with an accuracy of 90.4%.

Despite their popularity for activity data collection, wearable inertial sensors have limited sensing capabilities in capturing complex motions and offer an incomplete picture of the target activities. They also require users to wear or attach devices to their body, which is invasive and problematic in light of the recent sanitary crisis. The mentioned works share common limitations in terms of collecting sensor recordings that are salient for the target task. IMU data alone is prone to drift and noise, in addition to poor generalisation capability both for intra-individual and inter-individual recognition tasks. Furthermore, third party fitness trackers come with their own set of user privacy and data security concerns, and a well documented logging-fatigue that hinders continuous use [Gabriele and Chiasson, 2020].

2.2 Activity Recognition Based on Pressure and Force Sensing

Pressure sensing for activity recognition has been mostly explored through floor mat systems because of the ubiquity of mats in fitness settings and their suitability for recording relevant posture, pose, motion and action data for many common exercises. Pressure sensing mats are typically flat sensing devices used to detect pressure distribution on an analog scale. They generally comprise of sensor arrays or strain gauge platforms. The sensors are processed by a microcontroller which represents the data in various formats and displays.

Previous work has demonstrated the effectiveness of combining pressure sensing data and ML for human activity recognition and correction [Bourahmoune and Amagasa, 2019; Bourahmoune et al., 2022a; Bourahmoune et al., 2022b; Ishac and Suzuki, 2018]. These works have demonstrated improved well-being and learning skills through realtime physical sensing and providing assistive feedback in various modalities including haptic, kinetic and emotive.

[Jang et al., 2015] designed a system for yoga content personalisation based on a physical floor mat for pose recognition. The sensing system in the mat uses piezoresistive technology for real-time analog detection. User pressure data is collected for pose recognition using offline posetemplate generation and real-time pose-template matching. They achieved acceptable recognition performances on eight yoga poses, however, the recognition algorithms they proposed are rigid and do not take into account pressure distribution, balance and body characteristics which have a significant influence on pose recognition in such systems. [Cheng et al., 2016] also used pressure distribution data from a pressure sensing floor mat to recognise gym activities like stretching exercises and weight training. Similarly in [Jang et al., 2015], the exercises were recognised based on a template-matching method.

[Sundholm et al., 2014] and [Zhou et al., 2022] framed the recognition task as a computer vision problem and used intensity images of ground-performed exercises obtained from resistive floor mats (Smart-Mat and Quali-Mat) respectively. [Sundholm et al., 2014] used k-Nearest Neighbours (KNN) to recognise 10 ground-based common exercises with an accuracy of 88.7%, while [Zhou et al., 2022] used CNN to recognise 9 ground-based exercises with an accuracy ranging between 73.6% and 98.6%. However, it is important to note that the method of transferring pressure sensing data to the image domain for vision-based ML recognition severely limits the deployability of their system in a feasible, real-time real-world scenario.

[Gouwanda and Senanayake, 2008] proposed a modular force sensor mat system for sports biomechanics and gait analysis. This mat uses Force Sensitive Resistors (FSR) to detect walking, running and jumping. [Vanegas et al., 2021] similarly developed a matrix of FSR sensors embedded in a mat to detect motions related to jumping and jumping with arm swings. Works based on FSRs are limited by the low precision and measurement variability of FSR sensors which prevent their reliable application in recognising the complex motions involved in sports and fitness activities. Beyond these, force plates and balance boards have also been used for activity recognition, however, these systems are elevated and have a small execution area that not only restricts the natural execution of most activities but also pose a real hazard to user safety in dynamic environments such as gyms or the home [M¨oller et al., 2012].

Unlike the systems discussed above, our proposed LifeMat was designed to be a portable and lightweight IoT solution for activity recognition from the universal, transferable and biomechanics-rooted COP metrics using real-time ML. COP is an important metric in stabilometry, gate analysis and athlete performance assessment [Jeon and Cho, 2020; Agostini et al., 2013]. It remains underinvestigated in MLbased activity assessment and we demonstrate the strong discriminative and relational characteristics of COP-derived metrics both inter-activity and intra-activity respectively.

Additionally, the LifeMat was validated against state-ofthe-art medical-grade force plates used by medical professionals to assess balance impairments. Our proposed method is designed for real-time use in a natural real-world environment with minimal adoption barriers to allow for greater accessibility. Furthermore, we validate this in a long term realworld case study in a natural gym environment and discuss directions and challenges related to the accessibility of AIpowered human movement analysis system for greater social impact.

3 Proposed System for GymActivity Recognition

We propose a novel system for accessible training using a novel pressure sensing mat called LifeMat and ML for gym activity recognition and guidance based on real-world user data. We test our system on four common gym exercises, squats, deadlifts, bicep curls and shoulder presses. The choice of these activities was based on pilot surveys with the enusers and close consultation with the gym implementing the real-world case study. We use pressure sensing as the primary methodology of physical human sensing in this implementation as it is particularly suitable for fitness activity detection. During training it can capture the underlying movementspecific information of the different activities which is often lost in other modalities like inertial and vision sensing.

3.1 Novel Sensing Interface

The LifeMat is an internet of things (IoT) pressure sensitive interface embedded into a standard training mat interface. It uses piezoresistive technology and a pressure sensor matrix formation for measuring human posture in real-time. This also provides insight into the position, rotation and location of each individual foot on the mat. LifeMat is connected to a custom-made PCB developed by our research team which processes all the data from the pressure sensor matrix and communicates this information to the user facing interface which we have also developed. The PCB is connected to a Latte Panda SBC and communicates via USB-C protocol. The LifeMat can also operate wirelessly using Bluetooth Low Energy or WiFi communication for data transfer, but for the purposes of this research we have operated it in a wired connection state. The Latte Panda SBC is connected to an external HD monitor display via HDMI cable. The LifeMat device can be powered through a standard 5V power supply. The device is flexible, thin and lightweight, allowing a practitioner to perform their techniques in a natural setting and grounded surface as they would with regular training mat. Its dimensions are 1000 mm long, 500mm wide and 3mm thick. This large sensing area allows for accurate analysis of human performance in various sports, physical arts and exercises, which are not possible using conventional force plate systems. In our configuration, the LifeMat can sense 924 individual pressure points in 12-bit resolution in real-time.

3.2 ML-Based GymActivity Recognition

In this study, we focused on four basic and common activities found in gym-based fitness in order to validate the overall framework. These activities were the Squat, the Deadlift, the Shoulder Press and the Bicep Curl. With real-world implementation for social impact in mind, we consulted with the gym where the solution was validated to define these exercises as the target activities based on their popularity among the members of that gym. These activities are also an ideal target for modelling as they are familiar to most people in gym-based and home-based fitness scenarios, and are performed by users from a wide range of activeness level, fitness ability and age. Furthermore, each activity has unique biomechanical and kinematic characteristics, making them a good benchmark for testing the feasibility and validity of this framework in general and of ML-based activity recognition from sensing data in particular. Previous works have conducted different analyses of gym-based activities such as the squats and deadlifts focusing on muscle activation, kinetics and kinematics, COP and ground force. However, an in-depth study of the COP profile of each of these activities and how they compare to each other is still lacking. We investigated the COP profile of these activities in depth and in particular their salience as features for training machine learning models for gym activity recognition.

As the systemwas intended for real-world implementation, it was vital to collect real-world data in a typical gym environment from users familiar with the exercises. We conducted a two week long study at a major national stadium in Japan (Tokyo Dome) as part of a demo where members of the gym and the public were able to freely interact with the system.

The system set-up for this experiment is shown in Figure 1 and consisted of the LifeMat pressure sensing mat, a monitor display, the LifeMat training app, in addition to different types of weights (bar, dumbbells) for exercises that required them and users who preferred to use these accessories in their training. Prior to recording each activity, all participants were fully informed of the experiment protocol and instructed to perform their exercises naturally as they normally would during their fitness training. To ensure that all participants performed the correct exercise in a standardised duration, they were asked to follow the timing of a virtual avatar in the training app displayed on the monitor and animated to perform the same activity for that session (start time, execution time and finish time). The participants were instructed to stand in the same position according to each activity's foot placement requirements. The data recording started only once the correct position has been validated. We implemented a manual labelling tool that allowed for obtaining the ground-truth for each activity for training the ML models.

A stakeholder constraint of running this study required collecting fully anonymised pressure sensing data and not tracking participant personal information. We thus collected data from 83 training sessions using the same equipment, with each session lasting 5-15min, for a total of 8385 recordings where each session contained the same type of activity repeated at least three times. These sessions were distributed as follows: there were 33 sessions for the Squat, 19 session for the Bicep Curl, 16 sessions for the Deadlift, 15 sessions for the Shoulder Press. An opt-in user demographic and feedback survey was completed by 30 (17M, 12F) participants, although not every participant who performed a training session chose to respond to the survey.

We trained several classifiers on activity recognition task using 13 COP-centric features. These features are described in detail in Table 1 based on their type group. The performance of ML models was compared with the following criteria in mind for models to target for deployment: (1) highly accurate and (2) computationally cheap and (3) efficiently deployable on edge devices in real time. The investigated models were: Logistic Regression (LR), Linear Discriminant Analysis (LDA), k-Nearest Neighbours (KNN), decision trees (DT), Random Forest (RF), Naive Bayes (NB), Support Vector Machines (SVM), Multi-Layer Perceptron (MLP).

To investigate the importance of the COP features further, we ran ablation studies of the different COP features based on their type. We grouped the different features according to their measurement type and these were: displacementcentric (D), velocity-centric (V), acceleration-centric (A), and balance-centric (B).

We also investigated the effect of using combinations of the different feature groups shown in Table 1, which were displacement and velocity (D-V), displacement and acceleration (D-A), velocity and acceleration (V-A), displacement and balance (D-B), velocity and balance (V-B), acceleration and balance (A-B). Finally, we deployed the best performing model using our framework as part of the real-world case study previously mentioned.

4 Experiments and Results

Informed consent was obtained from all participants and the study received ethics approval from The University of Technology Sydney (ETH22-7457).

4.1 Sensing Interface validation

Although a detailed bench-marking of the LifeMat against state of the art force plates is beyond the scope of this paper, we note for completeness our validation of the accuracy and reliability of the COP data obtained from the LifeMat. We ran a clinical balance study with 16 healthy participants that did not suffer from any severe visual, vestibular or somatosensory impairments and demonstrated that the LifeMat captured accurate COP displacement both in the medial-lateral direction and the anterior-posterior direction for 5 principal human motions which are: leaning right, leaning left, leaning forward, leaning back and centre-balancing.

4.2 COP Profile of the Target GymActivities

Unlike previous works, we explore in this paper the more reliable biomechanics-based approach for training the ML models for gym activity recognition using COP features. To identify the relevant COP features for training ML models for activity recognition, we first compare the underlying COP characteristics of the different activities. Figure 2 show the change in COPX and COPY, in addition to the average, minimum and maximum COPX and COPY for all four activities. The different colours indicate different activity sessions.

Empirically, the results clearly show recurring patterns between the activities for both intra-activity and inter-activity. The differences between the mean COPX and COPY for the four activities are statistically significant (One-Way ANOVA, p < 0.001 for both COPX and COPY). Overall, for both COPX and COPY, the Deadlift recorded the biggest range in COP while the Shoulder press recorded the smallest range in COP. In the case of the Bicep Curl and the Shoulder Press, the change in COPX and COPY tends to be stable around the same value range. This is true for within individual sessions where the change in COP is minimal overall. In the case of the Squat and the Deadlift, the change in COPX and COPY is more pronounced, particularly within individual sessions. Interestingly, the Deadlift and the Shoulder Press recorded bigger changes in COPX rather than in COPY. This could be related to the use of the same heavy bar accessory in these two activities which requires rapid changes in COPX to offset the weight of the bar and maintain balance. While the target activities share a similar motion profile, these first results suggests that fine position changes in the medial-lateral and anterior-posterior COP hold activity-specific information.

Expanding on this, Figure 3(A) shows the COP excursion for all four activities. COP excursion is a measure of postural control used that has been used in stabilometry studies to approximate postural sway [Winter, 2009; Lafond et al., 2004]. COP excursion is calculated as the difference between the maximum COP displacement and the mean COP displacement. We show the COP excursion in both the medial-lateral direction and in the anterior-posterior direction. The results show that the COP excursion of the different activities diverge significantly from each other, with the squat recording the biggest excursion in both the mediallateral direction and the anterior-posterior direction (One-way ANOVA, p < 0.001). The smallest medial-lateral COP excursion was recorded by bicep curl, while the smallest anterior-posterior COP excursion was recorded by the deadlift. This is consistent with the empirical observation of the motion of these activities where the bicep curl is the most stable in terms of postural sway. Interestingly, the Shoulder Press showed a significantly bigger divergence between COPX and COPY than the other activities (Student T-Test, p < 0.05).

Figure 3(B) shows the average distance between the left foot COPX and the right foot COPX. This is not simply the distance between the position of the left foot and the right foot (L/R) but specifically the distance between their respective COPs. It show that there was a significant divergence between the four activities (one-way ANOVA, p < 0.001) with the squat recording the highest COP L/R distance and the bicep curl recording lowest COP L/R distance.

From these results, it is evident that COP data holds activity-specific information that allows for discriminating between the different activities. This suggest that COP features are good candidates for designing explained AIpowered solutions for gym activity recognition and guidance. This is a promising result as COP data is a universal indicator of balance and sway that can be obtained from a wide range of sensing systems including force plates and pressure sensing mats as discussed in Section 2. For the purpose of designing accessible fitness solutions, COP data is transferable across different sensing systems, and it can be obtained in a non-invasive manner that doesn't require users or participants to wear or attach any devices to their body. The latter is particularly important in scenarios that require data collection from real-world scenarios.

4.3 ML-Based Activity Recognition

Based on the results, we investigate the application of ML for activity recognition using COP data relating to COP displacement, balance, velocity and acceleration. Table 2 shows results for the accuracy and F1 score for the different models in the activity recognition task. We trained the different models using all the features discussed in Table 1 together and using each group individually. The results show that the models trained on all features and displacement-centric features performed the best. The best performing models were RF, SVM, KNN and CART with RF achieving the highest accuracy of 98% and a F1 score of 98%. These results outperform previous works discussed in Related Works and in particular all previous works that use pressure sensing data from a floor mat and ML to detect gym activities. For each model, the hyper-parameters were fine-tuned with grid-search (for RF we used 100 estimators, for KNN we used 10 neighbours, for SVM we used a RBF kernel, for CART we used a minimum sample leaf of 20).

We note a related work in the visual domain from [Zhou et al., 2022] that achieved accuracies ranging between 73.6% and 98.6%, however they used images of the pressure intensity maps as the input of a Conv3D model, which has serious limitations in terms of deployability because systems based on the transformation of pressure sensing data to the image domain for classification are not feasible for real-time deployment, especially on edge devices. Importantly, we used COP-related features which are universal, transferable and interpretable in the context of the design of accessible and biomechanics-based solutions for promoting human well-being through fitness. We also achieved a high accuracy using a common, fast and robust machine learning classifiers suitable for deployment on devices with limited computational resources. Thus, despite the appeal of using more complex methods such as deep learning for activity recognition, these would be computationally prohibitive with our IoT hardware, particularly in our user-interfacing real-time use case. A main goal of our approach is to facilitate deployment in the real-world, using development-friendly ML models and highly salient target-specific features. The improvement in classification accuracy achieved in our study is likely due to the combination of the sensing capabilities of the LifeMat which was designed to capture fine changes in human pressure distribution and the use of salient biomechanics-based training of the ML models. Furthermore, unlike previous research discussed in Related Works we targeted activities with high inter-activity similarity and deployed our method in a real-world gym use-case in real-time with natural users, which we discuss further below.

The model trained on the displacement-centric group alone performed better than the model trained on all features and this increase in accuracy is significant (Wilcoxon Signed Rank test, p < 0.05). This can be explained by the high degree of noise typically present in velocity and acceleration measures, and this is supported by the observation that the models trained on the velocity-centric and accelerationcentric features achieved significantly lower accuracies. Interestingly, models trained on the balance-centric features alone performed significantly better that the velocity- and acceleration-centric groups. COP-derived balance measures have not been investigated in previous works, but we show here that they are salient for fitness activity recognition.

We further analyzed the COP features relevant for activity recognition by investigating the different combinations of the feature groups shown in Table 1 and Table 2. Table 3 shows the performance of the different ML models when trained on a mixture of various COP features, which were displacement and velocity (D-V), displacement and acceleration (D-A), velocity and acceleration (V-A), displacement and balance (DB), velocity and balance (V-B), acceleration and balance (AB). Here, the best performance of 98.5% was recorded by the combined displacement and balance feature group using SVM, and this is significantly higher than the result achieved by training on all features for the same model (Wilcoxon Signed Rank Test, p < 0.05). This is consistent with the results shown in Table 2 where the displacement-centric and balance-centric group performed well in comparison with the other groups. As both these groups were capable of capturing salient information about the activities, their combination performs even better. The results also show that the combination of displacement with velocity and displacement with acceleration improves the recognition performance when compared with the displacement-centric group alone. However, the combination of velocity with balance and acceleration with balance did not improve the performance when compared with the balance-centric group alone. This points to an interesting relationship between COP displacement and COP acceleration for analysing human motion in general and fitness activities in particular, which was documented in medical and stabilometry literature [Winter and others, 2018]. COP acceleration and its related Centre of Mass (COM) acceleration are typically used for the quantification of postural instability and assessment of balance control [Winter and others, 2018; Wang et al., 2019].

Overall, results from the COP profile experiments and MLbased activity recognition experiments show that COP data is an important measure to consider for designing accurate, reliable, and explainable human activity recognition solutions for fitness training augmentation. As health awareness grows, the combination of low-cost systems like the LifeMat and ML can pave the way for improved accessibility of these emerging technologies to promote well-being and education. We aim to extend and provide our dataset with additional activities.

4.4 Real-World Case Study

This study addresses physical inactivity and promoting fitness for improved health and well-being for diverse demographics, including the elderly, using AI and IoT (SDG3, SDG10, SDG9). To ensure practical relevance and usability, we conducted a long-term user study lasting two weeks in a major national stadium gym in Japan (Tokyo Dome), directly engaging with end-users, and it involved collaboration and consultation with medical experts, gym establishments, technology developers, and researchers (SDG17).

The case study focused on two main aspects: activity recognition and activity correction. While the latter is beyond the scope of this paper, it requires the activity recognition step proposed in this work. All data presented in this paper was collected as part of the same set-up. In this casestudy, we validated the feasibility of deploying the LifeMat sensing interface with ML-based activity recognition using the displacement and balance COP features identified in our experiments and the best performing algorithm from the results. Users of different age groups participated in the study. Notably, feedback from the elderly group (40% of the survey respondents) highlighted the benefits of the system in terms of: 1) the usefulness of providing balance-based analytics, 2) the non-judgmental aspect of an AI trainer and 3) the reduced social complexity and higher accessibility of the system.

5 Conclusion

In summary, we introduced a novel ML-based fitness activity recognition platform that demonstrated promise for social impact through technology-empowered physical activity promotion. We collected real-world user data of fitness activities and analyzed their detailed and previously under-investigated COP profiles. We identified relevant COP displacement and balance metrics to be particularly important for ML-based activity recognition using similar systems going forward. We achieved 98.5% accuracy in recognising four main gym activities using a common, robust and deployable ML model. We demonstrated the feasibility of our system in a live realworld use case in a natural gym environment.

In future work we focus on our proposed platforms capabilities for activity assessment and skill coaching through providing AI-powered feedback based on real-time human activity recognition. Additionally, we will investigate different modalities such as video and inertial data and their fusion.