RobustHAR: Multi-scale Spatial-temporal Masked Self-supervised Pre-training for Robust Human Activity Recognition

1 Introduction

Wearable sensor-based human activity recognition (WSHAR) has seen rapid development with the increasing availability of low-cost wearable devices. Compared with visual data [Liu et al., 2024c; Liu et al., 2024a; Zhou et al., 2024] (such as RGB, depth, and skeleton), wearable sensors enable the recognition model focus more on motion information, with its robustness to environmental variations [Hong et al., 2024]. Hence, WSHAR has attracted increasing research attention and demonstrates extensive applicability across diverse domains (including industrial control, and smart classrooms).

∗Corresponding author

WSHAR is a standard time series classification task, owing to the temporal nature of the data generated by wearable sensors. Therefore, common time series classification methods like TimeNet [Wu et al., 2023] and Informer [Zhou et al., 2021] can enhance the performance of activity recognition to a certain extent. Nevertheless, traditional supervised learning methods are limited by extensive labeled datasets, which results in substantial labeling costs and potential inconsistencies. In practical application contexts, the collection and annotation of large-scale wearable sensor data often demand considerable time and financial resources, particularly in domains requiring specialized expertise. Furthermore, variations in labeling by different annotators can introduce inconsistencies that significantly affect the quality of model training, ultimately compromising its performance in real-world applications. To address the aforementioned issues, self-supervised pre-training can learn activity feature from large amounts of unlabeled data and then fine-tune for HAR with a limited set of labeled data.

Currently, contrastive learning [Zhang et al., 2025a] and masked modeling [Haresamudram et al., 2022] are two predominant approaches within self-supervised pre-training, widely used in WSHAR. Although contrastive learning can effectively extract features by data augmentation, it encounters considerable limitations in addressing data missing issues prevalent in real-world applications. This limitation arises from its reliance on complete sample pairs for training. Data missing compromises the construction of positive and negative pairs, leading to biased representations. In contrast, masked modeling can reconstruct the masked wearable sensor data based on the unmasked part. This approach facilitates the model's ability to learn robust activity representations without dependence on complete data. However, since semantic information of sensor-based action data is mainly contained in temporal variations [Dong et al., 2024], the existing masked modeling often ignores spatial correlated information among different wearable sensors. Spatial-temporal masked modeling addresses this gap by integrating spatial and temporal dimensions, allowing the model to leverage spatial relationships while capturing temporal variations. Common spatial-temporal masked modeling (such as MaskCAE [Cheng et al., 2024], STD-MAE [Gao et al., 2024], ST-MAE [Miao et al., 2024]) often separates temporal masking from spatial masking, which restricts the interaction between spatial and temporal information among wearable sensor data. Besides, different human activities often span across various spatial-temporal scales, which results in activity recognizer failing to capture intricate spatial-temporal semantic information, as shown in Fig.1.

Motivated by the above observations, we propose RobustHAR, a new HAR model with multi-scale spatialtemporal masked self-supervised pre-training. Firstly, RobustHAR extracts spatial-temporal correlated features among inter-sensor and intra-sensor by constructing locationinspired spatial-temporal 3D-variation modeling. Secondly, multi-scale spatial-temporal masked pre-training enables activity recognition model to learn robust feature across various temporal and spatial scales, thereby improving its capacity to capture the contextual information inherent in human activity. Finally, a small amount of labeled activity data is used to fine-tune the pre-training model for human activity recognition. Besides, Extensive experiments conducted on three public multi-sensor datasets demonstrate that RobustHAR outperforms the current state-of-the-art algorithms. The main contributions of our work are as follows:

• Location-inspired spatial-temporal 3D-variation modeling is constructed to mine spatial-temporal correlated information among human activity data.
• Multi-scale spatial-temporal masked pre-training with semantic-consistent multi-scale feature co-learning is designed for capturing robust spatial-temporal contextual information at different scales.
• RobustHAR fine-tunes the pre-training model with a small set of labeled data for human activity recognition by adaptive multiscale feature fusion.

2 RelatedWork

2.1 Wearable sensor-based human activity recognition

WSHAR can provide personalized solutions for humancomputer interaction by leveraging real-time, continuous data collection from wearable sensors. Early human activity recognition models often rely on a single wearable sensor, such as accelerometers, gyroscopes, and magnetometers [Chen et al., 2023]. However, using a single sensor for capturing human activity data may introduce measurement bias, which limits its ability to distinguish ambiguous activity with similar data sequences [Liu et al., 2024b]. Hence, integrated devices (such as smart wristbands, smartphones, and data gloves) have been increasingly utilized in HAR, a trend made possible by the rapid advancements in sensor technology. Meanwhile, the extensive application of deep learning (such as GRU [Lalwani and Ramasamy, 2024; Pandey and Kumar, 2024], Transformer [Kitaev et al., 2020; Zhou et al., 2021], GNN [Zhang et al., 2023; Wei et al., 2024; Wei et al., 2025], and Mamba[Li et al., 2024; Zhang et al., 2025b]) has substantially improved the performance of WSHAR. In real-world applications, factors such as sensor malfunctions and signal loss can easily lead to the loss of collected human activity data. Therefore, constructing a robust HAR model has become a primary focus of current research.

2.2 Self-supervised pre-training learning for human activity recognition

Common self-supervised pre-training learning for HAR includes contrastive learning and masked modeling [Haresamudram et al., 2022; Jain et al., 2022]. Contrastive learning [Haresamudram et al., 2021] aims to maximize the similarity between positive samples and minimize the distance between negative samples, which can guide the model to learn robust feature representations. However, contrastive learning is not suitable for data-missing scenario. The primary challenge lies in the fact that contrastive learning relies on maintaining consistency in the feature space across different action samples. In contrast, masked modeling [Cheng et al., 2024; Gao et al., 2024; Miao et al., 2024] can reconstruct missing activity data based on the unmasked data. It can dynamically compensate for missing activity data, thereby enhancing data integrity and improving the model's generalization capacity. Nevertheless, since semantic information in sensor-based activity data is often contained in temporal variations [Dong et al., 2024], most existing masked modeling often ignores spatially correlated information among different wearable sensors. Furthermore, the varying spatial-temporal scales of human activities often hinder activity recognizers from capturing fine-grained spatiotemporal semantics. As a result, existing self-supervised frameworks struggle to effectively model complex spatial-temporal dependencies under multi-scale data-missing conditions.

3 Methodology

As mentioned above, RobustHAR aims to capture multi-scale spatial-temporal correlated information among human activity data in the data-missing scenario, as shown in Fig. 2. The details of RobustHAR are described as follows.

3.1 Location-inspired spatial-temporal 3D-variation modeling

Human activity is temporally dynamic and spatially complex, especially in multi-sensor environments, where the positional relationships and interactions between different sensors can significantly impact the performance of HAR. By considering the spatial distribution of sensors, we can better understand the relative relationships of each body parts during movement, thereby extracting richer spatial-temporal activity features. Therefore, RobustHAR leverages locationinspired spatial-temporal 3D-variation modeling to capture the spatial-temporal correlated information among wearable sensor-based human activity data.

Given that the multi-dimensional data collected from N sensors is represented as D = {di,j}Ni=1 ∈ RT×N×C , j = 1, 2, . . . , T. where di,j denotes the measurement value of the i-th sensor at time j, C denotes the dimensionality of the features generated by each sensor, and T is the number of time steps. Meanwhile, each sensor's relative spatial position is represented by position matrix A. Therefore, we can transform the original multisensor human activity data into location-inspired spatial-temporal 3D-variation, and the transformation formula is shown as follows:

where X ∈ RT×C×N denotes the spatial-temporal 3Dvariation, as shown in Fig.1. The spatial-temporal 3D variation X represents human activity data that are spatially adjacent and also adjacent in the semantic space. This adjacency ensures the coherence and validity of human activity data.

Specifically, we map the sensors to the Z-axis of a 3D coordinate system according to their spatial positions on the human body. Subsequently, the time and feature dimensions of the sensors along the Z-axis are projected onto the X and Y axes of the 3D coordinates.

3.2 Multiscale spatial-temporal masked self-supervised pre-training

Multi-scale spatial-temporal masked: Human activities inherently exhibit multi-scale temporal and spatial characteristics. For instance, actions such as walking and running, though similar in basic motion patterns, differ significantly in their spatial-temporal representations. To capture such fine-grained distinctions, multi-scale spatial-temporal masked is used to extract rich contextual semantics across diverse scales. This approach not only enhances the model's ability to learn comprehensive activity features but also improves robustness in data-missing scenarios by leveraging spatial-temporal correlations at different scales.

Common masks in WSHAR include segment-wise mask, channel-wise mask, and sensor-wise mask. Segment-wise mask, channel-wise mask, and sensor-wise mask are in a similar manner, so we take sensor-wise mask as an example. Given spatial-temporal 3D-variation X ∈ RT×C×N , we randomly select Nm out of N wearable sensors for masking. Meanwhile, we define the spatial masked matrix M ∈ RT×C×N with the same shape as spatial-temporal 3Dvariation F, so M can be represented as follows:

where z represents the Z-axis index of spatial-temporal 3Dvariation X. So we obtain sensor-wise masked human activity data by the following equation:

where ⊙ represents element-wise product, Fsm means sensorwise masked human activity feature, Encoder(·) denotes feature extractor, we use transformer as feature encoder. Similarly, we can obtain segment-wise masked activity feature Fse m, and channel-wise masked activity feature Fcm.

After obtaining the masked multi-scale activity data features, we restore them to a form that closely approximates the original data by the decoder layer. This decoding process not only recovers the masked spatial-temporal activity data, but also reinforces the model's comprehension of data integrity by progressively reconstructing human activity features. Such a process enhances the model's robustness in the presence of data-missing interference, while facilitating the extraction of more meaningful features during training, thereby improving the overall performance in activity recognition tasks. We use Xsm, Xse m, and Xcm to represent reconstructed sensor-wise activity data, segment-wise activity data, and channel-wise activity data, respectively. Hence, the decoding process is shown in the following equation:

where Decoder(·) denotes decoder layer, we use Transformer as the decoder layer. Given ground truth of the masked activity data Xs , Xse , and Xc , we can compute the loss Lrec among ground truth and the reconstructed data by mean absolute error (MAE), as shown in the following equation:

where Lsrec = 1B ∑Bi=1 | Xs i -Xsm,i |, Lse rec = 1B ∑Bi=1 | Xse i -Xse m,i |, and Lcrec = 1B ∑Bi=1 | Xc i -Xcm,i | represent sensor-wise reconstructed loss, segment-wise reconstructed loss, and channel-wise reconstructed loss, respectively. B denotes the number of batch size.

Semantic-consistent multi-scale feature co-learning: When performing masking operations on multi-scale data (such as segment-wise mask, channel-wise mask, and sensorwise mask), HAR model needs to ensure that the information representation at different scales remains consistent. If the masking operations at each scale lead to inconsistent semantic information, HAR model may lose its overall understanding of human activity, resulting in a decline in recognition performance. Besides, semantic-consistent multi-scale feature co-learning can enhance the robustness of HAR model when wearable sensor data at one scale is incomplete, the others can still provide sufficient semantic information. Hence, we introduce similarity metric to ensure consistent semantic representations in the HAR model under different masking scales, as shown in the following equation:

where Xia , i ∈ {se, s, c} denotes complete human activity data that its masked portions has been replaced by the generated data, sim(·) represents similarity metric function (such as cosine similarity).

As a result, the loss function Lpre of multi-scale spatialtemporal masked self-supervised pretraining model mainly consists of multi-scale reconstruction loss and semantic consistency multi-scale feature co-learning loss, as shown in the following equation.

where λ represents balance coefficient in the loss function. The former is used to ensure the accurate reconstruction of the masked data across different scales, helping the model recover the missing information and preserve the original structure of human activity data. The latter enforces semantic consistency across the different scales, ensuring that the feature representations at each scale align in terms of their underlying semantic information.

3.3 Fine-tuning with adaptive multi-scale feature fusion

From multi-scale spatial-temporal masked self-supervised pretraining, we can obtain multi-scale feature encoders Encoders , Encoderse , and Encoderc . Given a small set of labeled human activity data Dl = {dli , yi}, i = 1, 2, · · · , nl, where nl denotes the number of labeled activity data, yi represents the label of activity sample data dli . We then extract multi-scale activity features by encoder after selfsupervised pre-training, as shown in the following equation:

where j ∈ {s, se, c} means sensor-wise operation, segmentwise operation, and channel-wise operation. Features at different scales can capture distinct aspects of human activity. For example, segment-wise features focus on subtle motion details, while sensor-wise features highlight the spatial correlations between different sensors. Based on it, we construct adaptive multi-scale feature fusion [Wu et al., 2024; Guan et al., 2025; Liu et al., 2025] to integrate features across various scales by cross-attention [Hou et al., 2019], as shown in Fig.2.

We use sensor-wise activity feature Fs and channel-wise feature Fc as examples. Firstly, we construct correlation layer to calculate a relationship between sensor-wise and channelwise human activity features with cosine distance, as calculated in Equation (9):

where Rs and Rc represent the relevance between sensorwise and channel-wise human activity features. Crossattention mechanism is then used to generate sensor-wise As and channel-wise Ac attention maps, respectively. Given the sensor-wise correlation Rs and channel correlation Rc , so fused feature can be calculated in Equation (10).

where Crossatt(·) denotes cross-attention. Subsequently, we integrate sensor-wise information and channel-wise information to more accurately capture the dynamic features of human activity, as shown in Equation (11).

where Fsc represents fused human activity features, ⊗ denotes dot product. Finally, the fusion of human activity features is used to predict the label of human activity data (Dl ). Meanwhile, the cross-entropy loss function is introduced to train HAR model.

3.4 Complexity analysis

The overall computing complexity of RobustHAR is O(LTNC 2 ), where L is the number of Transformer layers, T is the time step, N is the number of wearable sensors, and C is the dimensionality of the features generated by wearable sensors. It is slightly higher than other models (such as STMAE: O(LTNC)). However, its robustness and accuracy in data-missing scenarios far exceed those of the others.

4 Experiments

4.1 Datasets and EvaluationMetrics

Datasets. To validate the effectiveness of the proposed method, we conduct experiments on the following three multi-sensor public datasets: Opportunity [Roggen et al., 2010], RealWorld [Sztyler and Stuckenschmidt, 2016], and CZU-MHAD [Chao et al., 2022], the detailed description of datasets is shown in the Table.1.

Evaluation Metrics. The evaluation metrics are essential for assessing the robustness and generalization of the human activity recognition models. We employ accuracy (denoted as Acc), and F1 score as the evaluation metrics for the model.

4.2 Baselines

To evaluate whether RobustHAR can achieve performance comparable to supervised learning with limited labeled data, we have chosen four supervised HAR models (Reformer [Kitaev et al., 2020], Informer [Zhou et al., 2021], TimeNet [Wu et al., 2023], and MSGNet [Cai et al., 2024]). Meanwhile, to verify the superiority of RobustHAR in the spatial-temporal feature extraction, we have selected four self-supervised HAR models (SimMTM [Dong et al., 2024], MaskCAE [Cheng et al., 2024], STD-MAE [Gao et al., 2024], ST-MAE [Miao et al., 2024]).

4.3 Implementation Details

To ensure fair comparisons, we carefully tune all models, including both baselines and our proposed RobustHAR. Specifically, we first initialize hyperparameters in each baseline according to guidelines provided in the original papers, and then fine-tune them on our used datasets for ensuring their fair performance. The experiments are conducted with two NVIDIA GeForce RTX 4090 GPUs with 24GB memory. All models are implemented with Python 3.9 and PyTorch 2.4. In the pretraining phase, we set the batch size to 64 and the number of training epochs to 200, with a learning rate of 0.001. Encoder layer and decoder layer both use a single-layer transformer with 8 attention heads. During the fine-tuning RobustHAR, the batch size is set to 32 and the number of training epochs is set to 100, with the learning rate of 0.001. Meanwhile, we divide datasets into training, validation, and test sets in a ratio of 8:1:1. When fine-tuning the pre-training model with a small set of labeled data, the labeled data accounts for 15% of the training dataset. Besides, balance coefficient λ is chosen from {0.1, 0.3, 0.5, 0.7, 0.9}, and masking rate is selected from {0.1, 0.3, 0.5, 0.7, 0.9}. All experimental results are reported as mean values obtained from five independent trials.

4.4 Comparison with baselines

The experimental results demonstrate that RobustHAR achieves superior performance compared to other baselines on the Opportunity, RealWorld, and CZU-MHAD datasets, as shown in the table 2. Specifically, Transformer-based action recognition models (such as Reformer and Informer) can leverage the modeling capacity of a Transformer with an architecture that can capture long-sequence semantic information of activity data. However, these methods overlook the varying inter-series correlations across different time scales. Hence, TimeNet and MSGNet can improve the accuracy of activity recognition. Compared to supervised learning models, self-supervised learning-based human activity recognition models, particularly in masked modeling (such as SimMTM, MaskMAE, and ST-MAE), these models can enable the activity recognition model to learn global and contextual semantic features by masking portions of the input data. Meanwhile, STD-MAE can model the heterogeneity of spatial-temporal activity data by spatial-temporal-decoupled masked pre-training. While these masked modeling methods can effectively capture the spatial-temporal features of human activities, they often fail to account for the influence of spatial-temporal features across different scales on HAR. Therefore, RobustHAR achieves excellent recognition performance on the Opportunity (ACC:93.72%, F1: 93.74%), RealWorld (ACC: 95.76%, F1: 95.78%), and CZU-MHAD datasets (ACC:94.63%, F1: 94.69%).

4.5 Ablation Studies

RobustHAR primarily consists of location-inspired spatialtemporal 3D-variation modeling module (LST-3D), multiscale spatial-temporal masked self-supervised pre-training module (MSTM), and fine-tuning the pre-training model with adaptive multiscale feature fusion (AMFF). In order to evaluate the contributions of each individual module, we conduct a series of ablation experiments. These experiments systematically remove (denoted as w/o) specific components to assess their impact on the overall performance, as shown in Fig.3. Specifically, the experimental results indicate that the performance of RobustHAR is most significantly impacted when multi-scale spatial-temporal masked pre-training is not employed. On the RealWorld dataset, this results in a 10% reduction in recognition accuracy of RobustHAR. Furthermore, it is apparent that both the location-inspired spatialtemporal 3D-variation modeling and fine-tune pre-training model with adaptive multi-scale feature fusion play crucial roles in enhancing the final recognition performance. Overall, RobustHAR can capture the spatial-temporal relationships among inter-sensor and intra-sensor at different scales, thereby ensuring the semantic consistency of features across various scales.

4.6 Parameters Analysis

Impact of balance coefficient in the loss function. During the pre-training phase, loss function Lpre in RobustHAR is primarily composed of reconstruction loss Lrec and semantic-consistency loss Lcon. The former ensures that RobustHAR can recover the masked human activity data with the unmasked part, thereby capturing intricate spatial-temporal correlated information among inter-sensor and intra-sensor channels. The latter enforces semanticconsistent in activity features across different scales. Hence, how to balance the reconstruction loss and semanticconsistent loss is crucial to achieving optimal performance in RobustHAR. As shown in Fig.4(a), RobustHAR attains the highest recognition accuracy on the Opportunity and RealWorld dataset when balance coefficient is set to 0.5. Meanwhile, RobustHAR achieves the best performance on the CZU-MHAD dataset when balance coefficient is set to 0.7.

Impact of masking rate during the pre-training phase.

The masking rate in the pre-training phase plays a critical role in determining both the learning effectiveness and the ultimate performance of HAR model. An appropriately chosen masking rate strikes a balance between the complexity of HAR task and the capacity of HAR to learn meaningful and generalized representations from human activity data. When the masking rate is too low, HAR model has access to an excessive amount of information, facilitating rapid convergence but potentially leading to overfitting. It primarily stems from HAR model's excessive reliance on the available human activity data. Conversely, if the masking rate is too high, HAR model may struggle to learn sufficient meaningful patterns of human activity. This can result in hindering its ability to capture the underlying structure of human movements. As shown in Fig.4(b), when masking rate is set to 0.3, RobustHAR achieves the highest recognition accuracy on the Opportunity and CZU-MHAD datasets; when the masking rate is set to 0.5, RobustHAR performs best on the RealWorld dataset.

4.7 Robustness for HAR in the data-missing scenario

Human activity recognition (HAR) models often face challenges when data is missing, either due to sensor failures or noise in the environment. To evaluate the robustness of the RobustHAR model under such conditions, we simulate data-missing scenarios by randomly removing data points at different scales. Specifically, we analyze the robustness of different methods in data-missing scenarios by randomly setting different masking rates among different scales. For the data-missing rates, we select three conditions: 5%, 10%, and 15%. The experimental results are shown in Table.3. As can be seen from the table, the robustness of the self-supervised pre-trained model outperforms that of the supervised model. More importantly, RobustHAR achieves the best robust performance by capturing spatial-temporal dependencies at different scales.

4.8 Case study

The reconstruction performance during the self-supervised pre-training phase plays a crucial role in understanding the intricate internal structure of human activity data. We examine how well RobustHAR can recover the missing information, as well as the quality of the reconstructed features, by comparing the reconstructed data to the original input. As illustrated in Fig.5, RobustHAR exhibits varying degrees of reconstruction accuracy depending on the scale at which human activity data is masked. At the segment-wise mask, RobustHAR can capture long-range dependencies between different activities. Meanwhile, the reconstruction performance applied at the channel-wise mask remains robust. This suggests that RobustHAR can retain the channel-specific features for accurately recognizing human activities. Besides, at the sensor-wise level, where individual sensor data is masked, RobustHAR can effectively exploit cross-sensor correlations to reconstruct the missing sensor values, demonstrating its ability to capture spatial dependencies between different sensor locations. Although there is a discrepancy between the reconstructed human activity data (sensor-wise mask) and the original ground-truth data, sensor-wise features can ensure semantic coherence across different feature scales by semantic-consistent multi-scale feature co-learning. This consistency enables the model to better leverage a small amount of labeled data during the fine-tuning stage.

5 Conclusion

This paper proposes RobustHAR, a novel multi-scale spatialtemporal masked self-supervised pre-training framework.

It not only captures temporal-spatial correlations information among intra-sensors and inter-sensors by locationinspired spatial-temporal 3D modeling, but also effectively mines multi-scale robust activity features with multi-scale spatial-temporal masked pretraining model. Meanwhile, RobustHAR ensures semantic consistency across activity features at different scales. Besides, experimental results demonstrate that RobustHAR surpasses state-of-the-art baselines, and maintains robust performance for HAR in the datamissing scenario.

Acknowledgments

This work was supported in part by the Nature Science Foundation of China under Grant No.62277046, the Key Research and Development Program of Xuzhou under Grant No.KC23296, the Science and Technology Program of Xuzhou under Grant No.KC22047, the Graduate Innovation Program of China University of Mining and Technology under Grant No.2023WLKXJ179, the Fundamental Research Funds for the Central Universities under Grant No.2023XSCX048, the Postgraduate Research & Practice Innovation Program of Jiangsu Province under Grant No.KYCX23 2728.