Towards Micro-Action Recognition with Limited Annotations: An Asynchronous Pseudo Labeling and Training Approach

1 Introduction

Micro-action refers to the fast and tiny movements or body language signals that exhibit by humans during communication [Noroozi et al., 2021], which plays a significant Setting: Semi-Supervised Micro-action Recognition role in revealing individuals' inner states and true intentions. Traditional action recognition [Arnab et al., 2021; Feichtenhofer et al., 2019; Xie et al., 2018; Wang et al., 2024] typically focuses on identifying and classifying overt movements, e.g., running, jumping, etc. In contrast, MicroAction Recognition (MAR) [Chen et al., 2023b; Liu et al., 2021] targets the detection and analysis of minor movement changes, which are imperceptible to human eyes, making MAR inherently more challenging. Existing MAR methods are mainly developed under a fully-supervised context, which heavily depend on large amounts of high-quality labeled data [Caba Heilbron et al., 2015; Kay et al., 2017; Soomro and others, 2012]. However, acquiring labeled MAR datasets is labor-intensive and costly due to the subtlety of micro-action and privacy concerns. To alleviate the challenges of labeling MAR datasets, we introduce a SemiSupervised Learning (SSL) setting for MAR, called SemiSupervised MAR (SSMAR). As shown in Fig. 1, SSMAR aims to harness both labeled and unlabeled data to train a MAR model, enabling performance close to that of a model trained on fully-labeled data.

∗Corresponding authors.

SSL has been well established in many fields, such as image classification [Sohn et al., 2020; Berthelot et al., 2019], semantic segmentation [Chen et al., 2021; Luo and Yang, 2020], action recognition [Xing et al., 2023; Wu et al., 2023], etc. The popular SSL methods mainly focus on consistent regularization [Rasmus et al., 2015; Ke et al., 2019] and pseudo-labeling strategies [Lee and others, 2013; Xie et al., 2020b] (see Fig. 2, (c) left). However, directly applying these methods to the SSMAR task presents Comparison of pseudo-labeling accuracy on the MA-12

Comparison of pseudo-labeling accuracy on the MA-12

a new challenge, i.e., the model tends to accumulate pseudolabeling errors more significantly in the later stages of training. As shown in Fig. 2 (b), on a traditional action recognition dataset, UCF101 [Soomro and others, 2012], the popular method FixMatch [Sohn et al., 2020] is effective in mining a large number of accurate pseudo-labels. However, as the complexity of the action recognition task increases, such as HMDB51 [Kuehne et al., 2011], the accuracy of pseudolabels declines significantly and the issue of overfitting to incorrect pseudo-labels becomes increasingly severe. The problem is particularly acute in the context of MAR dataset, e.g., MA-12 [Guo et al., 2024] (examples of different tasks are shown in Fig. 2 (a)). Specifically, when applying FixMatch to the MA-12 dataset, the accuracy of pseudo-labels drops sharply from an initially high level (80%) after early training stages. Meanwhile, the model grows increasingly confident in its predictions and assigns pseudo-labels to all samples. These two observations indicate that FixMatch leads to an increasing number of incorrect pseudo-labels in late training stages. We argue that the primary cause of the overfitting issue is that the model relies on its own predictions as pseudolabels while simultaneously using those same pseudo-labels as supervisory signals for training. This approach works well when the task is relatively simple, as shown with UCF101, where it generates a large number of accurate pseudo-labels. However, as task complexity grows, this approach becomes more prone to inaccurate pseudo-labels, leading to error accumulation and performance degradation, as observed on the MAR dataset.

To tackle this challenge, we introduce a novel framework, called Asynchronous Pseudo Labeling and Training (APLT), which explicitly decouples the pseudo-labeling from the model training process (see Fig. 2, (c) right). The framework operates in two phases: 1) offline pseudo-labeling that aims to generate accurate pseudo-labels through a semisupervised clustering approach, and 2) online training that focuses on training the model using a prototype classifier in a robust manner. In the offline pseudo-labeling phase, we propose a non-parametric, semi-supervised k-means clustering algorithm to generate accurate pseudo-labels. Specifically, we use labeled data as anchor points to cluster unlabeled data, thereby generating initial pseudo-labels. To enhance stability, we introduce a labeled-augmentation technique that increases the diversity of labeled samples, making the clustering anchors more robust for guiding the clustering process. In addition, a self-adaptive thresholding strategy is proposed to filter out less reliable pseudo-labels. We then build a memorybased prototype classifier by averaging the features of samples that are assigned with pseudo-labels for each cluster. In the online training phase, the pseudo-labels generated in the offline phase are used to supervise the outputs of the memorybased prototype classifier. The prototype classifier remains fixed during training and is updated only in the offline phase. The offline and online phases are performed alternately, helping the model to avoid the overfitting problem in an asynchronous manner. Moreover, we reduce the update frequency of pseudo-labeling to several epochs instead of one epoch, further mitigating the overfitting risk. As shown in Fig. 2 (b), our method is more resistant to overfitting on inaccurate pseudo-labels compared to FixMatch.

The main contributions are summarized as follows:

• We introduce a new setting, SSMAR, designed to reduce the annotation requirements of MAR. Additionally, we identify a critical challenge in applying the FixMatch to SSMAR, i.e., overfitting on incorrect pseudo-labels.
• We propose a novel framework for SSMAR that explicitly separates the pseudo-labeling process from model training, making them asynchronous. This strategy can effectively mitigate the overfitting issue.
• Within our framework, we propose several strategies to enhance the reliability of pseudo-labels during the offline pseudo-labeling phase. Furthermore, we develop a memory-based prototype classifier to mitigate the overfitting issue during the online model training phase.
• Experiments on three SSMAR benchmarks demonstrate that our method significantly improves the performance of FixMatch, achieving state-of-the-art results.

2 RelatedWorks

Micro-Action Recognition (MAR). Unlike the traditional action recognition, MAR focuses on classifying subtle and transient human body movements. To facilitate the study of MAR, multiple MAR datasets are proposed. The iMiGUE dataset [Liu et al., 2021] combines fine-grained gesture actions with sentiment labels for micro-gesture understanding and sentiment analysis. Similarly, Chen et al. [Chen et al., 2023b] propose the SMG dataset, where participants perform various micro-gestures by narrating both false and true stories. Building on this, the MA-52 dataset [Guo et al., 2024] is proposed by capturing natural micro-actions from psychological interviews. Traditional action recognition has been benefited from large-scale labeled datasets to achieve high accuracy [Caba Heilbron et al., 2015; Kay et al., 2017; Soomro and others, 2012], allowing models [Wang et al., 2018; Lin and others, 2019; Bertasius and others, 2021]. However, obtaining labeled MAR datasets is much more difficult. The proposed SSMAR setting aims to utilize both labeled and unlabeled data to train a MAR model that can be comparable with the one trained with fully-labeled data.

Semi-Supervised Learning (SSL). SSL has gained significant attention in the community, aiming to leverage abundant unlabeled data alongside limited labeled data to enhance model performance. Two prominent SSL trends are pseudolabeling and consistency regularization. Pseudo-labeling approaches [Lee and others, 2013; Xie et al., 2020b] rely on adding high-confidence pseudo-labels to the training dataset. Consistency regularization approaches [Rasmus et al., 2015; Ke et al., 2019; Oliver et al., 2018] assume that applying perturbations to input samples or features does not change the outputs of the model. FixMatch [Sohn et al., 2020] combines pseudo-labeling and consistency regularization by using high-confidence pseudo-labels of weakly augmented samples to supervise strongly augmented samples. SoftMatch [Chen et al., 2023a] uses a Gaussian function to assign weights to samples, resolving the trade-off between quantity and quality of pseudo-labels. The above methods mainly are designed for image classification task. Instead, we propose a novel SSL framework for the MAR task, which performs pseudo-labeling and model training in an asynchronous way. Semi-Supervised Action Recognition (SSAR). The exploration of SSL in video recognition lags behind the progress in image classification. VideoSSL [Jing et al., 2021] compares SSL methods that are specifically applied to videos, revealing limitations in extending pseudo-labeling directly. LTG [Xiao et al., 2022] introduces temporal gradient as an additional modality to generate high-quality pseudo-labels for training. Recently, self-supervised learning has proven to be effective in learning powerful video representations [Dave et al., 2022; Feichtenhofer et al., 2021; Qian et al., 2024]. TimeBalance [Dave et al., 2023] leverages the temporal contrastive losses from TCLR [Dave et al., 2022] to learn the temporal distinctive teacher. SVFormer [Xing et al., 2023] explores the potential benefit of Video Transformers for SSAR. FinePseudo [Dave et al., 2025] is proposed to improve pseudo-labeling through temporal alignment for fine-grained action recognition under SSL context. Different from them, this work focuses on the SSMAR task, which is more difficult than traditional and fine-grained action recognition tasks.

3 Method

Task Definition. In SSMAR, we are given a set of human micro-action videos, defined as D = {Dl,Du}, where samples of Dl are labeled while samples of Du are unlabeled. Dl = {V i l , yi l }Nl i=1 consists of Nl videos and corresponding labels. The videos are from C categories, i.e., the label yi l is derived from the set of labels Y = {1, 2, · · · ,C}. Du = {U i }Nu i=1 consists of Nu unlabeled videos. The goal of SSMAR is to leverage both labeled and unlabeled data to learn an effective MAR model.

3.1 Overview

As shown in Fig. 3, we propose an asynchronous pseudolabeling and training framework, dubbed APLT, for SSMAR, which includes two phases: offline pseudo-labeling and online model training. During offline phase, we propose a non-parametric, semi-supervised k-means clustering method to obtain accurate pseudo-labels. In particular, labeled data serve as anchors to guide the clustering, producing initial pseudo-labels for unlabeled data. To improve robustness, we employ a labeled-augmentation approach that enhances labeled sample diversity, making the clustering anchors more dependable for guiding the clustering process. Additionally, a self-adaptive thresholding mechanism is presented to filter out less reliable pseudo-labels. Then, we construct a memory-based prototype classifier by averaging the features of samples assigned with the same cluster. During online phase, pseudo-labels produced in the offline phase are used to supervise the memory-based prototype classifier's outputs. This classifier remains unchanged throughout training and is only updated in the offline phase. By alternating between offline and online phases, the model effectively avoids overfitting in an asynchronous manner. Note that, the FixMatch is also applied in our framework in default, which uses the predictions of the parametric classifier as pseudo-labels to supervise the same classifier.

3.2 Warm-Up

Our asynchronous learning strategy is implemented after a warm up stage with FixMatch. Specifically, given a labeled video V i l , we calculate the cross-entropy loss for labeled set by:

where B is the batch size, ω(·) indicates the weak data augmentation function, pm(·) is the output probability from the parametric classifier, yi l is the ground-truth label for V i l , and H(·, ·) is the cross-entropy loss.

For the unlabeled set, we use a similar loss function but generate pseudo-labels for unlabeled data as we do not have ground-truth labels for them. The loss for unlabeled data is formulated as:

where qi is the predicted class distribution of unlabeled video data after weak augmentation: qi = pm(y | ω(U i )). ˆ qi = argmax(qi) is the predicted pseudo-label. A(·) means the strong data augmentation function. 1(·) is an indicator function of the confidence-based threshold and τ is the threshold. Overall, the loss function for warm up stage is:

After training the model with the basic FixMatch for several epochs, we will additionally include the proposed asynchronous method, which includes the offline pseudo-labeling and online model training phases.

3.3 Phase I: Offline Pseudo-Labeling

To mitigate the risk of overfitting on incorrect supervisions during training, we propose to generate pseudo-labels by a non-parametric clustering method instead of directly using the output of the parametric classifier, which includes three strategies: semi-supervised clustering, labeled-augmentation, and self-adaptive threshold strategy.

Semi-supervised Clustering. The key of semi-supervised clustering is using labeled data as anchors to guide the clustering on unlabeled data, which is inspired by [Vaze et al., 2022]. Specifically, after a training epoch, we first generate features for all data, F = {Fu, Fl}, in which the feature is obtained by the feature extractor fθ. Fl and Fu represent the features of labeled and unlabeled data respectively. We then initialize the C centroids by averaging the features of labeled data for each class and implement the following two stages. 1) Cluster Assignment: Each unlabeled instance is assigned with a cluster label by identifying its nearest centroid. 2) Center Update: The centroids are updated by averaging all data features within each cluster, where the cluster labels are ground-truth labels for labeled data and are the pseudolabels generated by first stage (cluster assignment) for unlabeled data, respectively. We iteratively repeat the above two stages until the algorithm converges, resulting in clustering pseudo-labels for unlabeled data.

Labeled-Augmentation. To improve the clustering stability, we introduce the labeled-augmentation by applying strong augmentation to the labeled data during the semi-supervised clustering. In this way, we can obtain the augmented labeled set Dsl which is also utilized to initialize and update centers in the clustering process. This strategy enables us obtain more robust clustering centers and thus produce more accurate pseudo-labels for unlabeled data.

Self-Adaptive Threshold Strategy. During clustering, hard samples will be assigned with wrong clustering labels. Using such incorrect pseudo-labels for model training will hamper the optimization. Thus, it is important to select reliable pseudo-labels. One common solution is using a threshold to filter out pseudo-labels with low-confidence. However, since micro-action categories are diverse, the difficulties of recognizing them are very different. Thus, using a single fixed threshold to constrain all classes is not reasonable. For example, the average confidence level for the head movementrelated category is typically higher than that for the upper limb movement-related category, as recognizing the former category is more easy.

To solve this problem, we propose a self-adaptive thresholding strategy, which is calculated by both global threshold and category-specific local thresholds. Specifically, given an unlabeled sample, we use its distance from the assigned centroid as the confidence indicator. The global threshold τglobal is set as the average distance of unlabeled data from their corresponding assigned clusters, reflecting the overall clustering status. Local thresholds respond to the clustering state of each class, in which we compute the class-specific local threshold for each cluster. By considering both global and local thresholds, our self-adaptive thresholding can be expressed as:

where U i c represents an instance assigned with category c in Du and N cu is the number of unlabeled instances assigned with category c. dis(.) indicates the distance to the assigned class center for an unlabeled instance. τadapt(c) is the adaptive threshold of class c. τlocal = [τlocal(1), τlocal(2), . . . , τlocal(C)] is the set of local thresholds. Given an unlabeled sample U i , if its distance to the assigned center c is lower than τadapt(c), we regard its predicted cluster label as a reliable pseudo label. Otherwise, we ignore its pseudo label. By doing so, we could obtain a filtered pseudo-label set Dsu = {U ip , yip }Np i=1 with Np instances. Memory-based Prototype Classifier. Most of the previous SSL approaches rely on the supervision of the output of the parametric-classifier during training on unlabeled data. Instead, we build a non-parametric memory-based prototype classifier based on the features of all data. Specifically, for a class c, we average all features of labeled data with class label c and unlabeled data assigned with pseudo-label c, obtaining corresponding prototype feature:

where Nc is the number of samples belonging to class c. F i c is the feature of an instance belonging to class c. Thus, the prototype feature is ρ = {ρ1, ρ2, · · · , ρC}, constructing the memory-based prototype classifier. This non-parametric classifier is directly added after the backbone for model training, like the parametric one.

3.4 Phase II: Online Training

During the online training, we not only train the model with the basic FixMatch loss based on parametric classifier but also with a margin loss based on the non-parametric classifier. Specifically, given a sample with strong data augmentation, we first calculate its feature and obtain the prediction based on the non-parametric classifier. The margin loss for labeled data is formulated as:

where F i l is the feature of the labeled sample i obtained by the current training model and ρc is the prototype feature corresponding to the ground-truth c.

Similarly, the margin loss for unlabeled data is as:

where F iu is the feature of the unlabeled sample i and ρc is the prototype feature of the assigned pseudo-label c of the unlabeled sample i.

By combining with the basic Fixmatch, our final loss is defined as:

where λ is a hyperparameter that balances the contribution between Llogit and Lmargin .

3.5 Phase Iteration

In our framework, the offline clustering phase and the online model training phase are performed alternately. Specifically, during model training, the memory-based classifier remains fixed and is used only to produce predictions. During the offline clustering, we both recalculate pseudo-labels based on the most recent model and update the prototypes of the memory-based classifier. Unlike FixMatch, this approach avoids creating pseudo-labels, updating the memorybased classifier, and producing predictions in the same iteration and manner, thereby mitigating overfitting to incorrect pseudo-labels. Furthermore, because deep networks possess a strong capacity to overfit, updating the prototypes and pseudo-labels too frequently can also lead to overfitting issue. Consequently, we run the online phase every 10 training epochs instead of every epoch.

During testing, we discard the parametric classifier and use the outputs of the memory-based classifier as the final predictions.

4 Experiments

4.1 Dataset and Experimental Setup

Benchmarks: We build semi-supervised micro-action recognition datasets based on three datasets, i.e., MA-52 [Guo et al., 2024], iMiGUE [Liu et al., 2021], and SMG [Chen et al., 2023b]. Note that, many classes in these three datasets are not satisfied for semi-supervised learning, as they do not have insufficient samples. For example, in the SMG dataset, actions like Touching or covering and Pulling shirt collar have only 11 instances each. Hence, we only select classes with sufficient samples to form the datasets, results in MA12, iMiGUE-11, and SMG-5, each of which has 12, 11, and 5 classes respectively.

Experimental Setup. For each video, we use 8-frame clips for training and testing. We use the ResNet-18 [He et al., 2016] as the backbone in default. For the warm up stage, we train the model with 15 epochs with only the FixMatch loss function. We then introduce our asynchronous learning strategy and train the model with additional 40 epochs. The model is updated by SGD optimizer with a momentum of 0.9 and a weight decay of 0.0005. The learning rate is set to 0.002 and follows a cosine decay schedule. The offline clustering phase is implemented after every 10 training epochs.

4.2 Comparison with State of The Arts

Comparison with Traditional SSL Methods. We first compare our method with 11 state-of-the-art SSL approaches designed for image classification. For a fair comparison, we use the same backbone and experimental settings (e.g., data augmentation, optimizer, and training epochs) for all methods. The results on MA-12, SMG-5, and iMiGUE-11 are shown in Table 1. Our method clearly outperforms all baselines in every setting. For instance, with only 10% labeled data on MA12, APLT surpasses FixMatch [Sohn et al., 2020] by 6.1%. Moreover, most of the compared methods fail to improve on MA-12 with 10% and 25% labeled data, reflecting the challenges posed by SSMAR. In contrast, our method significantly boosts the baseline under both conditions. We further demonstrate that APLT benefits from a more powerful backbone. For example, using ResNet-50 instead of ResNet-18 yields an additional 6.6% improvement on MA-12 with 50% labeled data. However, most of the other methods show only marginal or even negative gains when switching to ResNet50.

Comparison with SSL Methods for Action Recognition.

We also compare with the art SSL methods designed for action recognition, including TCL [Singh et al., 2021] , LTG [Xiao et al., 2022] and SVFormer [Xing et al., 2023]. Results on MA-12 are shown in Table 2. We can observe that our approach consistently achieves higher accuracies over all settings over the compared methods in which LTG and SVFormer use more complex backbones (3D-ResNet-18 or VITB). This further demonstrates the superiority of our method in SSMAR over previous SSL methods.

Further Comparison with FixMatch. Fig. 4 (left) illustrates the class-specific gains of APLT compared to FixMatch [Sohn et al., 2020]. We can find that the majority of classes benefit from our method and only two classes show comparable performance to FixMatch. Fig. 4 (right) provides a qualitative comparison between our APLT and FixMatch. Our approach accurately identifies different microactions that are misclassified by FixMatch.

4.3 Ablation Study

Analysis of Components of Non-Parametric Clustering. In Table 3, we evaluate the effectiveness of the components of the proposed non-parametric clustering. Based on the FixMatch (+SSL), we further train the model with variants of our method, in which we generate pseudo-labels by different methods, including pure k-means (+KM), semisupervised k-means (+SSKM), and SSKM with labeledaugmentation (+LA) and self-adaptive thresholding (+SAT). We also evaluate the effect of training the model with weak or strong augmentation. We make the following conclusions. First, semi-supervised learning obtains limited improvement for SSMAR. Second, using non-parametric clustering to generate pseudo-labels can significantly improve the accuracy. Third, updating the model with strong augmentations achieves slightly better results. Fourth, all the proposed components can consistently increase the performance and our full method obtains the best results on all settings.

Effectiveness of Non-Parametric Classifier and Asynchronous Strategy. The proposed non-parametric classifier and the asynchronous pseudo-labeling model training strategy are both critical components of our approach. In Table 4, we evaluate the contributions of these two techniques. Specifically, when applying the asynchronous strategy to FixMatch, we generate pseudo-labels via the parametric classifier in an offline manner and keep them fixed during training. Conversely, when using a synchronous strategy in our method, we take the non-parametric classifier's online outputs as pseudolabels, rather than the clustering results. We observe the following: 1) Using our non-parametric classifier substantially improves accuracy over FixMatch (Baseline), even under the synchronous strategy; 2) Both FixMatch and our APLT benefit from the asynchronous strategy, with APLT achieving a higher performance gain. These findings validate the effectiveness of both the proposed non-parametric classifier and the asynchronous strategy. Furthermore, when pseudo-labels and classifier outputs are generated through different mechanisms (e.g., our APLT), the asynchronous strategy delivers even stronger benefits.

GT: turning head Fix: shaking head Ours: turning head

GT: hands touching fingers Fix:tiptoe Ours: hands touching fingers

Table 4: Evaluation on synchronous strategy (SYN) and asynchronous strategy (ASY) on MA-12.

Further Evaluation on Non-Parametric Classifier. To further evaluate the benefit of the non-parametric classifier, we evaluate two variants. 1) Variant I: We remove the nonparametric classifier but further use the pseudo-labels generated by the clustering to train the parametric classifier that is used for testing. 2) Variant II: We train the model with only the basic FixMatch loss, i.e., the model is trained without the loss of Lmargin . However, we implement clustering with the final model and build the non-parametric classifier for testing. Results in Table 5 show that without using the non-parametric classifier during training significantly reduces the accuracies. This indicates the importance of the non-parametric classifier in our approach. On the other hand, only using the nonparametric classifier can also achieve a clearly higher results than the baseline, further indicating the high-quality of the pseudo-labels generated by our non-parametric clustering.

5 Conclusion

In this work, we introduce the Semi-Supervised MAR (SSMAR) setting. Through our evaluation, we show that traditional Semi-Supervised Learning (SSL) methods are prone to overfitting on inaccurate pseudo-labels. To overcome this, we present the Asynchronous Pseudo Labeling and Training (APLT) framework, which decouples the pseudo-labeling process from model training. In the offline pseudo-labeling phase, we propose a semi-supervised clustering approach to generate accurate pseudo-labels. During the online model training phase, we optimize the model with the proposed memory-based prototype classifier and the generated pseudolabels. By alternating between pseudo-labeling and training phases asynchronously, APLT effectively mitigates error accumulation and enhances the accuracy. Extensive experiments on three MAR datasets validate the superiority of APLT, achieving significant performance gains over state-ofthe-art SSL methods.

Acknowledgements

This work has been supported by the New Cornerstone Science Foundation through the XPLORER PRIZE. This work was also funded by the National Natural Science Foundation of China ( No. 72188101 &No. 62302140 &No. 62472139 ) and the Fundamental Research Funds for the Central Universities (No. JZ2024HGTB0261 & No. JZ2024HGTA0178). The computation is completed on the HPC Platform of Hefei University of Technology.