Shap-Mix: Shapley Value Guided Mixing for Long-Tailed Skeleton Based Action Recognition

1 Introduction

Human activity understanding is a crucial problem with wide applications in real life, e.g., healthcare and autonomous driving. 3D skeleton, as a highly efficient representation, describes the human body by 3D coordinates of keypoints. In comparison to other modalities such as RGB videos and depth data, skeletons are lightweight, compact, and relatively privacy-preserving. Owing to these advantages, skeletons have become widely used in human action recognition.

∗Corresponding author.

Skeleton-based human action recognition has achieved a remarkable success [Song et al., 2018a; Chen et al., 2021a; Zhang et al., 2023]. However, existing works are mostly targeted at balanced datasets, ignoring the prevalent long-tailed distribution phenomenon in the real world. For example, NTU datasets [Shahroudy et al., 2016; Liu et al., 2019], currently the most popular benchmarks for skeleton-based action recognition, are constructed with a balanced data distribution obtained by human intervention. In contrast, the human actions in real world are under long-tailed distributions [Zhang et al., 2021; Damen et al., 2022]. For example, there are lots of data on actions of "walking" or "standing" while little on some actions such as "abdominal pain" and "falling", which can be more important for some real applications, e.g., health monitoring. This gap results in a huge limitation for the deployment of these works in the real world, i.e., a significant performance degradation on tail categories when transferred to the long-tailed settings directly, due to the triggered biased training [Liu et al., 2023].

To deal with this prevalent challenge in real world, longtailed learning has attracted much attention. Generally, the training data under the long-tailed distribution brings two challenges to model [Chu et al., 2020]. The first challenge is the data imbalance, which can cause representation bias and negative gradient over-suppression problem on the tail classes [Hsieh et al., 2021; Tan et al., 2020]. To this end, many re-balancing methods have been proposed, including the re-sampling [Van Hulse et al., 2007; Han et al., 2005; Shen et al., 2016] and re-weighting [Lin et al., 2017; Ren et al., 2020; Park et al., 2021], to help the model learn a balanced representation space. They are built upon the assumption that the optimal decision boundary is still well-defined in the given partial data. However, it is often difficult to obtain the optimal decision boundary due to the scarce tail class data. This introduces the second challenge, i.e., the difficulty of recovering the underlying distributions from limited tail class samples. As an effective way to provide new training samples, data augmentation [Chu et al., 2020; Chou et al., 2020; Chen et al., 2022] has been widely studied to enhance the distribution learning for the tail categories. However, previous augmentation-based long-tailed methods are designed for images/videos and cannot directly transfer to the skeleton data, which necessitates more careful consideration of its crucial spatial-temporal motion patterns and has been overlooked before. Meanwhile, to get rid of the complicated multiple training stages in previous works [Chu et al., 2020; Chen et al., 2022], we also aim to develop a simpler augmentation-based method with end-to-end training.

In this paper, we focus on the long-tailed augmentationbased method. Considering that human action is often represented as the combination of different motion patterns in skeletons, we attend to the mixing method to integrate different motion parts. Specifically, we propose a novel method, Shap-Mix, as shown in Figure 1. It can generate representative samples for the tail categories guided by the Shapley value [Shapley and others, 1953], a important solution concept for allocation problems in cooperative game theory. To begin with, a simple random mixing technique is developed with effective spatial-temporal design, significantly improving the representation quality of the encoder backbone. Based on this, considering the difference of the head and tail categories, we further develop a saliency-guided mixing strategy, Shap-Mix. Concretely, we first utilize the Shapley value to perform the saliency estimation for different body parts of each action category. Then utilizing the obtained saliency maps, a tail-aware mixing policy is proposed to maintain the representative motion patterns for the tail categories. ShapMix promotes decision boundary learning for the minority categories by explicitly establishing the relationship between crucial motion patterns and high-level semantics. We conduct extensive experiments to verify the effectiveness of our method. Remarkably, a significant improvement is achieved with our method under both balanced and long-tailed settings. Our contributions can be summarized as follows:

• We propose an effective skeleton mixing method, ShapMix, for long-tailed action recognition, which consists of a novel skeleton saliency estimation technique based on the Shapley value in cooperative game theory. It jointly considers the relationship between different skeleton joints to
• obtain the corresponding importance more rationally.
• Based on the obtained joint importance distribution, a tailaware mixing strategy is proposed, which prefers to produce the mixed samples that are more representative for the tail categories. It alleviates the over-fitting problem of tail categories by explicitly establishing the relationship between crucial motion patterns and high-level semantics.
• A large-scale benchmark for long-tailed skeleton action recognition, covering three popular datasets and different methodological algorithms, is provided to benefit the community. Our method demonstrates the significant performance improvement for the long-tailed learning, and the notable effectiveness is also verified in the balanced setting with full datasets.

2 RelatedWorks

2.1 Skeleton-based Action Recognition

Skeleton-based action recognition aims to classify the action categories using 3D coordinates data of the human body. Previous works are mostly based on the recurrent neural network (RNN), and convolutional neural network (CNN), treating the skeleton in the temporal series [Song et al., 2017; Song et al., 2018a; Song et al., 2018b] or pseudo 2D-image [Ke et al., 2017; Liu et al., 2017]. Recently, inspired by the natural topology structure of the human body, graph convolutional neural network (GCN)-based methods [Yan et al., 2018; Shi et al., 2019; Cheng et al., 2020] have attracted more attention, achieving remarkable performance. Meanwhile, transformer-based models [Shi et al., 2020; Plizzari et al., 2021] also show promising results by learning long-range temporal dependencies, owing to the attention mechanism.

Different from the above works on model architecture, we focus on mixing augmentation design in this paper, which improves the performance as a plug-in design.

2.2 Long-Tailed Visual Recognition

There are two typical methods to tackle long-tailed learning, i.e., re-sampling and re-weighting methods. Re-sampling methods [Van Hulse et al., 2007; Han et al., 2005; Shen et al., 2016] deal with the data imbalance issue by oversampling the tail categories or under-sampling the head categories. Re-weighting methods [Lin et al., 2017; Ren et al., 2020; Park et al., 2021], often assign a higher weight to the tail categories to balance the positive gradients and negative gradients flowing. Recently, valuable efforts have been also made on the data augmentation [Chou et al., 2020; Du et al., 2023], ensemble learning [Wang et al., 2021], decoupled learning [Kang et al., 2019], and contrastive learning [Zhu et al., 2022], providing new perspectives on the image long-tailed learning.

In contrast, the long-tailed issue has not been well explored for skeleton data, especially for the modality-specific data augmentation methods. Previous works targeted at image data can be unsuitable and sub-optimal due to the sparse and compact body structure in the skeleton. Meanwhile, some skeleton augmentation methods [Xu et al., 2022; Zhan et al., 2022] do not well consider the crucial spatialtemporal dynamics in human skeleton, especially under the long-tailed distribution, leading to the generation of less representative data samples. To this end, we propose a customized mixing augmentation with saliency guidance to generate new data samples to benefit the long-tailed learning.

2.3 Shapley Value

Shapley value [Shapley and others, 1953] is one of the most important concepts in cooperative game theory. It is used to allocate the achieved overall worth in cooperation to each player. Let us consider the set P and a function f(·) indicating the corresponding worth achieved by some players as a real number. The Shapley value of player i is its average marginal contribution to all possible coalitions S ⊆ P that can be formed without i. Concretely, it is formulated as

where the comb(·, ·) is the function of combination number. The obtained SV (i) is the final allocated worth to the player i. It well considers the interaction between different players, giving a more rational allocation solution. Meanwhile, it is shown that Shapley value satisfies good properties, e.g., Efficiency, Symmetry, Linearity and Null player [Hart, 2016].

Due to its solid theory foundation and desirable properties, Shapley value has attracted much attention for machine learning study.However, one drawback is its high computation cost as an NP-complete problem [Deng and Papadimitriou, 1994]. In response, many works adopt the Monte Carlo Sampling to give an estimation.

In this paper, we innovatively apply the Shapley value to the skeleton saliency estimation, which is the first to our best knowledge. Thanks to the sparsity of skeleton data, our approach does not require too much computational overhead.

3 The ProposedMethod: Shap-Mix

We introduce our proposed method in this section. First, a simple yet effective skeleton mixing augmentation is presented. Based on this, we further propose a customized rebalanced mixing strategy guided by Shapley value to handle the long-tailed data distribution. Finally, the whole training scheme of the model is provided.

3.1 Preliminaries

Notations. Generally, a skeleton sequence (the ith sample) can be represented as si ∈ RC×T×V×M with its label as ci, where C, T, V,M are channel, frame, joint, and performer dimensions, respectively. Note that the number of head categories is often much larger than the tail ones in long-tailed recognition. Our goal is to train a model with good generalization capacity on different categories using long-tailed data.

Mixup and Cut-Mix. Mixup [Zhang et al., 2017] and CutMix [Yun et al., 2019] are proposed as a regularization technique to improve the generalization capacity of deep model. Two skeleton sequences, si and sj, are randomly sampled first. Then for Mixup, the two sequences are mixed in the spatial-temporal dimension sMix ij = λ ∗ si + (1 - λ) ∗ sj, where λ is the mixing ratio number, sampled from the beta distribution. And for Cut-Mix, a spatial-temporal binary mask m is generated, with a masking ratio of 1 - λ. Then the mixing operation is applied in a copy-paste manner, i.e., sMix ij =m⊙si +(1-m)⊙sj, where ⊙ is the element-wise multiplication operator. The corresponding mixed label in the two methods is obtained by cMix ij = λ ∗ ci + (1 -λ) ∗ cj.

3.2 A Simple SkeletonMixing Strategy

Mixing methods have been proven effective for representation learning and well-explored in image field [Qin et al., 2020; Walawalkar et al., 2020; Liu et al., 2022]. However, for skeleton data, there is still a lack of effective mixing design considering its complex spatial-temporal dependencies. To this end, we introduce a simple yet effective mixing method, ST-Mix, for skeleton data in the following.

Spatial-Temporal Mixing Design. To fully boost the modeling of skeleton spatial-temporal dynamics, we separately consider the mixing strategy on the spatial and temporal dimensions. Note that it is for Cut-Mix to generate a better mask, while is not required in Mixup.

For spatial dimension, many works [Hua et al., 2023; Zhan et al., 2022] have found that meaningful motion patterns are captured at the part level instead of the joint level. Therefore, we first divide the skeleton into five different parts, i.e., trunk, left arm, right arm, left leg, and right leg following the previous works. Then the mixing operation occurs at the level of body parts to maintain the crucial motion patterns. Specifically, we randomly sampled Ns ∈ [N ls , N u s ] body parts as the mixing targets of the spatial dimension. N ls and N u s are the lower- and upper-bound hyper-parameters.

For the temporal dimension, we can simply copy a subclip sampled from sj into the corresponding position of si directly. However, a short clip can not accurately represent the whole action sequence for model to recognize. For example, when only a short clip of a hand in raising is observed, many actions, e.g., making phone calls and drinking, can be the candidates, leading to the confusion of labels. Therefore, we propose a temporal down-sampling strategy to construct more informative mixed samples. Specifically, supposing Nt frames are to be mixed, a clip is sampled from sj with its length in [Nt, T] and then down-sampled into Nt frames to paste into si to generate the final mixed sample.

Joint Global-Local Mixture Learning. As discussed before, Mixup and Cut-Mix construct the mixed samples in different means, where the former produces the global mixture while the latter performs the copy-paste operation on the local patterns. To introduce more diverse motion patterns and further improve the generalization capacity, we bake these two mixture generation paradigms into our training process. Specifically, for each skeleton, we randomly choose and apply a mixing method, i.e., Mix-up or Cut-Mix, jointly learning the global and local mixture semantics.

Here we highlight these specific points to distinguish our method from other skeleton mixing works [Xu et al., 2022; Zhan et al., 2022], 1) the well-designed mixing policy in spatial-temporal dimensions, 2) the joint learning for the global and local mixed data. These simple yet effective designs significantly improve the model performance compared Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI-24)

with other mixing methods as shown in Table 6, providing a strong skeleton augmentation technique. Besides, different from other works, we focus on supervised learning, especially under the long-tailed data distribution, which will be discussed in the following sections.

3.3 Shapley-Value-Guided RebalancedMixing

Now we have obtained an effective skeleton mixing augmentation. However, it adopts a random mixing manner, i.e., treating all categories equally during training. Considering the data distribution in real world is often long-tailed, it can be unreasonable because tail categories often necessitate more struggle to learn the underlying distribution. As a remedy to this issue, we further propose a customized mixing strategy, Shap-Mix, guided by Shapley value [Shapley and others, 1953] to promote the decision boundary learning for tail categories as shown in Figure 2. Our key idea is to 1) obtain the saliency map of the skeleton joints first and then 2) use it to guide the synthesis of higher quality mixed samples for tail category learning. More details are provided as follows.

Part SaliencyEstimationBased onShapley-Value. Shapley value, as introduced in Section 2.3, is effective on the allocation of the worth to a set of players in cooperation. In addition to its own good properties and solid theoretical foundation, we list three key factors that motivate our choice of the Shapley value for skeleton saliency estimation:

• Shapley value takes into account the interactions between different players, i.e., body parts in our context, to give the saliency. For example, hands (in raising) are important to recognize saluting, while it can also be confused with other actions, e.g., making phone calls and drinking with similar hand motions. Therefore, the interaction between human trunk and hands should be considered jointly when measuring the importance of hands to action saluting.
• Shapley value is based on the input instead of the deep feature maps. It can avoid the adverse effect of oversmoothing [Liu et al., 2020a] problem, the exponential convergence of similarity measures on node features, which is widely existing in GCNs. In other words, the saliency estimation can be inaccurate using the methods based on the fusion of deep features, e.g., Grad-CAM [Selvaraju et al., 2017].
• A disadvantage of Shapley value is that its calculation can be computationally costly. Fortunately, skeleton is much sparser compared with image data, which can effectively reduce the computation overhead. Besides, we adopt an online estimation method to further improve the efficiency.

Specifically, we maintain a saliency value list vc for each category c, which stores vc b for every body part combination b, i.e., vc = {vc b }. Specifically, b denotes a body part combination that may appear in the spatial mixing. It is a subset of the universal set U containing all pre-defined 5 body parts, e.g., b ={left arm, right leg}. The overall worth in our context is defined as the predicted confidence of the model on the correct class c, i.e., f c (·). Then the Shapley value vc b is computed as:

where comb(·, ·) returns the combination number and | · | is the cardinality of a set. Intuitively, vc b represents the average marginal contribution of part combination b to the prediction confidence on the class c it belongs to.

Next, we provide some more implementation details for a better understanding. First, vc b is computed and averaged only on the skeletons belonging to class c, and all these skeletons share the same saliency list vc . When computing the prediction score in Eq. (2), the selected body parts, together with the other part regions which are set to the corresponding mean joint value of the dataset, make up the complete input skeleton for model. For efficiency, we do not consider the Shapley value of temporal dimension, i.e., we always replace the same body parts in all the frames for saliency estimation. Meanwhile, we compute the Shapley value in an online way during training to share the computational overhead. Specifically, we only sample the part combination r and b once to calculate the single marginal contribution at each iteration, and then update the vc b using exponential moving average (EMA) to get the average estimation during the whole training process. The obtained saliency results will be utilized to guide the mixing synthesis as introduced in the following.

Tail-Aware Mixing Synthesis. We find that the intra-class saliency estimation is easier than the inter-class decision boundary learning in Figure 3. Therefore, the obtained saliency maps can serve as guidance for the synthesis of mixed data. Due to the scarcity of tail class samples, we suggest explicitly guiding the model to capture the relationships between the crucial motion patterns and the action semantics, revealing the possible underlying distributions of tail categories. Therefore, we introduce a tail-aware mixing policy, to produce more representative mixed data for tail classes.

Specifically, given the source data si with label ci, and the target data sj with label cj, the body parts to be mixed are sampled from a specific importance distribution d(·), which can be formulated as:

where norm(·) is the l1 normalization and |b| is the joint number in the part combination b. τ is the temperature hyperparameter. A bigger vc b /|b| indicates a higher probability that the part combination b will be selected. Specifically, two possible cases are discussed:

• If the sample number of class ci is more than cj, a random body part combination would be sampled from d(cj), which is more likely representative for class cj. Then the selected body parts of sj are pasted into si to generate the mixed data.
• If the first case is not true, a body part combination would be sampled from d(ci). Then the actual body parts to be mixed are the complement of the sampled parts in sj. They are pasted into si to preserve the crucial motion patterns in minority categories.

By virtue of this, the mixed sample is expected to be representative for the minority categories. Although the majority categories are not explicitly considered in the mixing synthesis, we assume that the model can learn desirable representations for them using sufficient training samples in datasets. As for the label of mixed data, we simply follow the same definition as discussed in Section 3.1 to make our method more general and compatible with other re-weighting methods.

3.4 Training with Shap-Mix

In long-tailed recognition task, we utilize the Cross-Entropy loss with Balanced Softmax [Ren et al., 2020] to tackle the data imbalance problem as discussed in Section 1. During training, the proposed Shap-Mix technique is integrated, and the model optimizes the loss objective for the mixed data and original data jointly. Note that the saliency value obtained by the model can be inaccurate at the beginning of training. Therefore, we warm up the model in the first few epochs to obtain a stable estimation of Shapley-value, after which we use the estimation to guide the mixing data generation. Meanwhile, we compare our method with previous works in balanced setting to verify the effectiveness. Specifically, we utilize the proposed ST-Mix in Section 3.2 for efficiency because of the balanced class distributions in this setting.

4 Experiment Results

4.1 Datasets

NTU RGB+D 60 Dataset (NTU 60) [Shahroudy et al., 2016]. There are 56,578 videos with skeleton data of 25 joints, divided into 60 action categories. Two evaluation protocols are recommended: a) Cross-Subject (xsub): the training data are collected from 20 subjects, while the testing data are from the other 20 subjects. b) Cross-View (xview): the front and two side captured views are used for training, while testing set includes the left and right 45-degree views.

NTU RGB+D 120 Dataset (NTU 120) [Liu et al., 2019]. This is an extension to NTU 60, consisting of 114,480 videos with 120 categories. Two recommended protocols are presented: a) Cross-Subject (xsub): the training data are collected from 53 subjects, while the other 53 subjects are for testing. b) Cross-Setup (xset): the training data use even setup IDs, while testing data use odd ones.

Kinetics Skeleton 400 (Kinetics 400) [Kay et al., 2017]. This dataset contains the 2D skeleton data for action recognition, extracted from the Kinetics 400 video dataset using OpenPose toolbox. It is the largest skeleton-based action recognition dataset, containing more than 260k training sequences over 400 classes in a long-tailed distribution. The sample number for each category ranges from 200 to 1000.

For the imbalanced setting, we construct the long-tailed datasets based on NTU 60 and 120 (LT-NTU 60/120), and adopt the cross-subject evaluation protocol following [Liu et al., 2023]. The imbalanced factor (IF) is defined as the number of training samples in the largest class divided by the smallest [Cui et al., 2019]. Meanwhile, Kinetics 400 is used as a real long-tailed benchmark in the wild.

4.2 Implementation Details

Our method can be applied to any backbone for skeletonbased action recognition. Specifically, CTR-GCN [Chen et al., 2021a] is chosen as our backbone for comparison and we follow its training settings and data processing for fairness. For the implementation of Shap-Mix, we randomly sample 2 or 3 body parts to mix in spatial dimension. The mixed temporal length is from 40% to 70% of the original length. The temperature τ is set as 0.2, and the momentum coefficient in the EMA is 0.9. The warm-up phase is for the first 5 epochs. Due to the less data in the constructed long-tailed dataset, we increase the training epochs to 100.

4.3 Comparison on Long-Tailed Recognition

We compare the popular long-tailed recognition works including different methodological categories. Specifically, decoupling-based methods utilize two-stage training to decouple the learning of the feature extractor and classifier. Contrastive-learning-based methods apply balanced contrastive representation learning. Ensemble-based methods employ multiple experts and perform the knowledge ensemble to obtain the final predictions. Since these methods are not implemented on the skeleton data, we conduct extensive reproductive experiments to provide a benchmark. We strictly follow the setting fairness including training epochs and utilize the same backbone, using the official implementation code as possible.

Following [Liu et al., 2023; Du et al., 2023], we first report the accuracy on LT-NTU datasets of three splits of classes, Many-shot classes (training samples>100), Mediumshot (training samples 20∼100) and Few-shot (training samples<20), to comprehensively evaluate our model. As we can see in Table 1, our method achieves the best overall scores compared with different competitors. Notably, compared with GLMC, which is the SOTA long-tailed augmentation method, our method shows a desirable overall performance improvement, especially for the head categories. Meanwhile, compared with the baseline, our method can largely boost the performance on the medium- and few-shot classes without compromising that on many-shot classes, which is difficult for most long-tailed methods.

We also conduct the experiments on Kinetics 400, which is a long-tailed dataset in the wild, as shown in Table 2. Compared with PoseConv3D [Duan et al., 2022] and GLMC, which are the SOTAs in standard supervised and long-tailed learning methods, Shap-Mix achieves the best scores owing to the utilization of the crucial motion patterns.

Finally, we present the results with different imbalance factors in Table 3. Our method can improve both general representation quality and decision boundary learning for tail categories, achieving the best scores across different imbalance factors compared with other long-tailed methods.

4.4 Comparison on Balanced Recognition

Here we show the effectiveness of our method under balanced setting. The model is trained on the balanced (original) NTU datasets to give a fair comparison with previous skeleton-based action recognition methods. We first compare our method with the state-of-the-art methods, including SGN [Zhang et al., 2020], Shift-GCN [Cheng et al., 2020], MS-G3D [Liu et al., 2020b], MST-GCN [Chen et al., 2021b], CTR-GCN [Chen et al., 2021a], Info-GCN [Chi et al., 2022], EfficientGCN [Song et al., 2022], FR-Head [Zhou et al., 2023], StreamGCN [Yang et al., 2023], HD-GCN [Lee et al., 2023b], STC-Net [Lee et al., 2023a]. Following previous works, we report the results of multi-stream fusion, i.e., joint, bone (2-stream), joint motion and bone motion (4-stream). As shown in Table 4, our method achieves the best performance across different datasets. Remarkably, our method with 4 (/2) streams can outperform many latest methods with 6 (/4) streams, verifying the significant effectiveness.

Meanwhile, our augmentation method can be equipped with different backbones. Compared with another model-

Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI-24)

85.9

86.6

86.3

-

87.2

Spa. Temp. xsub

91.1

✓ 90.1

✓ ✓ 91.6

Table 7: Ablation study on

spatial-temporal mixing design in ST-Mix. "Spatial" is mixing in the part-level or joint-level. "Temporal" denotes to copy directly or after randomly down-sampling.

agnostic method, FR-Head [Zhou et al., 2023], our method can bring consistent performance improvement without introducing additional training parameters as shown in Table 5.

4.5 Ablation Studies

Next we present the ablation results conducted on the (LT) NTU 60 dataset under cross-subject protocol.

Effectiveness of ST-Mix Design. We compare our ST-Mix with different mixing methods for skeleton data in Table 6. The reported results are obtained under the balanced action recognition with the full NTU 60 dataset. Compared with Mixup and Cut-Mix, our method jointly learns these two mixture patterns, yielding a notable performance improvement. Meanwhile, due to the well-designed spatial-temporal mixing policy, our method outperforms other mixing methods, with their respective effects presented in Table 7.

Visualization of Saliency Estimation. We choose the part combination containing 2 or 3 parts and estimate the corresponding importance. As shown in Figure 3, we can obtain a rational skeleton saliency estimation for many-, medium-, and few-shot classes. For example, in handshaking action, the most salient part combination is obtained as the two legs with the right arm, which is reasonable because people usually stand with their hands out in this action. Meanwhile, it can be found hands and arms play an important role in most human actions, especially the dominant (right) hand of most people. These results are promising for more fine-grained action recognition and spatial localization. We hope that more works will emerge in the future to utilize and explore skeletal saliency maps.

Effectiveness of Shapley Value Guidance. The results of different guidance for mixing augmentation are provided in Table 8. As we can see, the model outperforms the mixing method without guidance either using re-samping or reweighting method. Meanwhile, we compare with another guidance, Grad-CAM [Selvaraju et al., 2017]. However, due to the over-smoothing problem, it can not achieve desirable performance improvement. In contrast, we use Shapley value as guidance, which is based on input and well considers the relationship between different joints, and brings further performance improvement.

5 Limitation and Conclusion

In this paper, we explore the long-tailed skeleton-based action recognition, which has been largely overlooked before, and propose a novel augmentation-based method, Shap-Mix. Specifically, we develop a saliency estimation method based on Shapley value and a tail-aware mixing policy to preserve more representative motion patterns, improving decision boundary learning of tail classes. One limitation is that due to the inherent computational complexity of Shapley value, our method is primarily for sparse data such as skeleton. This can be alleviated by performing estimation every n iterations, which is a trade-off between estimation accuracy and computation overhead. Extensive experiments on both balanced and long-tailed settings verify the effectiveness.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62172020, and in part by the State Key Laboratory of Media Audio and Video (Communication University of China), Ministry of Education, China.