Action Recognition with Multi-stream Motion Modeling and Mutual Information Maximization

1 Introduction

Having a mental representation of what actions other humans are performing is crucial for us to adjust our behaviors and plan our course of actions [Liu et al., 2021; Hao et al., 2022a]. Similarly, the capacity for machines to model and recognize human actions is very much coveted. Now, action recognition spawns a wide spectrum of applications including public space surveillance [Wang et al., 2016], human-robot interaction [Hao et al., 2022b], violence detection [Singh et al., 2018], and autonomous driving [Lu et al., 2018; Hao et al., 2022b]. As such, action recognition has attracted extensive attention in the past decade [Shi et al., 2019b; Chen et al., 2020b; Chen et al., 2020a].

* Corresponding author

https://github.com/ActionR-Group/Stream-GCN

Fundamentally, action recognition amounts to learning a mapping from a pose sequence to an action class label. The task is inherently challenging due to the high dimensionality of the input pose sequence and the subtle motion details which differentiate different actions. In this paper, we focalize action recognition with 3D articulated pose sequences as inputs, given that 3D articulated pose can now be conveniently acquired through commodity motion capture systems or extracted from videos via off-the-shelf pose estimation methods like [Wang et al., 2014].

Earlier approaches for action recognition [Liu et al., 2016]

cast the human body as a set of 3D joint coordinates, which in fact treat different joints as independent entities and fail to capture connections between joints [Liu et al., 2019b]. Recently, GCN (Graph Convolutional Network) based methods have become popular solutions as the human skeleton inherently possesses a tree-like graph structure. A first class of work puts the efforts in selecting and enriching features to better represent motion [Shi et al., 2019a; Shi et al., 2019b], facilitating action recognition from the input feature perspective. For instance, [Shi et al., 2019b] advocates incorporating the lengths and directions of bones as second-order input features, which are informative for action recognition. [Shi et al., 2019a] adds frame-wise bone and joint velocities as additional motion features to enhance the body representation. Another line of work resorts to devising sophisticated adjacency matrices, seeking to pursue better modeling of joint dependencies. For example, [Liu et al., 2020] introduces a multi-scale aggregation scheme, leveraging distance adjacency matrices to measure the degree of dependency between two arbitrary skeleton joints. [Li et al., 2019] presents a method to adaptively learn the non-physical dependencies among joints by an encoder-decoder structure, which automatically infers the link strengths between joints.

Upon investigating and experimenting on the released code of state-of-the-art methods [Chen et al., 2021a; Yan et al., 2018; Chen et al., 2021b], we empirically observe that: First, current approaches typically revolve around employing lower-order features such as joint coordinates or velocities in the Euclidean space. Despite their simplicity, these features might not be favourable for describing subtle motion details such as joint angular acceleration and motion trends. Figure 1 illustrates two similar actions, "drinking water" and "brushing teeth". It is evident to see that the joint coordinates of the two actions are very close to each other, while the velocity features of the two actions are also very similar. However, the joint angular accelerations of the two actions are quite different. Second, most existing approaches attend to each channel equally, dissevering the fact that different channels contribute unequally in recognizing an action. For instance, in identifying the action "drinking water", vertical movements (z-dimension) of the hand joint play a more vital role than horizontal movements (xy-dimensions). Apparently, it is not a good idea to assign equal weight to each dimension. Third, current approaches typically employ the conventional CE (Cross-Entropy) loss to supervise the learning of motion information, lacking an effective constraint on guaranteeing that task-relevant features are extracted.

In this paper, we embrace three key designs to tackle the challenges. Technically, (1) We consider incorporating angular accelerations as complementary information and extract them within the framework of rigid body kinematics. These higher-order acceleration features provide an additional input stream, on top of the original Euclidean space motion features, potentially improving the completeness and robustness of the model. (2) We introduce a novel GCN network equipped with multiple input streams and channel-wise attention, where different input streams are assembled while attention weights capitalize on those important channels. (3) We further engage an information-theoretic objective between the extracted deep features and action labels. Maximizing this mutual information objective drives our model to fully mine task-relevant information while reducing task-irrelevant nuisances.

Thereafter, we conduct extensive experiments on three large benchmark datasets including NTU RGB+D, NTU RGB+D 120, and NW-UCLA. Empirically, our approach consistently and significantly outperforms state-of-the-art methods. Interestingly, we observe that existing methods indeed have difficulties in distinguishing between similar actions.

To summarize, our key contributions are as follows: 1) A new motion representation is proposed, which enriches the lower-order motion representations with higher-order motion representations. 2) A multi-stream graph convolution network is presented to comprehensively extract knowledge frommultiple representations. Within the framework, the distinct weights of different channels are considered. To explicitly supervise the knowledge extraction from the input motion sequence, we theoretically analyzed the mutual information between the extracted deep feature and the label, arriving at a loss that maximizes the task-relevant information. 3) Our method achieves the new state-of-the-art performance on three benchmark datasets and overall provides interesting insights. The implementations are released, hoping to facilitate future research.

2 Method

Broadly, a human skeleton can be cast as a set of joints and bones. We refer to the set of joints as a set of nodes V = {vi}ni=1 . Further, we model the set of bones, each of which connects a pair of joints, as a set of edges E = {ei}n×n i=1 . Thus, we conveniently depict a human skeleton pose as a graph G = (V, E), where n = |V| and E ⊂ V × V.

Problem. Presented with an observed 3D skeleton motion sequence S = ⟨G1, G2, · · · , Gm⟩, where Gi is the skeleton pose of the human in the ith frame, we are interested in predicting its action class label y ∈ R. Put differently, we seek to learn the non-linear mapping from a motion sequence S to its action class label y.

Method overview. Overall, our method consists of three key components: a) motion representations engaging both lower-order and higher-order motion features, b) a novel Stream-GCN network that is equipped with channel-wise attention, and is able to integrate higher-order and lower-order representations, and c) a mutual information objective that effectuates feature-level supervision for maximizing the extraction of task-relevant information. Specifically, we first leverage multiple motion representations to frame the motion context. Then, the motion sequences are fed into the proposed Stream-GCN network to predict the action label. Within the network, we introduce a mutual information objective to supervise the prediction. We would like to highlight that our approach has an edge in engaging acceleration to capture subtle motion details, while our network games in maximizing the extraction of task-relevant information and getting rid of task-irrelevant nuisances. In what follows, we elaborate on the three key components in detail.

2.1 Motion Representation

To represent human motion, there have been efforts to outline a pose as 3D coordinates of all skeletal joints [Liu et al., 2019b]. The 3D joint coordinates offer an intuitive and direct way to precisely sketch the motion. Unfortunately, such a representation regards different joints as independent entities and is unfavourable for describing higher-order subtle motion details such as bone rotation trends [Shi et al., 2019a].

Motivated by this, we incorporate both higher-order and lower-order motion representations. For lower-order representations, we include joint coordinates, bone lengths, joint velocity, and bone velocity. For higher-order representations, we engage in joint and bone angular acceleration representations. These different representations (i.e., streams) supplement each other towards a more complete and precise motion characterization. Below, we formulate the joint (or bone) angular acceleration representation.

The articulate human skeleton can be considered as rigid bodies (i.e. bones) connected by joints. The motion of a joint in relation to its neighbouring joints can therefore be characterized in the framework of rigid body kinematics. Within this framework, the angular acceleration of a joint has a significant role. Since the acceleration of an object depends directly on the force acting on it, the acceleration change actually characterizes the changes of forces exerted upon joints, which serve as the driven source for motion. Take drinking water as an example, conventional methods merely consider the position displacement of the raised arm. They omit the acceleration change of the arm in the process of lifting the hand, moving upwards with a constant speed, and decelerating when approaching the mouth.

Instantaneous angle. We denote vp i as the coordinate of the i-th joint at the p-th frame. Supposing we are to calculate the acceleration feature of vp i . We set δp i = vp i - vp i-1 as an axis vector from joint vp i-1 to joint vp i . Upon this, we define the instantaneous absolute angle as follows [Aslanov and Ledkov, 2012]:

where θp i-1 is the instantaneous convected rotation angle of joint vp i-1 w.r.t. the fixed coordinate system, and ˜ θp i is the instantaneous relative rotation angle as opposed to joint vp i-1 .

Instantaneous angular velocity. Further, we obtain the instantaneous angular velocity ωp i of joint vp i by differentiating Equation (1) w.r.t. t:

Instantaneous angular acceleration. To facilitate understanding, we visualize instantaneous angle θp i and instantaneous angular velocity ωp i .Subsequently, we acquire the angular acceleration from the instantaneous angular velocity. Mathematically, we formulate the angular acceleration of joint vp i as εpi . Naively, we could obtain the acceleration by differentiating Equation (2) w.r.t. t, yielding:

where εpi measures the angular acceleration of joint vp i . It is evident to see that the derivation of the above process also applies to computing the angular acceleration of bones.

2.2 Stream-GCNwithMulti-stream Representation Fusion and Channel Attention

Up to this point, we have presented how to obtain higherorder motion representations. Next, we introduce our StreamGCN (GCN with multi-stream representation fusion and channel attention) network that extracts cues from higherorder and lower-order motion representations to recognize an action. The overall architecture of the network is shown in Figure 2. The network absorbs six input streams: four lowerorder motion representations (joint coordinates, bone lengths, joint velocity, and bone velocity) and two higher-order motion representations (joint and bone angular accelerations). Each input stream is processed by a Stream-GCN network to predict an action class distribution. Their class distributions are then ensembled to give the final class distribution.

Stream-GCN network architecture. Typically, the operation of a GCN layer is given by:

where Xr ∈ Rn×lr and Xr+1 ∈ Rn×lr are the features of rth and r+1th layers, respectively. n is the number of nodes in the graph, which translates to the number of joints in this task. lr is the length of joint features at the rth layer. σ(·) is an activation function, e.g., ReLU. Matrix M r ∈ Rlr×lr+1 is a network parameter (transformation matrix). Filter matrix ˆ A is computed based on ˆ A = ˜D-1/2 ˜ A ˜D-1/2 , where ˜ A = A + I and adjacency matrix A ∈ Rn×n characterizes the connections between joints. In the adjacency matrix A, Ai,j = 1 if there exist a bone connects the ith and j th joints, and Ai,j = 0 otherwise. ˜D ∈ Rn×n is a degree matrix and ˜Di,i = ∑j ˜ Ai,j .

Conventional methods such as [Shi et al., 2019a; Chen et al., 2021a] directly adopt equal channel weights for all joints and bones in each frame. This scheme fails to attend to the distinct roles of different channels in recognizing an action. Motivated by this, we propose a cross-channel attention module to adaptively assign weights to different channels. Formally, in our settings, the input feature is X ∈ Rn×c×m, where c is the number of channels and m is the number of frames in the given motion sequence. Put differently, the initial feature length (at the 0th layer) l0 = c ×m.

Different from Equation (4), upon the rth layer, we apply one layer graph convolution as below:

where τ(·) is a cross-channel attention function that boosts the synergy of different channels. More specifically, to compute the weights of different channels, τ (·) consists of a set of pooling, convolution, and subtraction operations, formulated in Equations (6)-(9).

Formally, we first apply linear transformation L, which is a 1×1 convolution, toXr , transformingXr fromRn×cr×m to Rn×c′r ×m. In favor of better aggregating spatial features, we apply a global pooling operation Ψ(·) in the temporal domain on features Xr to sketch its spatial features:

where ˜Xr ∈ Rn×c′r is a set of joint spatial features, namely ˜Xr = {˜xr1 , ˜xr2 , . . . , ˜xrn } and ˜xri ∈ Rc′r .

Next, we explore to capture the channel interaction across joint spatial features. As shown in Figure 2, given a pair of joints (vi, vj) and their corresponding joint spatial features (˜xri , ˜xrj ). We design a correlation modeling function Z(·) that aggregates channel-wise features between vi and vj [Chen et al., 2021a]:

To better exploit the contribution of each channel, we leverage a global pooling operation Φ(·) on Z(˜xri , ˜xrj ) to obtain a channel feature Φ(Z(˜xri , ˜xrj )) ∈ Rc′r . Thereafter, we employ attention mechanism [Wang et al., 2019] on the channel feature to quantify the contributions of different channels in action recognition:

where ϑ is an activation function, and F is a 1D convolution with kernel size of k. In summary, cross-channel attention function τ(·) has the form:

To model the action in the temporal domain, we design a multi-scale temporal modeling that follows [Liu et al., 2020]. As shown in Figure 2, the module consists of six parallel temporal convolutional branches. Each branch starts with a 1×1 convolution to aggregate features between different channels. The first four branches contain a 3 × 1 temporal convolution with four different dilations to obtain multi-scale temporal fields. The Maxpool of the fifth branch is applied to emphasize the most significant feature information among consecutive frames. The last 1 × 1 convolution is added to the residual-preserving gradient during backpropagation.

2.3 Mutual Information Objective

We can certainly train the Stream-GCN with CE loss, as is done in most previous methods [Ye et al., 2020]. Given our systematic examination of extracting motion features for action recognition, it would be fruitful to investigate whether introducing a novel supervision method at the feature level would facilitate the task.

Let Z denote the extracted deep features from input X. Inspired by [Liu et al., 2022; Tian et al., 2021], we try to enforce the correlation between extracted deep features Z and action label Y , seeking to maximize the extraction of taskrelevant information and get rid of task-irrelevant nuisances.

Mutual information (MI). MI measures the amount of information shared between two random variables. Formally, MI between Z and Y quantifies the statistical dependency of encoding variables Z and action label Y :

where p(Y,Z) is the probability distribution between Z and Y , while p(Z) and p(Y ) are their marginals [Liu et al., 2022].

Mutual information loss. In order to ensure that the maximum amount of information about Y is extracted to the deep feature Z, while reducing the information that is not relevant to Y in Z, our primary objective can be formulated as:

Due to the notorious difficulty of the conditional MI computations especially in neural networks [Tian et al., 2021], we perform a simplification. We factorize Equation (11) as follows:

The first term forces features to be informative about the label Y . The second term compresses Z to be concise. The third term constraints Z to include task-relevant information in X. These MI terms are depicted in Figure 3 and can be estimated by existing MI estimators [Tian et al., 2021]. In our experiments, we employ the Variational Information Bottleneck (VIB) [Alemi et al., 2016] and Variational Self-Distillation (VSD) [Tian et al., 2021] to estimate each MI term.

3 Experiments

In this section, we conduct extensive experiments to empirically evaluate our method on three benchmark action recognition datasets. Broadly, we intend to answer the following research questions:

• RQ1: How does the proposed method compare against the state-of-the-art approaches for skeleton-based action recognition?
• RQ2: How much do different components of StreamGCN contribute to its performance?
• RQ3: What interesting insights and findings can we obtain from the empirical results?

Next, we first present the experimental settings, followed by answering the above research questions one by one.

3.1 Experimental Settings

Datasets. We adopt three widely used action recognition datasets, namely NTU-RGB+D, NTU-RGB+D 120, and Northwestern-UCLA, to evaluate the proposed method.

NTU-RGB+D. NTU-RGB+D [Shahroudy et al., 2016] is tailored for the skeleton-based action recognition task. It contains 56,880 video samples from 60 action classes performed by 40 volunteers. Each sample contains one action and is guaranteed to have a maximum of two subjects, which is captured by three Microsoft Kinect v2 cameras. This dataset provides two sub-benchmarks: (1) Cross-Subject (XSub): data for 20 subjects is used as the training data, while the rest is used as test data. (2) Cross-View (X-View) divides the training and test sets according to different camera views.

NTU-RGB+D 120. NTU-RGB+D 120 [Liu et al., 2019a] is the current largest human action recognition dataset, which extends NTU-RGB+D with an additional 60 action classes and 57,600 video samples. It contains a total of 114,480 samples from 120 action classes, which are executed by 106 volunteers and captured by three Kinect cameras. Within the dataset, two benchmarks are maintained: (1) Cross-Subject (X-Sub), which categorizes 53 subjects into the training class and the other 53 subjects into the test class. (2) Cross-Setup (X-Set), which arranges data items with even IDs into the training group and odd IDs into the test group.

Northwestern-UCLA. The Northwestern-UCLA dataset contains 1,494 video samples in 10 classes [Wang et al., 2014]. These videos were collected by filming ten actors with three Kinect cameras. We follow the evaluation protocol mentioned in [Wang et al., 2014], where videos collected by the first two cameras serve as the training samples and the rest serve as test samples.

Implementation details. We conduct experiments on a computer equipped with an Intel Xeon E5 CPU at 2.1GHz, three NVIDIA GeForce GTX 1080 Ti GPUs, and the RAM of 64GB. We leverage PyTorch 1.1 to implement our model. We apply pre-processing method of [Zhang et al., 2020] to the data in NTU-RGB+D and NTU-RGB+D 120. Moreover, we initialize the data in Northwestern-UCLA according to the approach of [Cheng et al., 2020b]. We apply stochastic gradient descent (SGD) with 0.9 Nesterov momentum to train the Stream-GCN model. For NTU-RGB+D and NTU-RGB+D 120 datasets, the number of training epochs is set to 65 with the first 5 epochs being warm-up epochs, which help stabilize the training process. For NTU-RGB+D and NTU-RGB+D

120 datasets, the initial learning rate is set to 0.1 and decays by 0.1 every 35 epochs, the batch size is selected as 64. For the Northwestern-UCLA dataset, the initial learning rate is set to 0.01 and decays by 0.0001 every 50 epochs, the batch size is set to 16.

3.2 Comparison with ExistingMethods (RQ1)

We first empirically compare the proposed model with the state-of-the-art methods. The experimental results are summarized in Tables 1-3. Table 1 and Table 2 present results on NTU RGB+D 120 and NTU RGB+D datasets, while Table 3 illustrates results on the Northwestern-UCLA dataset. From the tables, we have the following observations. (a) The proposed method consistently outperforms state-of-the-art approaches on all three benchmark datasets. For instance, on the NTU120 RGB+D dataset, state-of-the-art method [Song et al., 2022] achieves a 88.3% action recognition accuracy. In contrast, Stream-GCN obtains a 89.7% accuracy. (b) Generally, GCN based approaches (such as [Shi et al., 2019a; Shi et al., 2019b; Liu et al., 2020]) perform better than RNN based methods (such as [Liu et al., 2016]). The empirical evidence suggests that GCN models are more suitable for action recognition tasks as they may conveniently model the graph structure of the human body. (c) Another finding is that multi-stream models usually outperform singlestream ones. For example, on the NTU RGB+D 120 dataset, multi-stream models [Cheng et al., 2020b; Song et al., 2022;

Ye et al., 2020] achieve much better results than single stream methods [Zhang et al., 2020; Si et al., 2019]. This reveals the importance of incorporating different representations to approach more precise action recognition.

3.3 Ablation Study (RQ2)

We perform ablation experiments to examine the influence of each component in our method (i.e., Motion Representation, Cross-channel attention Module, and MI Loss). All the ablation studies are conducted on the NTU-RGB+D 120 crosssubject benchmark dataset. We also investigate the impact of modifying the parameter settings of the cross-channel attention module.

Motion representation. To verify the impact of the proposed acceleration representations in modeling higher-order motion features. We tried removing the joint angular acceleration stream and bone angular acceleration stream respectively. Specifically, we first tried removing all acceleration streams. Then, we studied removing only the acceleration stream of the joint. Finally, we also tried removing only the acceleration stream of bone. The quantitative results are demonstrated in Table 4, empirical results reveal that using the high-order representation indeed boosts the action recognition accuracy.

Cross-channel attention module. Next, we are interested in the effect of removing the cross-channel attention module. The empirical results are demonstrated in Table 5. Remarkably, the result in the second line of Table 5 shows that when the cross-channel attention module is removed, the accuracy is decreased (89.7% vs 89.4%).

Mutual information loss. We validate the effect of the MI Loss in Section 2.3. As illustrated in the third and the fourth lines of Table 5, the 0.4% accuracy improvement provides empirical evidence that our proposed MI loss is effective. This reveals that the additional feature-level supervision facilitates the extraction of task-specific information.

Parameter settings. By default, we use cross-channel attention to assign different weights to different channels in recognizing an action. We explore the hyper-parameter settings in the cross-channel attention module with joint stream (i.e., joint coordinates). The key hyper-parameters include the activation function ϑ, the kernel size of the 1D convolution, and the mask on the adjacency matrix. As shown in Table 6, we observe that Stream-GCN obtains the best performance when ϑ is a Sigmoid function, the kernel size K = 1, and the adjacency matrix is without a mask.

3.4 Discussions (RQ3)

Visualize results. We visualize the learned attention maps of different layers (layer 1, layer 5, and layer 9) for the "drinking water" action in Figure 4. From the figure, we see that different layers of the learned attention map contain distinct semantics. Specifically, in the lower layers, it seems that the model focuses on the relations between hands, shoulders, and wrists. This is intuitive since the model will initially focus on the body parts (joints) where the movement takes place. In the intermediate layers, the arm is more concerned by the model, with attention to the other parts gradually diminishing. This demonstrates that the model is analyzing what exactly the arm is doing. In the higher layers, the information is highly aggregated. Meanwhile, the model will combine other joints that are helpful to identify the motion. As an example, for recognizing the action of "drinking water", the correlation between the hands and the head is detected and leveraged by the model. Experiments on three different benchmark datasets suggest that incorporating higherorder motion sequences achieves higher accuracy. Interestingly, a point that attracts our attention is: we empirically observe that as the number of action classes goes higher, although incorporating high-order representation still improves the accuracy, the advantage of incorporating high-order representation decreases. We plan to dive deeper into this phenomenon and come up with new solutions.

4 RelatedWork

Earlier skeleton-based action recognition methods model the motion sequences with convolutional neural networks or recurrent neural networks [Si et al., 2018]. As these conventional methods have inherent difficulties in capturing the connections between joints, they tend to suffer from unsatisfactory results. Recent efforts have embraced GCNs to model the human motion sequences as spatio-temporal graphs [Bian et al., 2021]. [Yan et al., 2018] is a pioneering work in this line of efforts, proposing a ST-GCN model to characterize the skeleton motion as a GCN. [Shi et al., 2019b] proposes an adaptive graph convolution network based on ST-GCN, incorporating skeletal bone features as an additional input stream and adopting self-attention to learn the graph parameters. [Shi et al., 2019a] introduces frame-wise bone and joint velocities as additional input streams for action recognition.

Existing methods tend to have difficulties in discriminating between similar actions. Meanwhile, to our best knowledge, the higher-order motion features and mutual information are rarely explored in skeleton-based action recognition [Tan et al., 2021; Chen et al., 2022]. This inspires us to seek methods to facilitate the model recognizing the action class precisely.

5 Conclusion

In this paper, we have proposed a motion representation derived from rigid body kinematics. The new representation captures the higher-order motion features, complementing conventional motion representations. We also present a Stream-GCN network equipped with multiple input streams and channel-wise attention, where different streams are ensembled while attention weights capitalize on those important channels. Finally, we introduce a mutual information objective for theoretic supervision on extracted features. Extensive experiments show that our method consistently surpasses state-of-the-art methods on three different benchmark datasets. Our code is released to facilitate researchers.

Acknowledgments

This research is supported by the National Natural Science Foundation of China (62276112), the National Natural Science Foundation of China Regional Joint Fund of NSFC (U19A2057), Jilin Province Science and Technology Development Plan Key R&D Project (20230201088GX), Collaborative Innovation Project of Anhui Universities (GXXT-2022044), and partly supported by the Key R&D Program of Zhejiang Province (No. 2023C01217).