Mutually Reinforced Spatio-Temporal Convolutional Tube for Human Action Recognition

1 Introduction

Human action recognition aims to recognize human actions by the visual appearance and motion dynamics of the involved humans and objects in video sequences. It is a fundamental yet challenging task due to the temporal dynamic of video content, low resolution and background interference, etc. Considerable efforts have been investigated for decades, and existing approaches could be roughly divided into two categories: 2D CNN based methods and 3D CNN based methods.

∗Equal contribution

†Corresponding author

The 2D CNN based methods use either Two-stream CNNs [Simonyan and Zisserman, 2014] or CNN-Long ShortTerm Memory (CNN-LSTM) [Donahue et al., 2015]. The Two-stream CNNs use two CNN branches to extract spatial features from RGB frames and temporal features from stacked optical flow, respectively. The Two-stream CNNs capture temporal dependencies with hand-crafted optical flow information, which suffers from expensive computational cost and cannot model the correlation between spatial and temporal information. The CNN-LSTM networks connect a LSTM layer to the bottom of the CNNs. They firstly extract appearance features with CNNs and then model temporal information from the video frames by a LSTM layer in tandem, which loses plenty of useful information.

Jointly modeling spatio-temporal information via 3D CNNs is a natural and efficient approach for human action recognition. However, the basic 3D CNN (C3D) [Tran et al., 2015] suffers from the high computational complexity of 3D convolution operation. Some works [Tran et al., 2018; Xie et al., 2018; Zhou et al., 2018] attempt to design improved architectures with the basic 3D convolution. Nevertheless, these methods still suffer from high computational cost using 3D convolutional operation. Other works [Sun et al., 2015; Qiu et al., 2017] decompose the 3D convolution into two separate convolutions, i.e., a 2D spatial convolution plus a 1D temporal convolution and thus significantly reduce the model size. However, these methods ignore the correlation between spatial and temporal information. Actually, the joint exploration of them could offer a comprehensive representation of videos and thus enhance the accuracy of action recognition. Human actions in videos contain both appearance information and motion information. For example, in a human playing basketball video, exploring the spatial and temporal information jointly can pay more attention to the human and ball's appearance information rather than the surroundings' since the human and ball are in continuous movement.

In this paper, we propose a Mutually Reinforced SpatioTemporal Convolutional Tube (MRST) towards robust and accurate human action recognition. It factorizes the 3D inputs into spatial and temporal representations, mutually enhances both of them by exploiting the interaction of spatial and temporal information, and fuses the enhanced representations to obtain effective spatio-temporal features, while reducing the complexity. Specifically, the MRST consists of a spatiotemporal decomposition unit, a mutually reinforced unit and a spatio-temporal fusion unit. An illustration of our MRST is shown in Fig.1. The spatio-temporal decomposition unit extracts spatial and temporal features with a 2D convolution and a 1D convolution, respectively. The mutually reinforced unit learns the correlation between spatial and temporal information by four fully connected layers, and utilizes the correlation to reinforce the discriminative capability of the spatial features and temporal features. The spatio-temporal fusion unit selectively emphasizes informative spatial and temporal features by a global pooling layer and a sigmoid layer and fuses them to get the effective spatio-temporal feature maps. In addition, we design three types of MRSTs according to the different order of spatial and temporal information enhancement, i.e., MRST-P, MRST-S and MRST-T. A novel deep network, MRST-Net, is also proposed based on the MRST to better explore spatio-temporal information in human actions. Experiment results show our proposed deep MRSTNet achieves state-of-the-art performance on three challenging action recognition datasets, Kinetics-400, UCF-101 and HMDB51 with only RGB inputs.

2 RelatedWork

Human action recognition is one of the core computer vision tasks and has been studied for decades. Here we outline work involving deep features and classify them into two categories: 2D CNN and 3D CNN based approaches, according to the convolutions used in feature learning.

2D CNN based. To explore the spatio-temporal information in human actions, the two-stream architecture is first proposed in [Simonyan and Zisserman, 2014] where two 2D CNNs are applied to the appearance (RGB frames) and motion (stacked optical flow) domains, respectively. Features from the two modalities are fused at the final stage and the two-stream architecture achieves high video recognition accuracy. Based on this architecture, several mechanisms are proposed for a better fusion of the two branch networks over the appearance and motion [Karpathy et al., 2014; Feichtenhofer et al., 2017; Tran and Cheong, 2017; Wang et al., 2018a]. On the other hand, early attempts which incorporate LSTM with traditional features have shown the potential of the CNN-LSTM architecture for modeling spatio-temporal information in action recognition [Donahue et al., 2015]. LSTM networks are employed to utilize the frame-level features of 2D CNNs to explicitly model spatio-temporal relationships. The work[Hu et al., 2018] extracts different channels' attention to further improve the quality of representations produced by a network.

3DCNN based. The 3DCNN for action recognition is first presented in [Ji et al., 2013] to learn discriminative features along both spatial and temporal dimensions. Later, the C3D feature along with the corresponding 3D CNN architectures are presented in [Tran et al., 2015]. Since they use 3D convolution kernels to model both spatial and temporal information rather than 2D kernels which just model spatial information, more complex relations between appearance and motion can be learned and captured. The Res3D[Tran et al., 2017] makes one step further by taking the advantage of residual connections to ease the learning process. Similarly, I3D[Carreira and Zisserman, 2017] is proposed to use the Inception network[Szegedy et al., 2015] as the backbone network rather than residual networks to learn video representations. Recently, there are many frameworks proposed to improve 3D convolution[Tran et al., 2018; Xie et al., 2018; Zhou et al., 2018; He et al., 2018]. However, all of the methods still suffer from an order of magnitude more computational cost than their 2D competitors due to the newly added temporal dimension, which makes the models difficult to train and unpractical in real-world applications. To decrease the number of parameters, these works[Sun et al., 2015; Qiu et al., 2017] decompose a 3D convolution kernel into a combination of a 2D spatial kernel and a 1D temporal kernel.

3 MRSTs and DeepMRST Network

In this section, we first give a detailed description of the three types of MRSTs. Each MRST consists of a spatio-temporal decomposition (STD) unit, a mutually reinforced unit and a spatio-temporal fusion (STF) unit. We then present our robust and efficient deep network, MRST-Net for human action recognition.

3.1 MRSTs

We design the three types of MRSTs according to the different order of spatial and temporal information enhancement, as shown in Fig.2. Each MRST consists of a spatio-temporal decomposition (STD) unit, a mutually reinforced unit and a spatio-temporal fusion (STF) unit. The STD unit aims to decompose the 3D input signal into spatial features and temporal features with a 2D convolution and a 1D convolution. The mutually reinforced unit learns the correlation between spatial and temporal features by four fully connected layers, and utilizes the correlation to mutually enhance the discriminative capability of the spatial features and temporal features. The STF unit selectively emphasizes informative spatial and temporal features and suppresses useless ones and fuses them together to obtain the efficient spatio-temporal features.

Spatio-Temporal Decomposition Unit

The spatio-temporal decomposition (STD) unit is designed to reduce the high computational complexity of 3D convolution.

It decomposes the 3D convolutional kernel into a 1 × 3 × 3 convolutional kernel and a 3 × 1 × 1 convolutional kernel. The 1×3×3 convolutional kernels are used to extract spatial features S and the 3×1×1 convolutional kernels are used to extract temporal features T. The number of a STD unit's parameters is an order of magnitude less than that of an original 3D convolutional kernel, which could reduce the model size enormously. We additionally employ two 1 × 1 × 1 convolutions at both ends of the path, which are applied to reduce and restore the channel dimensions respectively, in order to decrease the overall computational cost of MRSTs.

Mutually Reinforced Unit

The mutually reinforced unit is designed to generate the reinforced spatial and temporal representation, consisting of four fully connected layers. It takes the primitive spatial features S

and temporal features T as input and produces the reinforced spatial information ˜S and temporal information ˜T. As shown in Fig.3, we find that the primitive spatial features mainly concentrate on the objects with surrounds, such as the human appearance and the baluster. Besides, the primitive temporal features mainly concentrate on the objects which are in continuous motion, such as the human's arms and the ball. Based on the above observation, we develop a mutually reinforced unit to use the primitive spatial and temporal features to generate reinforced spatial and temporal representation. With the mutually reinforced unit, for spatial features, we can pay more attention to the objects which are in continuous motion. As for the temporal features, we can concentrate more on objects' surrounds. Specially, in Fig.3, we reinforce spatial features by suppressing useless features, which are irrelevant to sports, such as the baluster. In the meantime, we reinforce temporal features by learning the appearance of motion, such as the human's arms and the ball.

For a 3D input signal X, the primitive spatial features S and temporal features T generated by the STD unit have the same size L×H ×W ×C, where L,H,W,C refers to the length, height, width and the number of channels, respectively. We propose three different MRST mutually reinforced units which differ in the order of getting the reinforced spatial and temporal information, named as MRST-P unit, MRST-S unit and MRST-T unit, respectively. Detailed structures are provided as following:

(1) MRST-P unit: As shown in Fig.2(a), the MRST-P unit adopts a parallel structure, which reinforces the spatial and temporal features simultaneously. It consists of four fully connected networks. The temporal-to-spatial (TS) fully connected network is designed to learn the temporal features' guiding effect on spatial features. The spatial-to-spatial (SS) fully connected network learns the impact of spatial features on itself. Then the output of TS fully connected network and SS fully connected network are added together as the reinforced spatial information ˜S. Similarly, the spatial-totemporal (ST) fully connected network and the temporal-totemporal (TT) fully connected network are designed to learn the spatial features' guiding effect and temporal features' impact on the temporal features, respectively. The reinforced temporal information ˜T are obtained by adding the output of ST and TT fully connected layer. The formulation of obtaining the reinforced information is shown as follows:

where WSS, WST , WTS and WTT refer to the parameters of the four fully connected layers, respectively.

(2) MRST-S unit: As shown in Fig.2(b), the MRST-S unit employs a series structure, consisting of four fully connected layers. First of all, a ST fully connected network and a TT fully connected network are designed to learn the spatial features' guiding effect and temporal features' impact on the temporal features, respectively, for producing the reinforced temporal information ˜T. After that, the reinforced spatial information ˜S are obtained based on the primitive spatial features S and the reinforced temporal information ˜T. The formulation of the MRST-S unit is shown as follows:

where δ refers to the ReLU function.

(3) MRST-T unit: As shown in Fig.2(c), the MRST-T unit utilizes a series structure and consists of four fully connected layers. Firstly, a TS fully connected network and a SS fully connected network are designed to learn the temporal features' guiding effect and spatial features' impact on the spatial features, respectively, for producing the reinforced spatial information ˜S. Afterwards, the reinforced temporal information ˜T is obtained based on the primitive temporal features T and the reinforced spatial information ˜S. The formulation of the MRST-T unit is is shown as follows:

Spatio-Temporal Fusion Unit

The spatio-temporal fusion (STF) unit aims to selectively emphasize informative spatial and temporal features, which can be interpreted as a means of biasing the allocation of available computational resources towards the most informative components of features, and fuses them to obtain effective spatio-temporal features.

For any given transformation U = Ftr(X), where X is a three-dimensional input and Ftr refers to a 3D convolutional operator. We suppose U ∈ RL×H×W×C , and we can write it as U = [u1, u2, . . . , uC], where uc refers to the parameters of the c-th channel in U. We first use the global average pooling operation to generate channel-wise statistics. Formally, a statistic z ∈ RC is generated by shrinking U through its spatio-temporal dimensions L ×H ×W, such that the c-th element of z is calculated by:

To ensure that multiple channels are allowed to be emphasised, we follow the global pooling operation with a sigmoid function.

where σ refers to the sigmoid function. The final output of the STF unit X∗ is obtained by rescaling the transformation output U with the activation:

where X∗ = [x∗1 , x∗2 , . . . , x∗C ], and the scale function refers to channel-wise multiplication between the scalar sc and the feature map uc. Particularly, in our MRSTs, the STF unit takes the spatial features S, the temporal features T, the reinforced spatial information ˜S and the reinforced temporal information ˜T as inputs and the final output can be calculated by:

where σ refers to the sigmoid function, and Fgp represents the global average pooling operation, which is shown in Eq.4.

3.2 DeepMRST Network

We propose a simple yet efficient MRST-Net by stacking the MRST blocks. The proposed MRST-Net contains an initial convolutional block, five residual convolutional blocks and a LSTM [Hochreiter and Schmidhuber, 1997] layer. The initial convolutional block consists of six convolutional layers, which extracts primitive features. Each of the five residual convolutional blocks contains several MRSTs with Batch Normalization (BN) and Rectified Linear Units (ReLU). The main idea of the residual convolutional block is to learn the additive residual function with reference to the unit inputs which is realized through a shortcut connection, instead of directly learning unreferenced non-linear functions [He et al., 2016]. Moreover, the Max-Pooling operation is performed with the initial convolutional block and each residual convolutional block, reducing the size of the feature maps (All the Max-Pooling layers' kernel size are 1 × 2 × 2 and strides are (1,2,2)). We also add a Max-Pooling layer and a reshape operation after the conv6 x residual block to get the LSTM input. The LSTM layer progressively takes each time's feature output from the last residual block as input and decides whether to retain or discard the features from the current time and previous ones. Finally, a fully connected layer is designed to predict the classification of the input video sequence. More details of the network architecture are provided in Table 1.

4 Experiments

4.1 Datasets and Implementation Details

Datasets. Three well-known benchmarks, i.e., Kinetics400[Kay et al., 2017], UCF-101[Soomro et al., 2012], and HMDB-51[Kuehne et al., 2013], are included in the evaluations. The large-scale Kinetics-400 benchmark dataset consists of approximately 300, 000 videos from 400 action categories. UCF-101 consists of 13,320 manually labelled videos from 101 action categories. HMDB51 is collected from various sources, e.g. web videos and movies, which proves to be realistic and challenging. It consists of 6,766 manually labelled clips from 51 categories. Both UCF101 and HMDB51 are provided with 3 splits for training and testing and we report the accuracy by averaging over all 3 splits.

Implementation details. Our data augmentation includes random clipping on both spatial (firstly resizing the smaller video side to 256 pixels, then randomly cropping a 224×224 patch) and temporal (randomly picking the starting frame among those early enough to guarantee a desired number of frames). We average clip predictions which are uniformly sampled from the long video sequence to obtain the video predictions. Batch normalization is applied to all convolutional layers. We use the Adam Gradient Descent optimizer with an initial learning rate of 1e-4 to train the MRST-related networks from scratch. The drop out ratio and weight decay rate are set to 0.5 and 5e-5 . We set the initial learning rate is 1e-5 and utilize the gradient descent optimizer with a momentum of 0.9 to train our MRST-related networks initialized with the ImageNet-1k and Kinetics-400 pre-trained model.

We employ the higher drop out ratio of 0.9 and the weight decay rate of 5e-4 to prevent over-fitting.

4.2 Ablation Study

To demonstrate the effectiveness of each component of the proposed MRSTs, we conduct a series of ablation experiments on UCF-101 and HMDB51 datasets. We also make a comparison with C3D[Tran et al., 2015], the baseline 3D CNN. We choose C3D as our baseline since it is the most direct way to demonstrate the effectiveness of our final MRSTs. For fairness, all architectures use the same backbone (with 8 convolutional layers, 5 max-pooling layers, and 2 fully connected layers) and the same input 16 × 3 × 224 × 224 (16 is the number of frames in each input clip). Moreover, they are all pre-trained on the Kinetics-400 datasets.

Table 2 shows the comparison results in terms of the Top-1 classification accuracy and the model size on both UCF-101 and HMDB51 datasets. From Table 2, we can observe that our three MRSTs significantly outperform the baseline approach C3D. We also find that compared to the STD unit, the STD unit with STF improves accuracy by 1.9% on UCF-101 and 3% on HMDB51, which can prove the efficiency of attention recalibration. In addition, all the threeMRSTs outperform the STD (with STF) unit, which strongly demonstrates the effectiveness of the mutually reinforced unit. Meanwhile, we find that among all three MRSTs, the MRST-T yields the best performance. Compared to the MRST-P, the MRST-T increases accuracy by 1% and 1.7% on UCF-101 and HMDB51 respectively. And compared to the MRST-S, the MRST-T improves accuracy by 1.4% on UCF-101 and 1.7% on HMDB51. We conjecture that the human action recognition task relies more on temporal features, namely the objects which are in continuous motion. Moreover, some objects with surround in spatial features may interfere with extracting effective attentions. As for some tasks rely more on spatial features, such as the video person re-identification task, the MRST-S may get better performance than the others. We also evaluate the model size in all the six architectures. It is clear that the model size of MRST is much smaller than the C3D and is close to the STD model size, which proves that our MRSTs consume less computational resources.

4.3 Comparison to the State-of-the-Art Methods

We further demonstrate the advances of our proposed MRSTNet in comparison with state-of-the-art methods for action recognition. All methods use only RGB input. Based on the results in section 4.2, we use the MRST-T (which gets the best performance among three MRSTs) and ResNet101∗ (compared to the base ResNet101, we add a spatial pooling layer and change the channel numbers of some layers, as shown in Table 1.) structure as backbone. In order to control the number of parameters of the whole network, we use the input clip size as 16 × 3 × 224 × 224. We uniformly sample 20 clips per video and average these 20 clip predictions to obtain the video prediction. Related results on Kinetics-400, UCF101 and HMDB51 are shown in Tables 3 and 4, respectively.

Results on Kinetics-400. Table 3 shows the performance comparison of our proposed MRST-T-Net (pre-trained on ImageNet-1k) against nine state-of-the-art methods in terms of Kinetics-400 Top-1 classification accuracy and the total number of parameters and the FLOPs. The compared methods use only RGB inputs and they have almost the same backbone. The proposed MRST-T-Net achieves 74.1% Top-1 classification accuracy, and the total number of parameters is 31.7M, and the FLOPs is 99.6G. We can see that our method surpasses existing methods, improving the 2nd best compared method MF-Net by 1.3% at Top-1 classification accuracy. Moreover, our MRST-T-Net achieves significant performance improvement compared to the baseline C3D by 18% at Top-1 classification accuracy. Besides, the total number of parameters of our proposed MRST-T-Net is no more than half of the C3D. Compared to the same backbone structure ResNet101-StNet[He et al., 2018], our network has fewer parameters. We can also see that the computational cost (FLOPs) of MRST-T-Net is lower than those of most existing 3D CNN based methods. The comparisons indicate that the MRST-T-Net can learn and represent spatio-temporal features much more efficiently and accurately than other methods.

Results on UCF-101 and HMDB51. We also experiment with fine-tuning MRST-T-Net (pre-trained on ImageNet-1k and Kinetics-400) on UCF-101 and HMDB51 to evaluate the generality and robustness. From Table 4, we can observe that the proposed MRST-T-Net outperforms almost all the existing state-of-the-art methods with only RGB inputs on both UCF-101 and HMDB51, obtaining 96.5% Top-1 classification accuracy on UCF-101 and 75.4% Top-1 classification accuracy on HMDB51. In addition, MRST-T-Net boosts the baseline method C3D by 14.2% at Top-1 accuracy on UCF101 and 23.8% at Top-1 accuracy on HMDB51. The only method that has better performance on UCF-101 than ours is the R(2+1)D-34 layers network, but our proposed model require fewer parameters and less computational cost.

5 Conclusion

In this work, we propose a mutually reinforced SpatioTemporal Convolutional Tube (MRST) for human action recognition. It decomposes 3D inputs into spatial and temporal representations, mutually enhances both of them by exploiting the interaction of spatial and temporal information, and fuse them for extracting effective spatio-temporal features, while reducing the computational complexity. Experiment results show that the MRST-T yields the best performance among three MRSTs and significantly outperforms traditional 3D CNNs for action recognition. Moreover, the MRST-T-Net achieves the best performance on three datasets, Kinetics-400, UCF-101 and HMDB51 in comparison to stateof-the-art approaches, indicating that the proposed MRST-TNet is general and efficient.

Acknowledgments

This work was supported by the National Key R&D Program of China under Grant 2017YFB1300201, the National Natural Science Foundation of China (NSFC) under Grants 61622211 and 61620106009 as well as the Fundamental Research Funds for the Central Universities under Grant WK2100100030.