Representation Learning for Compressed Video Action Recognition via Attentive Cross-modal Interaction with Motion Enhancement

1 Introduction

Along with the development of web applications and innovation of imaging sensors, the amount of videos is explosively growing, and automatic analysis of video content has become extremely important. Action recognition is a fundamental topic and has recently received increasing attention within the community. The past two decades have witnessed the advances successively by hand-crafted features and deep Convolutional Neural Networks (CNNs). The majority of the efforts are made to capture both static and dynamic cues from raw videos with all RGB frames decoded [Wang et al., 2020], and report excellent performance on the public benchmarks

* Corresponding author.

(e.g. UCF-101 and HMDB-51). However, they generally require optical flows to represent motions, thus suffering from the drawbacks of high storage consumption and slow processing speed, and hardly meet the real-world demands.

More recently, action recognition in compressed videos is investigated as a promising alternative to handle the challenges above. In compressed videos, only a small proportion of frames are fully decoded, known as intra-frames (Iframes), and most frames are incompletely decoded, known as predicted frames (P-frames), which convey free but coarse motion information. I-frames, motion vectors and residuals are usually employed in compressed video based methods to replace decoded RGB frames and computed optical flows, where two major issues are critical, i.e. motion enhancement and multi-modal fusion.

On the one hand, those motion patterns recorded in compressed videos are not so accurate as those in raw videos as shown in Fig. 1, because the spatial resolution of the motion vector is substantially reduced (i.e. 16×) by block-wise matching and noise is inevitably induced by operations such as signal quantization and motion estimation in the encoding phase. For motion enhancement, taking optical flows as templates, [Shou et al., 2019] applies Generative Adversarial Networks (GAN) to refine motion vectors, and [Cao et al., 2019] removes unreliable movements in motion vectors according to a fixed threshold. Such techniques indeed contribute, but they update motion vectors or residuals in the input side and process them by the models designed for raw videos, which are not fully optimized in an end-to-end way, limiting further performance gains.

† The paper with Supplementary Material is available at https://arxiv.org/abs/2205.03569.

On the other hand, to build comprehensive representation, it is required to integrate the information provided by the multiple modalities in compressed videos, i.e. I-frames and Pframes (motion vectors and residuals). Current studies do not pay adequate attention to this step and generally apply simple early- or late-fusion as on raw videos. Unfortunately, Iframes and P-frames are produced at varying sampling rates, and such an inconsistency tends to impair the complementary among modalities. [Huo et al., 2019] designs Temporal Trilinear Pooling to induce temporal contexts, which makes use of feature ensemble on corresponding frames of different modalities as well as another following I-frame for prediction. Although this method proves effective, it lacks local cross-modal interaction especially in the low-level layers, leaving much room for improvement.

In this paper, we propose a novel approach to action recognition in compressed videos, namely Attentive Cross-modal Interaction Network with Motion Enhancement (MEACINet), which simultaneously addresses the problems of motion enhancement and multi-modal fusion. MEACI-Net follows the classic two-stream framework, for RGB (I-frames) and motion (P-frames) respectively. Considering the coarse and noisy nature of dynamics, we introduce a multi-scale building block in the Compressed Motion Enhancement (CME) network with a denoising module embedded, which guides the model to attend task-relevant spatial regions, resulting in more discriminative features to describe actions. Besides, we present the Selective Motion Complement (SMC) and CrossModality Augment (CMA) modules to learn local and global cross-modal interactions, thus facilitating feature fusion between modalities. We extensively evaluate MEACI-Net on three benchmarks, i.e. HMDB-51, UCF-101 and Kinetics400, and achieve state-of-the-art accuracies of action recognition on compressed videos, significantly reducing the gap to the ones on raw videos.

2 RelatedWork

2.1 Action Recognition in RawVideos

Action recognition has been extensively investigated during the past two decades. Early works employ hand-crafted features [Wang and Schmid, 2013] and the recent ones build deep representations [Simonyan and Zisserman, 2014], both of which are computed from RGB frames. According to the way how dynamics are captured, the deep learning networks are roughly grouped into three categories.

The first group encodes motion cues of RGB frames in optical flows, and 2D CNNs are applied to build stronger representations. The two-stream framework [Simonyan and Zisserman, 2014] is a representative, consisting of one 2D CNN that learns static spatial features and the other 2D CNN that models temporal information in optical flows, which are separately trained and then averaged for prediction. TSN [Wang et al., 2016] enhances this two-stream fashion by sparse sampling and temporal fusion to improve motion features.

The second group represents RGB frames by 2D CNNs and calculates dynamics in the ConvNet feature space. One way is to launch Recurrent Neural Networks. For instance, [YueHei Ng et al., 2015] exploits an independent LSTM network. Another choice is temporal convolution. [Jiang et al., 2019] shifts the features in the temporal dimension to optimize information exchange among neighboring frames, and [Jiang et al., 2019] designs separate modules to span the spatiotemporal feature space for encoding dynamics. [Li et al., 2020b] develops distinct blocks to capture both the short and long range temporal evolutions.

The third group takes RGB frames as input and 3D CNNs are launched to extract unified spatio-temporal features. I3D [Carreira and Zisserman, 2017] inflates pre-trained 2D convolutions to 3D ones. To decrease heavy computations in 3D CNNs, [Tran et al., 2018] decomposes 3D convolutions into 2D spatial convolutions and 1D temporal convolutions or adopts a mixup of 2D/3D CNNs. SlowFast [Feichtenhofer et al., 2019] involves two 3D CNN paths, with the slow one to extract spatial semantics and the fast one to capture finegrained motions.

Despite competitive results, raw video based methods are limited by storage consumption and processing speed.

2.2 Action Recognition in Compressed Videos

Regarding action recognition in compressed data, I-frames (sparsely sampled RGB frames) and P-frames (e.g. motion vectors and residuals) are used for feature extraction instead of all decoded RGB frames and computed optical flows. The early work [Zhang et al., 2016] attempts to replace optical flows with motion vectors for a higher efficiency. CoViAR [Wu et al., 2018] makes an extension by exploiting all the modalities, including I-frames, motion vectors, and residuals to bypass video decoding. To improve the dynamics, [Shou et al., 2019] adopts GAN to refine motion vectors and [Cao et al., 2019] applies image denoising techniques. [Huang et al., 2019] feeds concatenated motion vectors and residuals into a CNN to mimic a flow based teacher, and a similar idea is given by [Battash et al., 2020]. [Huo et al., 2019] presents Temporal Trilinear Pooling to integrate multiple modalities on lightweight models for portable devices. [Li et al., 2020a] follows SlowFast and proposes the Slow-I-Fast-P model, and pseudo optical flows are estimated with a specific loss.

The methods above boost the performance; however, they struggle with modeling coarse and noisy dynamics and fusing distinct modalities, which are addressed in this study.

3 The Proposed Approach

As displayed in Fig. 2, our proposed framework firstly applies encapsulation to extract I-frame and P-frame clips from compressed videos, where P-frames are given in Motion Vectors and Residuals, denoted as the MVR modality. Subsequently, MEACI-Net predicts the categories of actions presented in videos. Concretely, MEACI-Net follows the classic two-stream architecture with the RGB (I-frame) and motion (MVR) streams. The former simply adopts I3D-ResNet50, and the later also introduces the same backbone but extends it as a novel Compressed Motion Enhancement (CME) network to handle coarse and noisy motion cues by employing a Multi-Scale Block (MSB) embedded with a Denoising Module (DM).

Moreover, MEACI-Net presents the Attentive Cross-modal Interaction (ACI) mechanism to strengthen information exchange between the two streams. ACI consists of several Selective Motion Complement (SMC) units in the low-level layers followed by a Cross-Modality Augment (CMA) unit in the high-level layer. SMC enhances representation learning in the RGB modality by incorporating informative motion cues from the MVR modality. CMA further builds a cross-modal representation as an augmentation of the single-modal ones by aggregating multi-modal features via cross attention. The high-level features extracted from the two individual streams together with the one learned by CMA are fused at the score level for final prediction. In the rest part, we elaborate the technical details of the main components.

3.1 Compressed Video Encapsulation

The encoded compressed video is usually packed as multiple Group of Pictures (GOPs), each of which has an I-frame and several consecutive P-frames. The I-frame represents the reference RGB frame in each GOP, and the P-frames generally consist of the motion vectors and its residuals, where motion vector performs a coarse approximation on the movement of the target by encoding the displacement in each 16 × 16 macroblock relative to the I-frame and the residual is the difference between the raw RGB frame and the one reconstructed by the motion vector after motion compensation. Similar to [Wu et al., 2018], we calculate accumulated residuals and motion vectors, which are iterated to I-frames in GOPs. Since motion vectors contain main dynamics and residuals include rich boundaries, we concatenate them as the input to the CME network.

3.2 CompressedMotion Enhancement Network

As shown in Fig. 1, motion vectors are coarse and noisy, making it difficult to learn discriminative temporal features for recognizing actions, especially to those similar ones such as painting lipstick and brushing teeth. It is therefore of critical importance to introduce multi-scale stimuli to explore more useful patterns. In the meantime, the noise in motion cues needs to be suppressed. To that end, inspired by [Li et al., 2020b], we present the Compressed Motion Enhancement (CME) network based on I3D-ResNet50 as in Fig. 2, which replaces the basic bottleneck block by a dedicatedly designed Multi-Scale Block, embedded with a Denoising Module (DM).

TheMulti-Scale Block (MSB). Fig. 3 (a) and (b) show the difference between the basic Bottleneck block used in I3DResNet50 and the Multi-Scale block proposed in this study. Instead of building features using a 3×1×1 and a 1×3×3 convolution kernel, MSB has four separate branches with cascaded connections, and short/long-term dynamics are captured by varying kernel sizes (i.e. 1, 3, and 5), which efficiently extracts multi-scale motion patterns at multiple spatial granularities without adding too much extra computational cost. Specifically, MSB evenly splits the feature map after the first 1 × 1 × 1 3D convolution to four branches along the channel dimension, where the resulting group of feature maps is denoted by {Xi}i∈{1,2,3,4}. Except the branch X1, for i = 2, 3, 4, MSB removes the noise by the denoising module DM(·), leading to the refined feature map DM(Xi), and performs a cascaded convolution STi(·) on DM(Xi), where STi(·) is composed of a 3 × 3 2D spatial convolution followed by a 1D temporal convolution with kernel size (2i - 3). Since {STi(·)}i=2,3,4 have different sizes of receptive fields, MSB encodes more abundant spatio-temporal patterns than the original Bottleneck block does. For the last two branches (i.e. i = 3, 4), the output of the antecedent branch is also added into the input, i.e. the input for i = 3, 4 is Xi + STi-1(DM(Xi-1)). This hierarchical design enables certain motion information to be captured at multiple scales. Note that it is not applicable to i = 2, as X2 and the antecedent output X1 are at the same scale. Finally, to aggregate the multi-scale features, we concatenate all the output features from the four branches and fuse them by a 1 × 1 × 1 3D convolution. By this means, MSB represents coarse motion cues in a more comprehensive way, thus learning stronger features.

The Denoising Module (DM). As in Fig. 1, the motion noise is always subtle and randomly appears at different positions and we suppress them in both the spatial and temporal domains. Inspired by [Moon et al., 2013], it is promising to reduce this kind of motion noise by a multi-frame fusion. To that end, as shown in Fig. 3 (c), DM initially decreases the noise by aggregating multiple frames via temporal average pooling AvgPoolT (·), which is thereafter resized to the input size by a repeating operation Rep(·) formulated as below:

Next, DM spatially suppresses the noise by successively performing a spatial average pooling AvgPoolS,r(·), a 3×3 2D convolution Conv3×3(·) and an up-sampling UP (·) based on bilinear interpolation as follows:

Here, AvgPoolS,r(·) refers to average pooling with filter size r×r and stride r = 2i-1 . In this way, the informative motion is saved while the subtle noise is simultaneously mitigated.

Finally, a skip connection as well as the Sigmoid function σ are introduced to output the importance weights and the refined feature map is formulated as below:

where ⊙ denotes the element-wise product.

3.3 Attentive Cross-modal Interaction

In compressed videos, I-frames are sparsely sampled from raw RGB frames, with static appearances recorded. Despite being coarse and noisy, P-frames (i.e. the motion vectors and residuals) contain complementary motion cues. It is therefore natural to consider the interaction between the features from I-frames and those from P-frames, to mutually enhance representation learning. However, most existing studies combine the features from the two modalities in a straightforward way, ignoring the issue of cross-modal misalignment caused by the distinct importance of spatio-temporal cues in I- and P-frames to action recognition. To address this, as in Fig. 2, we propose the Selective Motion Complement units at the low level and the Cross-Modality Augment unit at the high level, to perform cross-modal interaction learning for feature fusion.

The Selective Motion Complement (SMC) Unit. Suppose FI,l and FP,l are the feature maps from the RGB (Iframes) and MVR (P-frames) modalities in the l-th layer

(l = 1, 2, 3, 4), respectively. The basic idea of SMC is incorporating aligned motion cues from the MVR modality into the RGB modality. To this end, we introduce a simple yet empirically effective unit by using attentions. Concretely, SMC firstly conducts spatio-temporal attention on FP,l

where MP(·) is max pooling, and AttSP (·) consists of two 1 × 1 × 1 3D convolution layers.

Afterwards, SMC applies channel-wise attention on the attended MVR feature F ′P,l , which is further added to FI,l from the RGB modality as follows:

where AttC(·) is a 3 × 3 × 3 3D convolution layer.

The Cross-Modality Augment (CMA) Unit. To further enhance cross-modal interaction, CMA is presented to fuse the high-level features from the two modalities, which adopts a transformer-like structure. Specifically, CMA employs FI and FP after the last convolutional layer as the highlevel features, based on which the key Km, query Qm and value Vm are generated from Fm by linear mapping, and m ∈ {I, P} indicates the modality. Thereafter, two features are learned for each modality by applying non-local cross attention as FI,att = Softmax( QPKT I √dk )VI and FP,att = Sof tmax( QIKT P √dk )VP , where dk is the dimension of the key and Softmax(·) denotes the softmax function. CMA then sums the features as the combined representation across modalities as Ffused = FI,att + FP,att. Finally, Ffused ensembles with the single-modal feature FI and FP at the score-level, yielding the overall prediction score for classification: s = 13 · (CLS(FI) + CLS(FP ) + CLS(Ffused)), where CLS(·) is the fully-connected classification head.

For more details about the network structures of SMC and CMA, please denote the Supplementary Material † .

4 Experimental Results and Analysis

4.1 Datasets and Evaluation Protocol

We adopt the following three benchmarks for evaluation.

HMDB-51 contains 6,766 videos from 51 action categories and provides 3 training/testing splits. Each split consists of 3,570 training clips and 1,530 testing clips. UCF101 includes 13,320 videos from 101 categories. Similar to HMDB-51, this dataset also offers 3 training/testing splits, each of which has approximately 9,600 clips for training and 3,700 clips for testing. Kinetics-400 is a large-scale dataset covering 400 categories, where 240k trimmed videos are used for training and 20k videos for validation.

By following existing works, we report the top-1 accuracy on average over the three training/testing splits.

4.2 Implementation Details

Similar to the case in the literature [Battash et al., 2020], we convert videos into the unified MPEG-4 Part2 coding format, where the size of GOPs is set as 12, and the frames are resized to 340×256. By following [Li et al., 2020a], we pre-train the model on Kinetics-400 [Carreira and Zisserman, 2017], and fine-tune it on HMDB-51 and UCF-101 by using the crossentropy loss and the SGD optimizer with the weight decay of 0.0001, momentum of 0.9 and batch size of 36. The learning rate is initially set to 0.0001 and decreased by a factor of 10 for every 40 epochs. In training, we uniformly sample 8 frames to generate the input clip from each video and apply random scaling, corner crop and horizontal flip for data augmentation. All the experiments are conducted on two NVIDIA 3090 GPUs. In testing, we uniformly sample 1 or 3 different clips from the input video and average the classification scores for final prediction when using 3 clips. Each frame is resized to 256×256 and cropped into a 224×224 image via center crop for inference.

4.3 Comparison with the State-of-the-arts

Comparison to Compressed Video based Methods. As demonstrated in Table 1, MEACI-Net outperforms the counterpart compressed video based methods by large margins. In particular, MEACI-Net promotes the second best method SIFP-Net by 1.7% and 2.1% in 1-clip test on HMDB-51 and UCF-101 respectively, thus reaching the new state-of-the-art. More importantly, MEACI-Net utilizes 21× less GFLOPs than SIFP-Net, highlighting its efficiency. By using more testing clips per video (i.e. 3 clips), the accuracies are further boosted with more GFLOPs. When comparing CoViAR and CoViAR+Flow, we can observe that optical flows remarkably improve the performance, since they contain much more accurate motion information than coarse motion vectors. In this case, our method still outperforms CoViAR+Flow with 54× smaller input size and 44× less GFLOPs. Other methods such as EMV-CNN, DTMV-CNN, IP TSN, DMC-Net and MFCD-Net additionally adopt optical flows or raw videos to guide network training, but are clearly inferior to MEACI- Net. We also make evaluations on large-scale datasets (e.g. Kinetics-400) to show its capacity. MEACI-Net achieves a get 2.1% higher accuracy than MFCD-Net which is the one compressed video based method reporting performance on Kinetics-400. These results validate that our method explores motion information cues more comprehensively and performs cross-modal interaction more effectively, therefore reaching the state-of-the-art both in accuracy and efficiency.

Comparison to Raw Video based Methods. As shown in Table 1, raw video based methods generally report higher accuracies than compressed video based counterparts, since raw videos convey much more complete motion information. However, MEACI-Net significantly reduces this gap in accuracy, and even outperforms some latest raw video based ones such as TEA. Unsurprisingly, MEACI-Net is much more efficient than raw video based methods, requiring at least 52× smaller input size and 16× less GFLOPs.

4.4 Ablation Study

To evaluate the effectiveness of the proposed components of MEACI-Net, we extensively conduct the ablation study.

Effect of Different Modalities. We investigate the contribution of each modality to the performance of our method. When separately analyzing the RGB and MVR modality, we utilize the I3D-ResNet50 and CME network as the learning models, respectively. In the presence of both modalities, we apply the full model MEACI-Net. As summarized in Table 2, when only using RGB or MVR, the accuracy is sharply decreased, since I-frames of the RGB modality are severely sparse, and P-frames of the MVR modality are coarse and noisy. Their combination via MEACI-Net clearly promotes the performance, showing that they contain complementary information for action recognition.

Table 4: Ablation results (%) on attentive cross-modal interaction and its components SMC/CMA. 'LA' is short for lateral connection.

Effect of CME (MSB and DM). To validate the effectiveness of the proposed CME network and evaluate the impacts of the MSB and DM components, we adopt I3D-ResNet50 as the baseline, denoted by 'B1' ('Baseline 1'), and the other three compared baselines, i.e. 'B1+MSB', 'B1+MSB * ' and 'B1+DB', indicating the model which replaces the Bottleneck block by MSB with or without varying temporal kernel, and adding DM after the 1 × 3 × 3 2D convolution layer of the Bottleneck block, respectively. The CME network is the model by combining MSB and DM as shown in Fig. 3. As reported in Table 3, MSB and DB clearly promote the accuracy of the I3D-ResNet50 backbone, since they separately handle the coarse and noisy motion vectors. Varying temporal kernel delivers a gain of 0.7% on UCF-101 compared to that of a fixed size (i.e. 3). A combination of them further boosts the performance. In fact, CME improves the accuracy of I3D-ResNet50 by 6.8% and 3.6% on HMDB-51 and UCF101 respectively, while saving about 15% in size compared to B1. We visualize the activation maps by Grad-CAM of B1, B1+DM and B1+DM+MSB (CME) in Fig. 4, and we can see that DM helps to focus more on motion areas (in red boxes) as it suppresses local noise by assigning bigger weights to foreground regions. For more visualization results, please refer to the Supplementary Material † .

Effect of ACI (SMC and CMA). We compare the proposed ACI mechanism with other commonly used multimodal fusion strategies. All the counterpart fusion techniques are implemented based on our two-stream framework as shown in Fig. 2 with slight modifications. B2 ('Baseline 2') is the baseline method without cross-modal interactions, i.e. by removing the SMC and CMA units from MEACI-Net. 'B2+Add' refers to the model directly adding the features of the two streams, acting like SMC but without attentions. Since SlowFast [Feichtenhofer et al., 2019] also considers cross-modal interaction by lateral connections, we therefore compare to it, denoted by 'B2+LA'. 'B2+SMC (CMA)' stands for the variant of our method by individually employing the SMC (CMA) unit. MEACI-Net is the full model by combining SMC and CMA. As shown in Table 4, all fusion techniques can improve the performance of the baseline, proving the necessity of cross-modal interactions. B2+SMC outperforms B2+Add, showing that low-level interaction across modalities facilitates multi-modal fusion. It is also shown that when high level interaction via CMA is integrated, the performance reaches the best.

Inference Speed. As most methods do not report time costs or release codes, it is difficult to make fair comparison. In this case, we select open-sourced I3D and CoViAR as representatives of the raw and compressed video based methods. The experiments are conducted on a workstation with an Intel 2.3GHz CPU and an NVIDIA RTX 3090 GPU and the results are displayed in Table 5. With the proposed light-weight modules, MEACI-Net takes 85.6 ms for 8 sampled frames, i.e. running at 93.4 FPS, faster than CoViAR and I3D.

5 Conclusion

In this paper, we propose an Attentive Cross-modal Interaction Network with Motion Enhancement (MEACI-Net) for compressed video action recognition. It employs the CME network to learn discriminative motion patterns from the coarse and noisy dynamics, and performs attentive crossmodal interaction for fusing features from multiple modalities at both the low and high levels. Extensive experimental results validate that MEACI-Net outperforms the state-of-theart in both the accuracy and efficiency.

Acknowledgments

This work is partly supported by the National Natural Science Foundation of China (No. 62022011), the Research Program of State Key Laboratory of Software Development Environment (SKLSDE-2021ZX-04), and the Fundamental Research Funds for the Central Universities.