Long-Short Term Cross-Transformer in Compressed Domain for Few-Shot Video Classification

1 Introduction

Few-shot learning (FSL), a fundamental problem in machine learning, aims to learn information about categories from a few training samples. This topic become increasingly popular because in practice collecting a large amount of labeled data is often difficult. Recently, FSL has reached Few-shot video classification [Dwivedi et al., 2019; Cao et al., 2020; Zhang et al., 2020; Perrett et al., 2021] has been tried recently, but most existing methods fail to sufficiently capture the temporal cues. On the one hand, frames are usually evenly sampled, and each frame is treated as equally important. In actuality, the information intensity of video is typically not uniformly distributed. As a result, some critical information may be missed. On the other hand, the heuristic model simply encodes the equally treated frames in sequence, which results in the lack of both long-term and short-term temporal modeling and interaction. For example, in Figure 1, the evenly sampling strategy often extracts some useless information (e.g., the single head in (a) or the background in (b)). Besides, some class information reflects in the long term while others reflect in the short term. Thus, the long-shortterm temporal modeling is extremely necessary.

∗Equal contribution.

†Corresponding author.

To alleviate these problems above, we explore coarse and fine-grained temporal information and more accurate temporal relation, to better match the query and support. On the one hand, more effective temporal data is used. We find that the compressed domain natively contains useful information. For example, the motion vectors (MVs) show the short-term temporal and motion information of each frame. The MV intensity can also indicate the importance of the current frame. For long term, the video is split into different groups of pictures (GOPs) according to the content, and the frames with the same scene and content are usually included in one GOP. Besides the temporal cues, the intra-frame (I-frame) of each GOP also provides the appearance information. More importantly, the compressed domain data can be acquired at a low cost. We only need to entropy-decode the video bitstream to access the compressed domain data, rather than fully decode the RGB frames.

On the other hand, a more accurate spatial-temporal feature space and temporal relation are constructed. Taking advantage of compressed domain knowledge, we propose a Longshort Term Cross-Transformer (LSTC) for few-shot video classification. Firstly, the informative data is adaptively selected from the multi-modal compressed domain data with long-short term selection. Then for the short-term temporal module, a multi-modal integration network is designed, in which the I-frames and MVs sequences from the same GOP interact with each other. Finally, the long-term temporal module performs self-attention for long-term modeling and then computes the cross-attention between tuples of query embeddings and those of support for more sufficient matching. The prototype for each support class is generated. The label is predicted by the distance between the query and each prototype.

Our main contributions are summarized as follows:

• We proposed a novel framework called long-short term cross-transformer (LSTC) for few-shot video classification, which can deeply exploit the long-short term video information and well match the query-support pair.
• We take advantage of the compressed domain knowledge to obtain effective spatial-temporal information at a low cost and adaptively select the informative data to be processed.
• Experimental results show that our method is effective and outperforms the state-of-the-art (SOTA) on both large-scale and small-scale datasets, including SSV2 [Goyal et al., 2017], Kinetics [Carreira and Zisserman, 2017], UCF [Soomro et al., 2012] and HMDB [Kuehne et al., 2011].

2 RelatedWork

Few Shot Video Classification. Few-shot learning (FSL) addresses the challenging problem of learning from a few labeled examples. Videos reside in a higher-dimensional space than images, which increases the difficulty to learn a strong classifier with limited samples. Compound Memory Network (CMN) [Zhu and Yang, 2018] constructs a two-layer compound memory structure to store video features for matching. Temporal Attentive Relation Network (TARN) [Bishay et al., 2019] performs segment-wise alignment before matching support and query videos with relation network [Sung et al., 2018]. EOSVR [Fu et al., 2019] proposes embodied one-shot video recognition with synthetic data. OTAM [Cao et al., 2020] aligns videos with differentiable dynamic programming. Temporal Cross-Transformers (TRX) [Perrett et al., 2021] adapt Cross-Transformers [Doersch et al., 2020] to constructs query-specific prototypes from tuples of frames. The problem of video embedding is discussed in [Zhu et al., 2021]. The previous methods attempt to capture the temporal structure of videos but achieve limited success. A critical problem is that their input is RGB frames, where temporal clues must be inferred indirectly and often implicitly. AMeFu-Net[Fu et al., 2020] exploits depth as additional input. In contrast, our method considers compressed domain data that contains direct temporal information, exploiting global and local temporal structure with a long-short term model that better suits the new input modality.

Compressed Video Classification. Compressed domain data provides simple and fast temporal information, which can improve the performance of video classification methods with limited overhead. The acquirement of compressed domain information is nearly cost-free compared with conventional methods that fully decode the input videos. Moreover, MVs contain the movement at the block level so they can serve as coarse motion estimation. CoViAR [Wu et al., 2018] pioneers compressed video classification, replacing optical flow in [Wang et al., 2016] with MVs and residuals. DMC-Net [Shou et al., 2019] refines MVs with the supervision of optical flow. Slow-I-Fast-P [Li et al., 2020] establishes pseudo optical flow fromMVs and residuals for the fast path of SlowFast Network [Feichtenhofer et al., 2019]. These methods reveal the potency and efficiency of compressed domain features for video classification. Nonetheless, they treat compressed domain features as insertion into conventional architecture instead of developing a new befitting compressed video classification framework.

3 The ProposedMethod

We propose a novel Long-Short Term Cross-Transformer (LSTC) for few-shot video classification. It takes advantage of the compressed domain knowledge and performs accurate query-support matching with a long-short term structure. Here, we introduce the proposed method, including the framework, formulation, and technical details.

3.1 Framework and Formulation

The overall framework is summarized in Figure 2. We first extract the compressed domain data by partially decoding the video bitstream at a low cost. Specifically, the I-frames and the motion vectors (MVs) from different Groups of Pictures (GOPs) are obtained. We denote the frames in the t-th GOP as {Gt,l}Lt l=0 , where Gt,0 is the I-frame. The MVs are denoted as Mt,l. The I-frames are essentially RGB images, containing the appearance information. The MVs denote the motion from the source positions in the previous frame to the destination positions in the current frame, containing the temporal information. The I-frames and MVs are extracted from the video stream by entropy decoding.

Secondly, we design a long-short term selection module to extract the informative I-frames and MVs from long-term GOP-level and short-term frame-level. After that, a shortterm module is embedded to exploit the short-term temporal information and fuse the multiple modals of I-frames and MVs at GOP level. Fed with the embeddings from the shortterm module, a cross-transformer is constructed to build the long-term temporal information and to explore the relationship between the support and the query.

In this paper, we consider the C-way K-shot few-shot video classification problem, in which an episode [Vinyals et al., 2016] consists of C classes with K support videos for each class. The target of this problem is to classify the query video into one of the classes c ∈ {1, 2, · · · , C}.

3.2 Long-short Term Selection

The input frame selection is an important problem for video classification. Usually frames are randomly sampled either evenly or successively with a fixed stride [Wang et al., 2016; Cao et al., 2020; Perrett et al., 2021]. This selection strategy may miss some keyframes and introduce some irrelevant background frames. In our method, we utilize the compressed domain to conduct long-short term information selection.

There are useful cues to distinguish the informative frames in the compressed domain. In the long term, the GOPs with different lengths have already divided the contents into several parts. In the short term, the MVs reflect the importance of the current action. Hence, we define a metric to evaluate the importance of the l-th frame in the t-th GOP:

where | · | is the cardinality of a set, || · ||1 is the L1 norm, (x, y) is spatial location, and α ≥ 0. Based on this metric, we obtain the importance score of the t-th entire GOP:

According to this importance score, in the long term, we extract the informative GOPs with probabilities of {P sel t ∝ I(GOPt)}Tt=1 . Likewise, in the short term, we extract the informative MVs with probabilities of {P sel l ∝ I(Mt,l)}Lt l=1 . We sample them with a probability rather than select the fixed top-n frames because the fuzzification in the proposed selection scheme can tolerate some noise in these frames. This alleviates the influence of some outliers and noise on the performance. g GOPs are sampled for each video, and the Iframe and m additional frames (P-frames) are selected for each sampled GOP. We then accumulate the MVs and conduct an alignment between the accumulated MVs and the Iframes. The detailed process is described in the Supplementary Materials. The selected I-frames and aligned accumulated MVs are used as input to the Long-short Term CrossTransformer.

3.3 Long-short TermCross-Transformer Short-Term Module

We conduct short-term temporal interaction in each GOP using the designed short-term module (STM). The embeddings generated by STM are enhanced with local temporal relation, providing better appearance and motion modeling than features from RGB frames only. Compressed domain natively decouples the original video into appearance part and motion part (i.e., I-frames and MVs). Hence, we construct a twobranch network, one for appearance modeling (I-branch, i.e., fI(·)), and the other for motion modeling (MV-branch, i.e., fMV(·)). Each branch employs a staged convolutional neural network (CNN) as the backbone and extracts features for each frame. Between these two branches, they interact with each other at each stage r (i.e., r = 1, 2, ...,R). The interaction is performed between the I-frame feature maps and the MV feature maps from the same GOP.

In our experiments, we use ResNet [He et al., 2016] as the backbone. The appearance-motion interaction is performed at the first residual blocks of stages Conv2-Conv5. As depicted in Figure 3, the feature maps of each branch are encoded with conv1 × 1 and added to the residual path of the other branch. Only a proportion p ∈ [0, 1] of I feature channels interact with MV features so that STM can preserve the original appearance in the other (1 - p) I-branch channels. All MV features participate in the interaction. The interacting I feature channels are evenly divided into m groups, each of which exchanges information with the MV features of a P-frame by lateral connections. STM is applied independently for each GOP. Finally, for each video, the output features of all the I-frames at I-branch are stacked as ZI ∈ Rg×dI , and the output features of all the P-frames at MV-branch are stacked as ZMV ∈ Rgm×dMV , where dI and dMV are the output dimensions of I-branch and MV-branch, respectively.

Long-Term Module

After obtaining the embeddings, we match the query and support videos with long-term module (LTM). LTM consists of two Cross-Transformers, one for the appearance embeddings from the I-branch and one for the motion embeddings from the MV-branch. For each Cross-Transformer, a self-attention layer is first adopted to the embeddings Z ∈ {ZI,ZMV } of each video. Given query/key/value matrices WQ S ,WK S ,WVS ∈ Rd×d , where d ∈ {dI, dMV } is the column dimension of Z, the self-attention is calculated as:

Position encoding [Vaswani et al., 2017] is applied on both Z and H. After self-attention, we construct embeddings for tuples of length n from each video's embeddings H, allowing for fine-grained matching. For the query video, the tuple embedding qb is constructed as following:

in which�denotes vector concatenation, {hj1 , · · · , hjn } are n different row vectors fromH. Similarly, the tuple embeddings of the support videos are obtained in the same way. Then for each class c, we stack all possible tuple embeddings from every shot as row vectors to obtain the class representation Sc . Given query matrix WQ ∈ Rnd×dk , key matrix WK ∈ Rnd×dk and value matrix WV ∈ Rnd×dv , where dk, dv are hidden dimensions, the support prototype of class c is obtained by cross-attention:

Subsequently, the distance between the query and the c-th class support is computed:

We extract all the possible tuples of the query video to match the support videos. The average distance is calculated:

The predicted result is the class with the smallest distance between the support and the query.

3.4 Optimization

To train and optimize the proposed Long-short Term CrossTransformer, we minimize the distance between the query video and the matched support video. In the loss function, negative distances are regraded as logits to compute the crossentropy loss:

where hc,i denotes the label of the i-th training sample.

4 Experiments

4.1 Settings

Datasets. Our evaluations are conducted on four datasets, including Kinetics [Carreira and Zisserman, 2017], Something-Something V2 (SSV2) [Goyal et al., 2017], UCF [Soomro et al., 2012] and HMDB [Kuehne et al., 2011]. The first few shot video classification dataset is constructed from Kinetics by CMN [Zhu and Yang, 2018]. In their setup 100 classes are sampled from the total 400 classes. Selected classes are then split into train/val/test sets of 64/12/24 classes without overlapping, and 100 videos are sampled for each class. On SSV2 we evaluate our method with the split proposed by [Cao et al., 2020] which contains 64/12/24 classes for train/val/test with approximately 1000 videos for each class. Following [Zhang et al., 2020], we use 70/10/21 classes as train/val/test set for UCF and 31/10/10 for HMDB, respectively. All reported results are measured over 10,000 randomly sampled episodes from testing sets.

Implementation. MPEG-4 encoded videos are used as input. We sample g = 4 GOPs from each video and extract the corresponding I-frames and the MVs of m = 2 P-frames. When the video has no enough GOPs or P-frames, the GOPs and P-frames may be sampled several times. The horizontal and vertical components of MVs are rescaled respectively by the width and height of the video. For the short-term module, we use ResNet-50 [He et al., 2016] as the backbone for I-branch and ResNet-18 for MV-branch, both initialized with ImageNet [Deng et al., 2009] pre-trained weights. We set α = 0.4 in long-short term selection. For the LSTC, we set p = 0.5, the latent dimensions are set to dk = dv = 1024 and dk = dv = 512 for I and MV, respectively. The two CrossTransformers in the long-term module are implemented with n = 2 and n = 3, respectively. Their predictions are merged by averaging. During training, the I-frames and MVs are randomly cropped to 224×224. During testing, they are resized to height 256 before center cropping. Following [Wu et al., 2018], we augment I-frames with random color jittering.

The whole network is trained using SGD optimizer [Bottou, 2010] with the learning rate of 0.025. Cross-entropy loss is calculated on a batch of 32 episodes. The training continues for 60,000 episodes on SSV2, and 10,000 episodes on other datasets. We use 4 NVIDIA RTX2060 GPUs for training.

4.2 Performance Comparison

To verify the effectiveness of the proposed method, 7 SOTAs are taken into account for comparison, including MatchNet [Vinyals et al., 2016], MAML [Finn et al., 2017], CMN [Zhu and Yang, 2018], TARN [Bishay et al., 2019], OTAM [Cao et al., 2020], ARN [Zhang et al., 2020] and TRX [Perrett et al., 2021]. The comparative results are summarized in Table 1. The proposed method achieves SOTA performance on all four benchmark datasets. For the 5-shot setting, the proposed method surpasses the previous SOTA by 2.1%, 1.2% respectively on SSV2 and HMDB. Note that the gains come with little overhead. Although MVs are introduced as an additional stream to I-frames, our elaborate design of short-term modeling module reduces the convolutional feature dimension by a factor of m, which greatly decreases the number of parameters of the Cross-Transformer. Furthermore, the acquirement of I-frames and MVs is more efficient than traditional fully decoding, because, in the extraction of compressed domain data, most of the computationintensive decoding steps can be skipped. Compared with prior SOTA methods which typically take 8 RGB frames as input, the proposed method is fed with only 4 I-frames in a video, which contain much less appearance information than 8 RGB frames. Therefore, the major contribution to the SOTA performance of LSTC is the effective utilization of temporal information contained in MVs and the appearance information in I-frames.

4.3 Ablation Study

In this section, we conduct extensive experiments to further analyze and discuss the effectiveness of the proposed method. The results of ablation study on constituent parts of our method are summarized in Table 2. Baseline model consists of I-frames and MVs streams with traditional selection strategy, i.e., GOPs and MVs are uniformly sampled.

The baseline model replaces the short-term module with plain ResNet backbones, without interaction between two streams. The long-term module is removed and the maximal cosine similarity between frames is used to measure the distance between two videos. In addition, we provide analysis on g, m, and few-shot settings, and compare compressed domain data with optical flows. See the Supplementary Materials for more quantitative and qualitative results.

The Effect of Motion Vectors and Frame Selection

We argue that densely sampled RGB frames are highly redundant for video classification. In the traditional setting [Perrett et al., 2021; Cao et al., 2020] 8 frames are drawn from the original video to construct a video representation. In our method, merely 4 I-frames containing RGB information are selected, and MVs provide additional motion information for accurate classification. As demonstrated in Table 2, utilizing I-frames and MVs already yields a few-shot video classifier with considerable accuracy. The performance is future improved with the introduction of long-shot term selection, which utilizes the imbalanced distribution of informative frames and the correlation between MVs and the significance of motion. Moreover, after removing the selection, a 1.5% drop and a 1.0% drop are observed on HMDB and UCF, respectively. This proves the effectiveness of the proposed long-short term selection procedure.

Although a few-shot classifier based on MVs and longshort term selection is a strong baseline, the performance of such a naive combination is far from satisfactory. The naive approach fails to operate on I-frames and MVs jointly and derive enough information. To achieve SOTA performance, we must design an elaborate method to fuse them effectively.

The Effect of STM and LTM

In our method, we propose short-term module (STM) and long-term module (LTM) to facilitate the fusion of appearance encoded by I-frames and motion encoded by MVs. In Table 2, results demonstrate the considerable improvement with the introduction of STM and LTM. With STM, the accuracy improves by 4.3% on HMDB and 4.0% on UCF compared with the baseline model with MV and long-short term selection only. LTM brings an improvement of 5.0% and 4.6%, respectively. These results prove that our STM and LTM design can bridge the gap between two modalities of appearance and motion, generating distinctive features from these two streams. Moreover, when MVs in the full method are replaced by all-zero input, i.e., no MVs are provided, the performance dropped significantly by 2.9% and 2.5% on HMDB and UCF, respectively. The ability of STM and LTM to dig and utilize distinctive video features is considerably impaired without MVs.

Varying the Number of Sampled GOPs and P-frames

The numbers of sampled GOPs (g) and P-frames per GOP (m) are both hyperparameters in our method. We choose g = 4 and m = 2 in previous experiments. This setting allows us to make a fair comparison with previous methods which typically take 8 RGB frames as input. To demonstrate the scalability of our method, we summarize the performance on HMDB with different g and m in Table 3. All other settings are the same as section 4.1. Accuracy is generally improved with more GOPs and P-frames per GOP, but the computational cost also increases with more I-frames and MVs to process. Since the numbers of GOPs and P-frames in each GOP have upper bounds for videos in a certain dataset, sampling more GOPs or more P-frames per GOP than those existing in the video is pointless. A practical choice for g and m depends on the average length of videos, encoding, and computation ability.

Performance with Different Few-Shot Settings

Table 4 shows the impact of the few-shot setting, namely the number of ways (C) and shots (K), on performance. In general, our method performs better with more shots, i.e. more "hints", and fewer ways, i.e. fewer "possibilities". The same phenomenon is observed in previous works [Zhu and Yang, 2018; Bishay et al., 2019; Perrett et al., 2021] and is likely ubiquitous in few shot video classification. Nevertheless, such a problem may not be acute in the real world. In practice, usually more than a few samples could be retrieved for a well-defined category, albeit still considered "few-shot" as opposed to the traditional many-shot scenario which requires thousands of training samples.

Comparison with Optical Flow

In Table 5 we replace MVs in our method with optical flows and compare the performance. The optical flows are extracted and processed following [Wang et al., 2016]. All other parts of our method remain the same. Data time, inference time, and accuracy on HMDB are reported. As demonstrated by the results, extracting compressed domain data is by orders of magnitude faster than calculating optical flows, saving a significant amount of time and computing resources. On the other hand, the performance of the optical flow-based method is worse than the MV-based method. This may be caused by the fact that the optical flows lack the required GOP structure, which indicates that the compressed domain data is not a simple replacement of optical flows in our method.

5 Conclusion

This paper proposes a novel framework called Long-short Term Cross-Transformer (LSTC) for few-shot video classification. It takes advantage of compressed domain data to match the query and the support videos. In particular, LSTC consists of long-short term selection, short-termmodule, and long-term module. The long-short term selection adaptively selects and extracts the informative data by analyzing the implicit cues in the compressed domain. The short-termmodule integrates the multi-modal compressed domain data (i.e., Iframes and MVs) and makes them interact with each other, to obtain fine-grained spatial-temporal features. Given these short-term embeddings, the long-term module computes the global temporal representations and cross-attention between the query and support. Experiments show the effectiveness of the proposed method on various datasets.

Acknowledgments

This work was supported by the National Key Research and Development Program of China (Grant No.

2020AAA0106800), the Natural Science Foundation of China (Grant No.61902401, No. 62192785, No. 61972071, No. U1936204, No. 62122086, No. 62036011, No. 62192782, No. 61721004 and No. 61906052), the Beijing Natural Science Foundation No. M22005, the CAS Key Research Program of Frontier Sciences (Grant No. QYZDJSSW-JSC040). The work of Bing Li was also supported by the Youth Innovation Promotion Association, CAS.