TEST-V: TEst-time Support-set Tuning for Zero-shot Video Classification

1 Introduction

In the past few decades, the research on behavior recognition has made rapid progress and has been successfully applied in many fields, including intelligent robotics, security surveillance, and automatic driving. However, the number of behaviors existing methods can identify is still limited, which makes it difficult to meet the growing needs of practical applications. In recent years, the powerful ability of large-scale pre-trained Vision-Language models (VLMs) in zero-shot generalization has promoted the rapid development of zero-shot/general behavior recognition. VLMs bring hope for breaking through the perceptual boundaries of existing behavior recognition technologies.

∗Extended version on arxiv: https://arxiv.org/pdf/2502.00426

Recently, it has become a trend to project the test video and class names into the joint feature space based on the pretrained models, and then assign a label to the video according to its feature similarities. Although the amount of pretraining data is huge, the out-of-distribution issue cannot be thoroughly eliminated in the test environment. Hence, it is necessary to further tune the parameters or prompts for the pre-trained model from different modal perspectives.

To address this challenge, existing solutions can be roughly divided into the following two categories. i) Training-based: Conventional methods fine-tune part of parameters belonging/appended to the image-based pre-trained model (e.g., CLIP [Radford et al., 2021]) or extra learnable prompts appended to (visual/text) inputs, based on a mass of video data. ii) Training-free: To avoid high training costs, some recent Test-time Prompt Tuning (TPT)-based methods [Shu et al., 2022; Yan et al., 2024] append learnable prompts to visual/text input sequence and tune prompts in a selfsupervised manner (e.g., multi-view prediction consistency) based on the single test video during test time (Figure 1 (a)). Beyond that, some recent works [Udandarao et al., 2023; Zhang et al., 2021] build a video support-set according to unseen class names via retrieval or generation(i.e., converts class names to videos) and performs intra-modality (videovideo) alignment (Figure 1 (b)). TPT freezes the visual space and only tunes the text space, which cannot reduce the modality gap. On the contrary, the support-set-based methods reduce the gap via inter-modality alignment, but the supporting samples are fixed.

To this end, we proposed a novel framework namely TEsttime Support-set Tuning for Video classification (TEST-V) in Figure 1 (c). It builds a semantically diverse supportset via off-the-shelf text-video generation models and tunes learnable weights to select effective support samples from the set via two core modules. i) Multi-prompting Supportset Dilation (MSD): generate multiple text prompts from the class names via LLMs and feed them into the text-video generation model (i.e., LaVie [Wang et al., 2023b]) for building a semantically diverse support set. ii) Temporal-aware Support-set Erosion (TSE): tunes a few learnable weights to dynamically reshape the contribution of each video frame in a self-supervised manner, according to the multi-scale semantic consistency hypothesis in the temporal domain. Extensive experimental results show that TEST-V improves the state-of-art pre-trained VLMs, namely CLIP [Radford et al., 2021], BIKE [Wu et al., 2023] and VIFi-CLIP [Rasheed et al., 2023] by 2.98%, 2.15%, and 1.83% absolute average accuracy respectively across four benchmarks.

Our main contributions can be summarized as follows:

• We propose a novel framework, namely TEst-time Support-set Tuning, which first dilates and then erodes the visual support-set to enhance the zero-shot generalization on activity recognition during test time.
• To ensure the diversity of the support-set, we propose the Multi-prompting Support-set Dilation (MSD) to generate videos via a text-to-video model as supporting samples for each class according to multiple descriptions.
• To mine critical supporting cues from the supportset, we proposed a Temporal-aware Support-set Erosion (TSE) module to tune factorized learnable weights with different temporal-scale features in multiple steps supervised by the prediction consistency.

2 RelatedWork

2.1 Zero-shot Activity Recognition

Zero-Shot Activity Recognition (ZSAR) aims to recognize actions that are not observed by the model during training, which is a non-trivial task yet useful for practical applications. Early work learn a joint embedding space by aligning video representation with corresponding textual semantic representation, e.g., manually defined attributes [Liu et al., 2011; Zellers and Choi, 2017], word embeddings of category names [Qin et al., 2017; Shao et al., 2020], and elaborative action descriptions [Chen and Huang, 2021; Qian et al., 2022]. With the advent of deep neural networks, many works [Chen and Huang, 2021; Lin et al., 2019; Lin et al., 2022] make use of various video architectures (e.g., I3D [Carreira and Zisserman, 2017] and TSM [Lin et al., 2019]) to obtain high-level visual representation from videos and further project such visual representation into semantic embedding space. Recently, large-scale Vision-Language models (e.g., CLIP [Radford et al., 2021]) have shown great zero-shot transfer ability. Thus, many ZSAR solutions attempt to take advantage of CLIP to align video features and textual features, showing impressive performance in ZSAR. ActionCLIP [Wang et al., 2021b], A5 [Ju et al., 2022], and XCLIP [Ni et al., 2022] adapt CLIP for videos by training additional components, e.g., temporal transformer blocks and text prompts. ViFi-CLIP [Rasheed et al., 2023], BIKE [Wu et al., 2023], and MAXI [Lin et al., 2023] fully fine-tune the encoder of CLIP to learn video-specific inductive biases, without introducing any additional learnable parameters. Unlike current approaches that rely heavily on many labeled video samples to fine-tune network parameters, our work pioneers the idea of building an efficient visual support set for zeroshot classification without training.

2.2 Adaptation of Visual-Language Models

Recently, pre-trained Vision-Language Models (VLMs) (e.g., CLIP [Radford et al., 2021], ALIGN [Jia et al., 2021]) have shown impressive zero-shot transfer ability in recognizing novel categories. Thus, many works attempt to adapt VLMs for downstream tasks through prompt tuning or training-free methods. CoOp [Zhou et al., 2022b] fine-tunes the learnable text prompts on downstream training data instead of using handcrafted templates [Zhou et al., 2022b], effectively improving CLIP zero-shot performance. CoCoOp [Zhou et al., 2022a] extends CoOp to learn input-conditioned prompts, leading to better generalization to unseen classes and different domains. Despite impressive, these works need access to labeled samples from the target distribution, and further train prompts over these samples. In addition, recent training-free methods such as TIP-Adapter [Zhang et al., 2021] leverage a few labeled training samples as a key-value cache to make zero-shot predictions during inference, avoiding the conventional model parameter fine-tuning. Yet these methods still rely on samples from the target distribution. To discard costly labeled data and training, SuS-X [Udandarao et al., 2023] utilizes category name and text-to-image generation model to construct a visual support-set, thus enhancing zero-shot transfer abilities. Our test-time support-set tuning shares a similar spirit of pursuing a visual support-set curation strategy to adapt VLMs, but we strive to construct a diverse video support set guided by elaborated descriptions of each class name, and make full use of each sample in the support-set by dynamically adjusting the contribution of each frame.

2.3 Test-time Tuning

Test-time Tuning (TTT) [Shocher et al., 2018; Nitzan et al., 2022; Xie et al., 2023] aims to improve the generalization capabilities of pre-trained models by utilizing test samples to fine-tune the model, especially when faced with data distribution shifts. One of the key challenges in TTT is designing effective optimization objectives. Early works [Sun et al., 2020; Liu et al., 2021] enhance the training process by integrating a self-supervised multitask loss. Besides, TENT [Wang et al., 2021a] minimizes predictions' entropy on target data to improve generalization capability, yet requiring multiple test samples. To address this issue, MEMO [Zhang et al., 2022] and TPT [Shu et al., 2022] utilize multiple augmented views generated from a single test sample through data augmentation for further test-time tuning. Recent methods extend TTT to the video domain by self-supervised dense tracking [Azimi et al., 2022] or aligning video features with different sampling rates [Zeng et al., 2023]. Inspired by this, we tune the support-set via learnable parameters with temporal prediction consistency loss for mining reliable supporting cues.

3 Preliminary and Definition

3.1 Problem Statement

Zero-Shot Activity Recognition (ZSAR) aims to recognize unseen actions during model testing. Specifically, given a test video Vtest and a set of class names Y = {y0, y1, · · · , yc}, we employ the pre-trained visual (Fvis) and text (Ftxt) encoders e.g., CLIP [Radford et al., 2021], to encode them and then calculate the feature similarity for prediction as follows,

Here, f andW denote visual and text features, respectively, and "·" calculates their cosine similarity.

Although the pre-trained representation is robust enough, it cannot avoid the inherent semantic gap between visual and text features (a.k.a, modality gap). To this end, some recent methods (e.g., TIP-Adapter [Zhang et al., 2021], SuSX [Udandarao et al., 2023]) attempt collecting a set of visual samples (support-set) as the replacement of class names and perform zero-shot matching in the visual space for reduce such gap. We first briefly review such support-set-based zero-shot matching methods as below.

3.2 Support-Set-based Zero-shot Prediction

Support-Set Construction. Given a set of class name Y , the text-to-image (T2I) generation model (e.g., Stable Diffusion [Rombach et al., 2022]) is employed to generate K images for each category to construct the support set. To obtain more diverse generated samples, SuS-X [Udandarao et al., 2023] explains each class name in detail through LLMs (e.g., GPT-3 [Brown et al., 2020]) as input of T2I model following CuPL [Pratt et al., 2023]. Thus, this support-set consists of C × K visual samples where C and K denote the number of classes and the number of generated samples for each class.

TIP-Adapter Inference. Based on the above support-set, CLIP's visual encoder Ev is applied to extract its visual features F ∈ RCK×d , and its labels are converted into the onehot vector L ∈ RCK×C . TIP-Adapter aims to calculate the distance between the test video feature f and support videos' feature F , and then make predictions (ZTA) with the help of label information L as follows,

Here, β adjusts the sharpness and affinities calculated from fF T is used as attention weights over L to achieve logits.

TIP-X Inference. To avoid the uncalibrated intra-modal feature distances in TIP-Adapter, TIP-X [Udandarao et al., 2023] uses text modality as a bridge between visual modalities. TIP-X calculates the distance between the test video feature f and text features W, and between support videos' feature F and text feature W, respectively. TIP-X applies KL-divergence to construct the affinity matrix for final prediction (ZTX) as follows,

Here, fW T and FW T are normalized via softmax, KL(·) computes the similarity between two probabilities (KL(P ∥ Q) = ∑i Pi log Pi Qi ) and ψ is the scaling factor.

4 Methodology

4.1 Multi-prompting Support-set Dilation (MSD)

As shown in Figure 3, the performance of the support-set based on SuS-X is extremely saturated when the number of videos per category increases, suggesting that the generated video samples' representations are highly overlapping/similar in each class. This inspires this work to improve the diversity of generated samples to refine the semantic boundary of each class within the support-set. To this end, we propose the Multi-prompting Support-set Dilation (MSD) module to generate diverse video samples via a video-text generation model with different prompts.

Diversified Motion Description. Given a set of action classes Y = {yi}Ci=1 , we employ a large language model (LLM) (e.g., ChatGPT [OpenAI, 2023]) to generate M different prompts for each class as

Here, query template(·) is a crafted text template used to wrap each action class into a text query as the input of LLM. After traversing the entire action class set Y , we can achieve a diversified motion description set D = {di}Ci=1 in which each class corresponds to M different explanations and C is the number of action classes.

Video Generation from Diversified Prompts. For the i-th action class, we generate K = m × n video samples via the text-to-video model T2V(·, ·) according to text prompts stored in di as follows,

Here, we Sample m prompts for each class from di and repeatedly generate n videos corresponding to each prompt. K is set to be always factorable for ease of implementation, two hyper-parameters (i.e., m and n) are ablated in the experiments (as shown in Figure 3). After traversing the entire description set D, we can construct the video support-set S = {si}Ci=1 in which each class has K support videos.

Different from previous methods (SuS-X [Udandarao et al., 2023], CaFo [Zhang et al., 2023b], and DMN [Zhang et al., 2024]) which explain class names repeatedly, MSD constructs diverse descriptions for each class name with different context/setting/perspectives.

4.2 Temporal-aware Support-set Erosion (TSE)

The support set generated via MSD has a greater diversity in each category (as shown in Figure 4). MSD refines the boundary of each class but still contains many invalid samples (i.e., outliers or duplications), which motivates us to further adjust the support-set, i.e., "eroding" some invalid information from the set. A straightforward solution is to apply learnable weights on video samples and tune them with a selfsupervision loss, inspired by TPT [Shu et al., 2022]. However, video-level tuning can miss the potential fine-grained supporting cues hidden in video data. Hence, it is necessary to tune the weights on frame-level or patch-level visual features, but patch-level introduces so many parameters that test-time tuning techniques cannot accomplish optimization. To this end, we proposed a novel Temporal-aware Supportset Erosion module to reweigh the importance of each frame from supporting videos via factorized video-frame weights, which is tuned via different temporal-scale features in multiple steps during test time.

Factorized Re-weighting. Given the support set S generated from MSD, we utilize the pre-trained visual encoder (Fvis) from popular VLMs (e.g., CLIP) to extract visual features and concatenate them together as follows,

Here, C, K, T, and d represent the number of classes, supporting samples of each class, frames, and channels, respectively. After that, we reweight the support-set feature F in video and frame level via factorized learnable weights as follows,

Here, "⊙" denotes element-wise matrix multiplication (i.e., Hadamard product). rvid ∈ RCK×1×1 and rfr ∈ R1×T×1 are frame-level and video-level weights, respectively.

Multi-scale Temporal Tuning. Inspired by Test-time Prompt Tuning (TPT [Shu et al., 2022]), we tune learnable weights (r) via minimizing the prediction consistency between re-weighted support-set features and the augmented test video features as follows,

Here f S ∗ = Fvis(AugSn (Vtest)) denotes the augmented test video features and AugSn (·) performs spatial augmentation n times. Pred(·) computes multi-view predictions from the support-set features and augmented test video features according to Eq. (3). L calculates the averaged entropy from high-confident predictions [Shu et al., 2022].

However, the temporal evolution rate of different human activities is variable, and building multi-scale representations from the temporal domain has proven beneficial in video representation [Feichtenhofer et al., 2019; Yan et al., 2024]. Therefore, Eq. (8) can be rewritten as follows,

Here, AugTm(·) augments two feature sets synchronously via m time sampling at different temporal scales. Different sampling strategies are compared in Table 4 and discussed in Sec. 5.2. Notably, this loss function is optimized in m steps with different temporal-scale features.

5 Experiments

We evaluate the effectiveness of the proposed method on four popular video benchmarks, i.e., HMDB-51 [Kuehne et al., 2011], UCF-101 [Soomro et al., 2012], Kinetics-600 [Carreira et al., 2018], ActivityNet [Fabian Caba Heilbron and Niebles, 2015].

5.1 Comparison with State-of-the-Arts

We evaluate the effectiveness of TEST-Vwith existing stateof-the-art methods on four popular action recognition benchmarks in Table 1. Existing methods can be categorized into three types: i) Uni-modal zero-shot video recognition models: they are trained on video data with elaborate representation engineering; ii) Adapting pre-trained CLIP: they adapt CLIP to video data via additional temporal learners or VL prompting techniques; iii) Tuning pre-trained CLIP: they fully fine-tune the CLIP model via video data.

We apply off-the-shelf pre-trained visual/text encoders from VLMs, e.g., BIKE [Wu et al., 2023] and ViFi-CLIP, to extract features and tune the proposed TEST-V during test time without any training data. Our method outperforms the conventional uni-modal zero-shot video recognition models by a large margin (15% ∼ 20%). Moreover, compared with the methods adapted from pre-trained CLIP with video data, our method achieve consistent improvements across all benchmarks without training. Furthermore, our method can significantly improve some recent methods fully-tuned from CLIP, thanks to the reduction of modality gaps through the tuned support set. In general, the proposed TEST-V achieves state-of-the-art performance on four popular benchmarks in a training-free manner.

5.2 Ablation Study

Choice of LLM and T2V for MSD. As described in Sec. 4.1, MSD builds multiple descriptions (i.e., prompts) for each class via the Large Language Model (LLM) and then generates video samples via the Text-to-Video (T2V) model according to multiple prompts. There are many options for LLM and T2V, thus we compare them in Table 2 and 3, and make the following analysis. i) LLMs: We compare different Large Language Models (LLMs) for motion description generation defined in Eq. (4) in Table 2. In general, our approach works well on all LLMs, and new versions of the same model perform better. ChatGPT achieves the best performance and is selected as the default LLM. ii) T2Vs: The proposed method is compatible with various Text-toVideo (T2V) generation models as shown in Table 3. Benefited from the extra training videos data, LaVie is slightly better than others and is adopted by default.

1 including Show-1 [Zhang et al., 2023a], HiGen [Qing et al., 2024], TF-T2V [Wang et al., 2024], ModelScopeT2V [Wang et al., 2023a], and LaVie [Wang et al., 2023b]

Importance of Diversity of the Support-set To figure out the key factor of the quality of the support-set, we ablate the support-set hyper-parameters of the proposed MSD, i.e., the number of supporting videos for each class K and the number of videos repeatedly generated for each prompt n defined in Equation (5), as shown in Figure 3. The performance of all models (baseline SuS-X and the proposed MSD with different n) gradually plateaus as the support-set size (K) increases across two benchmarks. Besides, the proposed multiprompting method (MSD) outperforms the single-prompting baseline (SuS-X) when K ̸= 0, demonstrating the importance of sample diversity in the support-set. Meanwhile, we ablate the number of repeatedly generated videos n when the total number of videos K is fixed. (K = m× n). We found that n = 4 brings stabilized gains compared to the baseline and set n = 4 for all benchmarks.

Multi-scale Temporal Tuning. To find the suitable temporal scale used in TSE, we try different sampling strategies and report the performance in Table 4. "random" sampling frames from the feature sequence randomly, while "top" selects the top k features with higher frame-level weights (rf ). We observed that multi-scale tuning with the "random" strategy does not work compared with the single-scale tuning. Besides, multi-scale tuning with the "top" strategy of "4,6,8" achieves state-of-the-art results.

Component Analysis. We explore the effect of each module designed in TEST-V in Table 5. Compared with the baseline, MSD brings approximately 1% improvements on two benchmarks, indicating that the diversity of supporting samples in each class is beneficial in enhancing the quality of the support set. Meanwhile, TSE demonstrates the ability to select critical visual cues from highly redundant video data, which improves the baseline by approximately 1% ∼ 2%. Furthermore, the combination of MSD and TSE yields even more significant performance gains, confirming the complementarity of the two modules.

As defined in Eq. (7), learnable weights rvid and rfr are designed to adjust the contribution of video-level and framelevel features, respectively. As reported in Table 6, each of them brings performance gains, and they are complementary.

Generalization to Different VLMs. We also extract visual/text features via different pre-trained VLMs, i.e., Vanilla CLIP [Radford et al., 2021], and BIKE [Wu et al., 2023], with various network backbones, and report top-1 zero-shot accuracy on two benchmarks in Table 7 compared with SuSX [Udandarao et al., 2023]. Our approach shows consistent improvement across two benchmarks with different models and backbones, demonstrating good generalization across different video representations.

Visualization. We randomly sample 20 videos from the support-sets generated with multiple prompts and the single prompt for four benchmarks, respectively, and encode them via the pre-trained visual encoder from BIKE [Wu et al., 2023] with 4 frames. Feature distribution of sampled videos in each benchmark is visualized via t-SNE [Van der Maaten and Hinton, 2008] in Figure 4. We observed that single-prompting samples are more likely to cluster together, whereas multi-prompting samples are more dispersed. It indicates that the multi-prompting strategy can refine the supporting boundary to enhance the quality of the support-set.

6 Conclusion

This work presents a novel framework, namely TESTV which dilates and erodes the support set to enhance the zero-shot generalization capability of video classification methods during test time. TEST-V applies multiple prompts to enrich the support set (Dilation) and then mines critical supporting samples from the support set with learnable spatio-temporal weights (Erosion). Our method demonstrated superior performance to existing state-of-arts on four benchmarks. While TEST-V effectively adapts pre-trained models (i.e., CLIP) to out-of-distribution domains during test time for video data, it still has two potential limitations: i) extra spatio-temporal tuning costs may increase with the length of the video (i.e., long video). ii) relying on the quality of LLMs and text-to-video generation models.

Acknowledgments

The work is supported by the National Natural Science Foundation of China (Grant No. 62472208 and 62172226), the Postdoctoral Fellowship Program of CPSF (Grant No. GZB20230302). Rui Yan and Jin Wang contributed equally to this work, and Xiaoyu Du is the corresponding author.