Reliable and Diverse Hierarchical Adapter for Zero-shot Video Classification

1 Introduction

Large amounts of labeled data, such as K400 [Kay et al., 2017] and K600 [Carreira et al., 2018], are typically required in deep learning based video classification tasks for training. However, annotating sufficient samples is time-consuming and resource-intensive, which poses significant challenges for practical applications. In recent years, many works [Rasheed et al., 2023; Wang et al., 2023; Ju et al., 2022] focus on leveraging zero-shot learning techniques to transfer the learned knowledge to predict novel categories. They have attempted to tune an off-the-shelf visual encoder, originally developed for the image domain, on large-scale video datasets for zeroshot video classification. Though these works have improved the model architecture to enhance motion semantic representations, they still suffer from the issue of data shift, which degrades the classification performance and is a challenge commonly encountered in real-world scenarios.

Test-time adaptation (TTA) offers an alternative approach for zero-shot transfer, which can effectively address the domain shift problem. Existing TTA methods can be divided into two types, i.e., training-required and trainingfree. Training-required TTA approaches [Feng et al., 2023; Yan et al., 2024; Yan et al., 2025; Qu et al., 2025a], represented by Test-time Prompt Tuning (TPT) [Shu et al., 2022a], fine-tune visual or textual representations via an unsupervised semantic consistency loss. Specifically, TPT fine-tunes the context of the prompt by constraining the consistency of augmented samples, which helps precisely retrieve the knowledge of Vision-Language Models (VLMs). Recent works have attempted to improve TPT, such as enhancing the diversity of augmented samples through generative models [Feng et al., 2023] and enriching text descriptions based on large language models [Yan et al., 2024].

However, training-required approaches involve significant computational overhead during inference, which hinders their practical applications in computation-limited downstream tasks. Training-free TTA methods [Udandarao et al., 2023;

Karmanov et al., 2024; Zhang et al., 2024b] typically use support set or cache to store prototypes for each category of the target domain, and then use these prototypes to adjust the predictions of the VLM, bridging the gap between the source domain and the target domain. The support set or the cache can be derived from generative models [Udandarao et al., 2023], historical test samples [Karmanov et al., 2024], or boosting samples [Zhang et al., 2024b].

Although these methods have made progress in the image domain, directly applying them to zero-shot video classification would encounter the following issues. i) How to represent a multi-frame video: Simply averaging multi-frame representations tends to introduce substantial noise, thereby compromising representation quality. Extracting only a single key frame, on the other hand, leads to the attenuation or loss of crucial temporal cues. ii) How to represent a complicated action: Although entropy-based cache update algorithms can select high-confidence samples, these samples tend to exhibit high visual similarity, making it difficult to construct a semantically diverse action feature bank.

To this end, we propose a reliable and diverse Hierarchical Adapter, which is the first training-free adapter for zeroshot video classification. This framework consists of two core modules. Frame-level Cache Refiner (FCR): To capture rich and effective motion features, we design a comprehensive frame selection strategy with two metrics: prediction entropy and temporal probability difference. Prediction entropy independently measures the model's confidence in the prediction results for each frame, while temporal probability difference assesses the model's sensitivity to temporal discriminative cues. We also propose a two-step top-k approach to effectively integrate them. Video-level Cache Updater (VCU): To represent the underlying data pattern, we introduce a diversity criterion to improve the cache update algorithm.

To summarize, we make the following contributions:

• We propose a criterion for selecting reliable and diverse video data that effectively captures the underlying data manifold, which is conductive to enhancing cache representations.
• We design a hierarchical adapter, a novel adaptation strategy in test-time adaptation of VLMs for video classification, which improves training-free dynamic adapters by progressively filtering out unreliable and redundant data at frame-/video- level.
• Extensive experiments over four benchmarks demonstrate that the reliable and diverse hierarchical adapter achieves superior performance while maintaining competitive computational efficiency.

2 RelatedWork

2.1 Zero-shot Activity Recognition

Zero-shot action recognition [Liu et al., 2011; Qian et al., 2022; Rasheed et al., 2023] refers to the process of identifying actions in videos without having seen any samples of those activities during model training, which is crucial for real-world applications with limited annotated data. Early studies mainly focus on designing semantic representation of actions. Many attempts have been made in this regard, such as using manually defined attributes to represent actions [Liu et al., 2011; Gan et al., 2016b], mining objects as attributes [Jain et al., 2015; Gan et al., 2016a; Gao et al., 2019], and utilizing word embeddings of action names or action descriptions as semantic representations [Qian et al., 2022; Mandal et al., 2019; Qin et al., 2017; Xu et al., 2017]. Differently, recent studies generally delve into adapting large pre-trained VLMs (e.g., CLIP [Radford et al., 2021]) to zero-shot video recognition. For instance, ViFiCLIP [Rasheed et al., 2023] fully tunes CLIP on videos with minimal design changes. ActionCLIP [Wang et al., 2023] introduces temporal encoder to strengthen the video representation. PromptCLIP [Ju et al., 2022] also adopts a lightweight Transformer on the top of the CLIP image encoder for temporal modeling.

2.2 Prompt-based Learning for VLMs

Prompt learning, derived from natural language processing, has been studied extensively to leverage the existing knowledge of VLMs to boost their generalization. CoOp [Zhou et al., 2022b], a typical example of prompt learning for VLMs, learns prompt context knowledge by inserting learnable vectors into the class embeddings and optimizing the tokens using supervised classification loss. CoCoOp [Zhou et al., 2022a] extends CoOp by conditioning the text prompts on image embeddings to solve the issue of overfitting. Although these methods have demonstrated significant performance improvements, their reliance on a large amount of training data from the target domain hinders their practical applications in downstream tasks. To this end, Shu et al. propose a new paradigm, test-time tuning (TPT) [Shu et al., 2022a], which optimizes prompts dynamically via an unsupervised semantic consistency loss during inference. TPT has attracted much attention and has been extensively explored in recent researches [Feng et al., 2023; Yan et al., 2024; Zhang et al., 2024a; Abdul Samadh et al., 2024]. For example, DiffTPT [Feng et al., 2023] leverages generative models (i.e., Stable Diffusion) to augment test images, making augmented views more diverse. PromptAlign [Abdul Samadh et al., 2024] extends TPT with the token alignment strategy, which enforces to bridge the data shift in the test data. DTSTPT [Yan et al., 2024] transfers TPT to the video domain, considering the diversity of motion semantics. Nevertheless, these methods require gradient descent during inference, which is computationally expensive and time-consuming, thereby conflicting with the principles of test-time adaptation. This paper seeks to achieve efficient test-time adaption by leveraging test samples cache.

2.3 Memory-based Learning for VLMs

In recent years, it has become a trend to apply memorybased learning to various tasks in computer vision [Zhang et al., 2022; Karmanov et al., 2024; Zhang et al., 2024b; Udandarao et al., 2023; Qu et al., 2024; Qu et al., 2025b] and natural language processing [Grave et al., 2017; Merity et al., 2016]. As a parameter-free technique, memory-based learning enhances test-time adaption by providing efficient inference. Tip-Adapter [Zhang et al., 2022] is the first to adapt VLMs via memory banks, in which a key-value cache model is constructed from the few-shot training set. SuS-X [Udandarao et al., 2023] forms a support set with the help of Stable Diffusion, which generates salient and informative support images. To overcome the unavailability of source data during zero-shot inference and alleviate data shift caused by generative models, TDA [Karmanov et al., 2024] designs a dualcache model constructed from reliable images and their corresponding pseudo-labels during testing. BoostAdapter [Zhang et al., 2024b] combines instance-agnostic historical cache in TDA with instance-aware boosting cache, achieving promising results. In this work, we extend memory-based learning to the video domain for zero-shot activity recognition.

3 Preliminary

3.1 Problem Statement

This work focuses on transductive test-time adaption for zeroshot video classification. Given a set of C class labels Y = {y1, y2, · · · , yC} and a sequence of N testing videos V = {v1, v2, · · · , vN}, the task of zero-shot video classification is to predict the label of each testing video v as ˆ yc ∈ Y while Y is unseen to the pre-trained model. Specifically, inference is performed under a transductive setting, prediction for i-th test sample vi may therefore depend on the representations and predictions of the first i - 1 samples.

3.2 Memory-based Adaption

Zero-shot Matching. Given an arbitrary pre-trained VLM (such as CLIP [Radford et al., 2021] and ViFi-CLIP [Rasheed et al., 2023]) consisting of a visual encoder Ev and a textual encoder Et, we first encode the test video vi as fvi = Ev(vi) and get text features (i.e., zero-shot classifier) as {fc|fc = Et(tc)}Cc=1 , where tc is a manual-crafted text prompt corresponding to class label yc, e.g., "a video of a person doing [class yc]." The zero-shot matching result of vi and yc can be denoted as

which represents the cosine similarity between multi-modal features after normalizing fc and fvi respectively. Furthermore, the complete output logits of the test video vi is p(vi) = [p1(vi), p2(vi), · · · , pC(vi)] ∈ RC .

Training-free Dynamic Adapter. To address the issue of data shift between source domain and target domain, a widely used approach is to construct a cache based on target domain, leveraging the knowledge of target domain data distribution to guide model inference [Zhang et al., 2022; Karmanov et al., 2024]. In few-shot action recognition, the support set is constructed using annotated training videos, while in zeroshot action recognition, the cache can be progressively built during inference, based on test samples and pseudo-labels. The prediction from the cache can be calculated as

where A(x) = exp(-β(1 - x)) represents the scaling function with a smoothing scalar β, Fcache denotes visual features of samples in the cache, and Lp is the corresponding pseudolabels in the form of one-hot vectors.

4 Reliable and Diverse (R&D) Hierarchical Adapter

Inspired by Training-free Dynamic Adapter (TDA) [Karmanov et al., 2024], this work aims to adapt VLMs during inference by leveraging knowledge from the cache. However, the cache in TDA adopts a simplistic approach to storing support samples, failing to capture complicated action semantics. Moreover, the entropy-based cache update strategy in TDA cannot ensure the diversity of support samples, which is contradictory to the inherently complex and diverse nature of motions in real-world scenarios. To this end, we propose a reliable and diverse hierarchical adapter to enable efficient and effective test-time adaptation with VLMs. As shown in Figure 2, R&D Hierarchical Adapter is mainly composed of Frame-level Cache Refiner (FCR) and Video-level Cache Updater (VCU). In FCR, each video sample is refined at frame level based on prediction entropy and temporal probability difference before entering the corresponding cache. In VCU, the cache is updated based on the principle of diversity.

4.1 Frame-level Cache Refiner

To enhance video representations, we propose a Frame-level Cache Refiner (FCR), which discards low-confidence frames based on prediction entropy and temporal probability difference. In TDA [Karmanov et al., 2024], a single entropy-based criterion is employed to measure sample reliability. Given a test video v, the prediction entropy of v is calculated as e = - ∑Cc=1 p( ˆ y = c|v)log p(ˆ y = c|v). Although frames with lower prediction entropy have a lower likelihood of causing error, entropy is not always reliable as a confidence metric under biased scenarios. To avoid selecting overconfident samples based on incorrect cues, DeYO [Lee et al., 2024] further introduce a probability difference metric to ensure sample reliability, which we refer to as spatial probability difference (SPD). SPD quantifies the influence of reliable static cues, such as structure information, on inference by measuring the pseudo-label probability difference between the original frame and its spatial-shuffled variant independently.

Given a test video v and corresponding spatial-shuffled video vs, SPD is calculated in a parallelized manner as

where c∗ = arg max c p(ˆ y = c|v) is the pseudo-label of v predicted by the VLM. The model is expected to be sensitive to discriminative factors, for which frames with higher dspatial are more reliable.

Algorithm 1 Reliable and Diverse Cache Update

Input: test video v, cache, pseudo-label of test video c∗ Parameter: cache size n, similarity threshold τ , Output: cache updated

• 1: if The cache of class c∗ is not full then
• 2: Add new sample to the corresponding cache.
• 3: else
• 4: for i = 1 to n do
• 5: Calculate similarity between v and i-th sample in the cache.
• 6: end for
• 7: if similaritymax > τ then
• 8: Sample fusion based on momentum update.
• 9: else
• 10: Remove the sample with the lowest confidence.
• 11: end if
• 12: end if
• 13: return cache updated

To further assess the model's sensitivity to temporal dynamic information, we design a temporal probability difference (TPD), which is shown in Figure 2 (b). TPD measures the influence of temporal dynamic cues on inference by calculating the pseudo-label probability difference between the original video and its temporal-shuffled version. Predictions with larger TPD are more likely to rely on temporal dynamic cues, indicating that the pseudo-labels are more reliable.

Given a test video v and its temporal-shuffled variant vt, TPD is obtained as

where c∗ = argmax c p(ˆ y = c|v) is the predicted category of the test video v.

Taking into account that VLMs exhibit bias in per-class accuracy, we select top-K confident frames for each video, instead of keeping all frames with confidence scores higher than a pre-defined threshold, which is a commonly adopted strategy [Lee et al., 2024]. For a test video v ∈ RT×d , the refined video v′′ ∈ RK2×d is obtained by

where v′ ∈ RK1×d serves as an intermediate result and is not utilized in the following stages.

4.2 Video-level Cache Updater

The cache stores prototypes of different activities as a database. During inference, the test video is used as a query to aggregate information from the cache via similarity-based retrieval. However, in zero-shot video classification, the model cannot access the ground truth labels of historical samples and must rely on pseudo-labels to construct the cache. This inevitably introduces noise, which negatively impacts the model's performance. Therefore, it is necessary to update the cache progressively during inference. In TDA, the cache is implemented as a priority queue, where entropy serves as the criterion for prioritization.

The semantics of the same activity are diverse in visual space, but samples with high reliability are often visualsimilar, which hinders the cache in capturing the underlying data manifold. To this end, we propose Video-level Cache Updater (VCU) to maintain a diverse cache.

Specifically, when the test video v is added to the cache, the prototype most similar to v is updated as

where µ ∈ [0, 1] is the momentum coefficient, fv is video embeddings of v, and q is the prototype most similar to v.

In the image domain, the similarity between two images can be measured by calculating the cosine similarity. However, in the video domain, temporal sequence matching is required to evaluate the similarity between videos. We follow [Haresh et al., 2021] to use dynamic time warping (DTW) to compute the temporal sequence similarity. Given two video features fv and fw to be matched, we can obtain the similarity matrix S ∈ Rn×m based on cosine similarity, where S(i, j) = fvi · fwj . DTW can adaptively find the path with the highest similarity in S. Let the similarity between the first i frames of fv and the first j frames of fw be denoted as Sseq(i, j), then the following state transition equation can be established.

the similarity between fv and fw is given by Sseq(n, m).

For clarity, we provide the whole cache update process in Algorithm 1 in the form of pseudo-code.

5 Experiments

5.1 Datasets

HMDB-51 [Kuehne et al., 2011] is a small-scale action recognition dataset. It contains around 7,000 labeled videos sourced from YouTube, covering 51 activity categories. UCF-101 [Soomro, 2012] consists of 13,320 videos covering 101 categories, which can be further grouped into five main categories: Body motion, Human-human interactions, Human-object interactions, Playing instruments, and Sports. Kinetics-600 [Carreira et al., 2018] is a large-scale video dataset, containing 600 human action classes, with at least 600 video clips for each action. Each video is collected and annotated from YouTube and lasts approximately 10 seconds. ActivityNet-200 [Fabian Caba Heilbron and Niebles, 2015] is also a large-scale action recognition benchmark, but it provides about 20k untrimmed videos of 5 to 10 minutes from 200 activity categories.

5.2 Implementation Details

We utilize a pre-trained ViT-B/16 of CLIP as the foundation model, and the model is not fine-tuned on extra large video datasets. In test-time adaption, we sample T = 32 frames from each test video. We use top-1 accuracy(%) as our evaluation metric. We perform a search for hyperparameter on the validation set of Kinetics-400. In FCR, we select 8 frames based on prediction entropy, and subsequently select 5

frames based on TPD to construct refined video embeddings. When calculating TPD, each frame is divided into 7×7 image patches, and temporal shuffling is applied between adjacent 2 frames. In Algorithm 1, cache size n is set as 10 and similarity threshold τ is 0.95. In Eq. 2, β is 8 according to TDA, and in Eq. 7, µ is set to 0.5. All the experiments are conducted using a single NVIDIA 3090 24GB GPU.

5.3 ComparisonWith Other Methods

We conduct a comprehensive comparison of the proposed Hierarchical Adapter with popular zero-shot video classification approaches spanning various methodological categories. Specifically, uni-modal zero-shot video recognition models are trained on video data with elaborated representation engineering. Adapting pre-trained CLIP involves additional temporal learners or vision-language prompting techniques without training the encoders while tuning pre-trained CLIP means fully fine-tuning the CLIP model via video data. Following [Rasheed et al., 2023], we report the mean and standard variance of the results.

As we can see in Table 1, our method surpasses conventional uni-modal zero-shot video recognition methods, e.g., ER-ZSAR [Chen and Huang, 2021], JigsawNet [Qian et al., 2022], and ResT [Lin et al., 2022], by a significant margin on all benchmarks. Our approach also outperforms models such as Vita-CLIP [Wasim et al., 2023] and VicTR [Kahatapitiya et al., 2024] that adapt pre-trained CLIP. Compared with ViFi-CLIP [Rasheed et al., 2023], which serves as a baseline for our method and fine-tunes the pre-trained CLIP on Kinetics-400, our method is also superior.

5.4 Ablation Study

Component analysis. To verify the effectiveness of the proposed FCR and VCU, we conduct ablation experiments on HMDB-51 and Kinetics-600 benchmarks. As shown in Table 2, FCR brings 2.7% and 1.9% performance gains on HMDB51 and Kinetics-600, respectively, which indicates that FCR can focus on reliable frames. In addition, VCU improves on the two datasets by 2.3% and 0.9%, respectively, which suggests that VCU can retain diverse video samples in the cache. Moreover, by combining the two modules, our full model achieves better results, confirming the complementarity and effectiveness of the proposed framework.

Reliabilitymetric. We examine the impact of the proposed temporal probability difference by contrasting it with the model with prediction and spatial probability difference or with prediction entropy only. A shown in Table 3, temporal probability difference achieves better performance, while spatial probability difference has almost no effect. This study confirms the complementarity between prediction entropy and temporal probability difference.

Diversity strategy. To investigate the effectiveness of diversity criterion, we compare it with the cache updater without feature fusion. As shown in Table 4, our proposed feature fusion rule brings 0.9% and 0.6% performance gains on HMDB-51 and Kinetics-600, respectively, indicating that the cache we construct better represents motion semantics.

Table 6: Top-1 accuracy(%) on HMDB-51 and Kinetics-600 using different VLMs, i.e., Vanilla CLIP [Radford et al., 2021] and ViFiCLIP [Rasheed et al., 2023].

Temporal sequence matching. To validate the superiority of Dynamic Time Warping, we compare it with various temporal sequencematching strategies. InMax strategy, themaximum of the similarity matrix S is taken as the final result. In Mean setting, the average of S is the similarity between the two videos. In Diagonal strategy, the diagonal of the similarity matrix S is involved in temporal sequence matching. The optimal path from the top-left corner to the bottom-right corner of S is found by DTW to measure the similarity between the two videos. The results are provided in Table 5, which indicates Dynamic Time Warping outperforms the other three strategies.

Generalization to different pre-trained VLMs. We use Vanilla CLIP with ViT-B/16 as the VLM of choice throughout our ablation studies. In Table 6, we demonstrate results when our proposed Hierarchical Adapter is applied on top of ViFi-CLIP and three versions of Vanilla CLIP. Hierarchical Adapter improves model performance on both HMDB-51 and Kinetics-600, indicating the proposed framework can be applied to an arbitrary VLM.

Sensitivity to test time sample order. As Hierarchical Adapter classifies videos online, the performance of the model is inevitably influenced by the order of the test videos. To investigate this influence, we conduct five rounds of experiments on HMDB-51 and Kinetics-600, with each round employing a different sample order. As we can see in Figure 3, the model's performance is slightly influenced by the sample order fluctuations on both datasets. Regardless of the sample order, the performance of our proposed model consistently surpasses Vanilla CLIP.

Visualization. In Figure 4, we apply t-SNE to visualize the stored video features in the cache under the framework of TDA [Karmanov et al., 2024] and our proposesd Hierarchical Adapter on the HMDB-51 [Kuehne et al., 2011] dataset. The stored video features are highlighted using different colors while the others are marked in gray. The visualization results indicate that Hierarchical Adapter is able to construct and update reliable and diverse prototypes to represent motion semantics.

6 Conclusion

In this work, we propose an effective and efficient Hierarchical Adapter, which is the first training-free test-time adapter for zero-shot video classification. This framework aims to select reliable and diverse visual features at frame level and video level, which consists of two core modules: 1) Framelevel Cache Refiner for selecting rich and effective motion features; 2) Video-level Cache Updater for capturing the underlying data manifold. Experimental results on four video classification benchmarks demonstrate the superiority of our Hierarchical Adapter against existing methods. In future work, exploring the potential of leveraging the rich text representations provided by large language models to enrich the cache holds promising prospects.

Acknowledgments

The work is supported by the National Natural Science Foundation of China (Grant No. 62222207, 62427808, 62472208), and the open Foundation of the Key lab (center) of Anhui Jianzhu University Anhui Province Key Laboratory of Intelligent Building & Building Energy Saving (Grant No. IBES2024KF02). Wenxuan Ge and Peng Huang equally contributed to this work, Xiangbo Shu and Rui Yan are the equal corresponding authors.

Figure 4: t-SNE visualizations of the stored video features in the cache under the framework of TDA (left) and our proposed Hierarchical Adapter (right).