DTS-TPT: Dual Temporal-Sync Test-time Prompt Tuning for Zero-shot Activity Recognition

1 Introduction

Since the era of deep learning, activity recognition technology has developed rapidly and achieved significant improvements in predefined categories such as K400 [Kay et al., 2017] and K600 [Carreira et al., 2018]. However, due to the diversity of human activity, it is difficult for humans to list all human activity categories in the real world completely. Therefore, it is urgent to develop zero-shot algorithms to predict unseen but known activity categories, i.e., zero-shot activity recognition [Brattoli et al., 2020; Liu et al., 2011] via matching the video and class name feature directly. This research has made progress in recent years but is still insufficient, mainly hindered by challenges such as i) Intra-modal Distribution Shift [Lin et al., 2022]: Within each modality, visual/textual features of each semantic should be clear and distinguishable; ii) Inter-modal Semantic Gap [Lin et al., 2022]: Between two modalities, the feature distribution of visual samples and labels with the same semantics should be more aligned/consistent.

Existing solutions can be divided into two types, i.e., training-based and training-free. The training-based methods tune an off-the-shelf visual encoder on a certain scale of video data for adapting to zero-shot activity recognition. To enhance the discriminability of features from each modality (visual or text), previous works build fine-grained motion features from the objects or attributes in videos [Gao et al., 2019; Liu et al., 2011], or enhance activity class name embeddings with object tags [Gao et al., 2019] or elaborative descriptions [Chen and Huang, 2021]. To facilitate video-text semantic alignment, some recent works [Lin et al., 2022; Wang and Chen, 2017] try to convert text features of unknown class names into the video feature space according to the knowledge association between known and unknown class names.

However, with the rise of large Vision-Language models technology, intra-modal representations are becoming more robust, and the inter-modal semantic gap (between visual and text) is gradually narrowing. Thanks to this, many trainingfree methods built on pre-trained large visual and text models have recently emerged in zero-shot visual classification. The training-free method usually fixes the visual embedding, and then i) replaces the fixed class embedding with a more flexible support set [Zhang et al., 2022b; Udandarao et al., 2023]; or ii) tunes the class embedding in a self-supervised manner on a single sample during the test time for better alignment (Testtime Prompt Tuning, TPT [Shu et al., 2022]), as shown in Figure 1 (a).

Without a doubt, we can apply these works to zero-shot activity recognition but will encounter the following issues. i) The samples of the support set are retrieved from the large database or generated via large generative models according to class names, in which the quality of visual samples is limited and the cost of video data generation is expensive; ii) TPT relies heavily on high-quality augmentation views to calibrate the embedding of class labels. However, on the one hand, it is difficult to mine rich and effective motion semantics via directly extending spatial augmentation to video data with high redundancy; on the other hand, simple class names are insufficient to describe human activity semantics, which may limit the effectiveness of the semantic consistency objective.

To this end, we propose a Dual Temporal-Sync Testtime Prompt Tuning framework that tunes the class embedding with different temporal structures in multiple steps supervised by dual semantic consistency, as shown in Figure 1 (b). This framework consists of the following two modules. Temporal-Sync Test-time Prompt Tuning (TS-TPT): To capture rich and effective motion features, we append learnable prompts to the original class embedding and tune these prompts with different temporal scale video sequences in multiple steps. Specifically, we build a set of video augmentations via spatial AugMix [Hendrycks et al., 2019] and then select high-confident augmentations for further tuning. We sample frames from selected video augmentations synchronously at the same scales in a single step but at different scales in different steps. Details are in Sec 4.1. Dual Semantic Consistency (DSC): To more accurately calibrate the motion semantic in the language space, we adopt both original class names of human activities and the corresponding explanations enquired from LLMs for semantic consistency. Learnable prompts are equipped and optimized via different backward propagation solutions including Parallel, Hybrid, and Switch Backward. Details are in Sec 4.2. We extensively evaluate the proposed framework on various benchmarks and quantitative and qualitative results show its substantial improvements and strong interpretability.

2 RelatedWork

2.1 Zero-shot Activity Recognition

Zero-shot action recognition aims to identify novel action classes that do not appear during model training, which is more appropriate for practical applications. The typical pipeline of zero-shot matching [Brattoli et al., 2020] extracts the video and class text features and then computes the distance between them in the feature space for final prediction. To build robust representation, earlier works represent the video via handcraft attributes [Liu et al., 2011] or building knowledge graphs on object tags [Gao et al., 2019], and enhance the class text features with the help of object tags [Gao et al., 2019] or elaborative descriptions [Chen and Huang, 2021] retrieved from dictionaries or Wikipedia according to class names. Some recent works adapt the large pre-trained model (e.g., CLIP) to zero-shot activity recognition via fully tuning (e.g., ViFi-CLIP [Rasheed et al., 2023]) or partial visual prompt tuning (i.e., ActionClip [Wang et al., 2021]) or partial visual/text prompt tuning (i.e., XCLIP [Ni et al., 2022]). The above works still need to be fine-tuned on lots of video samples during the training phase. However, this work is the first to append several learnable prompts to the label and tune them via a single sample during inference.

2.2 Prompt Tuning for VLMs

In recent years, it has become a trend to adapt VisualLanguage Models (VLMs) pre-trained on large-scale visualtext corpora to various tasks in computer vision [Radford et al., 2021; Chen et al., 2020]. As a heuristic way, prompt tuning appends some learnable prompts to inputs and then tuning them on the few training samples (training-time tuning) [Zhou et al., 2022c; Zhou et al., 2022b; Gao et al., 2023] or a single test sample without annotation (test-time tuning) [Shu et al., 2022]. For instance, CoOp [Zhou et al., 2022c] inserts a set of learnable vectors to class embedding in different positions as a prompt context which is directly optimized by the classification loss. To avoid the over-fitting issue, CoCoOp [Zhou et al., 2022b] is proposed to learn a lightweight network to make the prompt conditioned on model inputs. Inspired by this, recent work [Sun et al., 2022] decomposes backbone parameters into successive matrices and efficiently tunes part of them for few-shot visual recognition, achieving promising results. To get rid of annotation cost, some recent works [Huang et al., 2022; Zhou et al., 2022a] optimize the prompts in an unsupervised manner. However, this line of work still requires downstream training data, which is not compatible with the zero-shot setting. To this end, Test-time Prompt Tuning (TPT) [Shu et al., 2022] is proposed to learn adaptive text prompts dynamically with a single test sample during inference, which reduces the cost of labeling and calculation. In this work, we extend TPT to the video domain, taking into account the diversity of motion semantics, for zero-shot activity recognition.

2.3 Test-time Tuning

Test-time Tuning (TTT) seeks to leverage unlabeled test data to enhance the model's generalization capabilities for the target task in the presence of data distribution shifts, which has been well explored for several applications [Shocher et al., 2018; Nitzan et al., 2022; Xie et al., 2023]. The key to TTT is to formulate an effective test-time objective. In the image domain, previous works apply rotation prediction [Sun et al., 2020; Liu et al., 2021] or image reconstruction [Gandelsman et al., 2022; Wang et al., 2023] as the self-supervision task to optimize the model during inference. Besides, TENT [Wang et al., 2020] utilize entropy minimization [Roy et al., 2019; Saito et al., 2019] to tune the parameters of BN layers to overcome the distribution shift [Zhang et al., 2022a]. To extend TENT [Wang et al., 2020] to scenarios involving a single test sample, MEMO [Zhang et al., 2022a] proposes utilizing data enhancement methods to generate diverse enhanced views from the single test sample for test-time tuning. In the video domain, recent works attempt to extend TTT to video data via self-supervised dense tracking [Azimi et al., 2022] or segmentation on video streams [Wang et al., 2023]. However, the optimization cost of tracking and segmentation is higher than entropy minimization (on class predictions). Therefore, this work applies entropy minimization as the objective function for prompt tuning but optimizes it at different scale video sequences in multiple steps.

3 Preliminary

3.1 Problem Statement

Given a set of N class labels Yu = {y0, y1, · · · , yN}, the goal of zero-shot activity recognition is to predict the label of a testing video sequence Vtest as yi ∈ Yu. Specifically, a zero-shot pipeline i) adopts pre-trained models Fvis and Ftxt respectively to encode the input video and class labels into visual and textual features with the same dimension, ii) then calculates the distance among them in feature space, and the nearest category is regarded as the prediction result.

Notably, the conventional methods finetunes Fvis and Ftxt on an extra large video dataset (e.g., K400) with the labels of Ys and have the constraint of Ys ∩Yu = ∅. However, inspired by [Shu et al., 2022], this work focuses on directly applying an arbitrary VLM pre-trained model (such as CLIP [Radford et al., 2021] and BIKE [Wu et al., 2023]) to identify the human activity in downstream tasks, which gets rid of training data and is more practical.

3.2 Test-time Prompt Tuning

Zero-shot Matching. Given a test video Vtest and all class labels Yu, we first encode them as f vis = Fvis(v) and f txt i = Ftxt(yi). For zero-shot visual recognition, we calculate the probability that the test video v belongs to category yi as

Here, sim(·) is usually implemented by computing cosine similarity between the two feature vectors, N is the total number of class labels, and τ is the temperature parameter.

Prompt Tuning. Large pre-trained models have strong semantic representation capabilities but still need to be further generalized to downstream tasks that may contain outof-distribution samples. For zero-shot testing, a common method is to add learnable prompts to class labels and tune these prompts with only a single sample in a self-supervised manner, which is called Test-time Prompt Tuning [Shu et al., 2022]. The optimization objective can be formulated as

where F is a pre-trained model consisting of visual and textual encoders. p is the learnable prompts appended to the tokenized text labels as {p, yi}.

In the zero-shot inference phase, only one unlabeled test video is given to match all class labels, thus we can only optimize the above objective function in a self-supervised manner. Hence, Shu et al. implemented the loss function L via minimizing the averaged entropy of predictions from K augmented views as follows.

Here, AugSj denotes the random spatial augmentation and pp(yi|Augj (Vtest)) is the prediction probabilities calculated between the j-th video augmentation and i-th class label with the prompt p.

To reduce the noise from random augmentations, Shu et al. [Shu et al., 2022] further select confident samples whose prediction entropy is lower than the threshold θ fromK augmentation views as

Here, H calculates the self-entropy of the prediction on an augmented view and 1[·] is the indicator function.

4 Methodology

Inspired by Tese-time Prompt Tuning (TPT), this work aims to tune the embedding of activity labels for zero-shot activity recognition during test time. However, monotonous spatial augmentation and short class names used in the standard TPT framework are insufficient for tuning complicated motion semantics in the language space. To this end, this work proposes a Dual Temporal-Sync Test-time Prompt Tuning (DTS-TPT) framework consisting of two core modules (i.e., Temporal-Sync Test-time Prompt Tuning (TS-TPT) and Dual Semantic Consistency (DSC)) as shown in Figure 2. TS-TPT builds multi-scale video features from all spatialaugmented videos synchronously and then tunes the prompts with video features with different temporal scales in multiple steps. In each tuning step, DSC minimizes the average entropy of predictions from all augmented samples with both class names and descriptions generated by LLMs [OpenAI, 2023].

4.1 Temporal-Sync Test-time Prompt Tuning

As mentioned above, the key to TPT is augmenting the test visual sample into multiple views for tuning class embedding. For the sake of clarity, we firstly rewrite the standard TPT (i.e., Eq. (2)) as

where F(AugS (vtest))) denotes that TPT performs spatial augmentation AugS on the test video and extracts the visual features from them via the pre-trained model F.

Based on this, how to augment the video data is the difficulty in adapting TPT to zero-shot activity recognition. A

Algorithm 1 Pseudocode of DTS-TPT

• 1: Input: a single test video vtest, class names {yj}, learnable prompts p, video encoder Fvideo, and text encoder Ftext .
• 2: Perform spatial augmentation on the test video K times and extract the visual features as xSk = Fvideo(AugSk (vtest)), thus we get a set of augmented video features XS = {xS1 , xS2 , xSK }.
• 3: Extract text features xtext j = Ftext({p, yj}).
• 4: for m= 1, 2, · · · ,M do
• 5: Temporal augmentationXTm = AugTi (XS ) ∈ RK×d ;
• 6: Calculate the predictions pp ∈ RK×N based onXTm
• 7: Select confident ones ˆ pp ∈ RK′ ×N with Eq (5);
• 8: Minimize the averaged entropy of ˆ pp with Eq (3)
• 9: Perform backward propagation for p.
• 10: end for

video. Thus, there are two core design points, i) Tuning Times: How many times should we tune? ii) Tuning Structure: What temporal structure (i.e., AugT) should be used each time?

For tuning structure, we gradually sample frame-level features from the sequence via different strategies, including Random Sampling, Center Sampling, Left-side Sampling, and Right-side Sampling, as shown in Figure 4. Based on this, we try different tuning times in the experiment, details in Sec-

straightforward approach is to perform spatial (AugS ) and temporal (AugT) augmentation on the video data and combine them for a single-step optimization of TPT as

However, spatio-temporal augmented videos are usually different from the original video and they should not be directly used for consistency constraints, as shown in Table 5.

To this end, we designed a multi-step optimization strategy with temporal synchronization, i.e., the same temporal structure should be used in single-step optimization, but different temporal structures should be used in different steps.

Single-step Tuning. Formally, we sample the video features extracted by F(AugS (vtest))) with different temporal scales and tune the prompt in a single step as

Here, AugT aims to further augment the given spatial views in the temporal domain, and can essentially be viewed as a Temporal Sampling function. We implement AugT via Videoversion AugMix [Hendrycks et al., 2019] which simply extends the spatial AugMix technology to the spatial-temporal data.

Multiple-step Tuning. Given a test video, we aim to tune the learnable prompt multiple times by video sequences with different temporal structures augmented from the original test tion 5.4. To facilitate the understanding, we present the whole tuning process in Algorithm 1 in the form of pseudo-code.

4.2 Dual Semantic Consistency

Since the semantics of the same behavior are diverse in visual space, it is insufficient to describe human behaviors with short class names. A straightforward solution is to use dictionaries or handcrafted templates to empirically enhance these class names, but this will ultimately be limited by the size of the predefined knowledge database. This work aims to apply Large Language Models to explain the meaning of each human behavior in detail, which will greatly facilitate crossmodal alignment of behavioral semantics.

Specifically, we ask LLMs the question of "If you're an expert in video classification. Please answer the following questions in one or two sentences. What does [CLASS NAME] mean?". After that, we clean up the noisy words in the answers from LLMs and limit the descriptions to 77 words (max-length limit of Tokenizer). The explanation of each CLASS NAME generated by LLMs can be found in Supplementary Materials.

Moreover, how to optimize the tuning objective defined in Eq. (8) with the class name and the corresponding explanations becomes an issue. In this work, we provide three different backward propagation for semantic consistency, including Parallel, Hybrid, and Switch Backward. Notably, Parallel Backward tunes two sets of learnable prompts separately, but Hybrid and Switch Backward tunes the same prompts serially or alternately, as shown in Figure 3. Comparison results can be found in Table 6.

5 Experiments

5.1 Datasets

HMDB-51 [Kuehne et al., 2011] contains 51 activity categories and 6, 766 manually annotated videos extracted from various sources ranging from digitized movies to YouTube. UCF-101 [Soomro et al., 2012] collects a total of 13, 320 real activity videos in 101 categories from YouTube, totaling more than 27 hours. All categories can be classified into 5 types: Body motion, Human-human interactions, Human-object interactions, Playing instruments, and Sports. Kinetics-600 [Carreira et al., 2018] is an extension of the Kinetics-400 [Kay et al., 2017] and consists of approximately 480K videos covering 600 categories. Each video is a 10second activity clip annotated from the original YouTube video. ActivityNet [Caba Heilbron et al., 2015] provides a large-scale video dataset covering the 200 actions most relevant to human daily life. It contains a total of 19, 994 untrimmed videos, with 137 videos per category.

5.2 Setup

We sample T = 8 frames from test video and augment them K - 1 (K = 32) times using AugMix [Hendrycks et al., 2019]. We create 3 learnable tokens as text prompts and initialize them as "an action of". We employ CLIP with ViTB/16 as the visual encoder and use the corresponding textual encoder of CLIP for text encoding. Our model does not need any training data such as K400. For each inference, we compute predictions based on a batch of 32 augmented views (including the original one) and then select top 20% confident predictions for further optimization. We adopt the AdamW optimizer with a learning rate of 0.001.

5.3 Comparisons with State-of-the-art Methods

We compare the proposed DTS-TPT with existing zero-shot video activity recognition methods on four benchmarks with 8 frames, as shown in Table 1. Notably, DTS-TPT does not need training data but still achieves better generalization when utilizing a pre-trained model such as BIKE [Wu et al., 2023]. Following [Ni et al., 2022; Rasheed et al., 2023; Lin et al., 2023], we report the mean and standard deviation of results on three official validation sets.

Our approach outperforms uni-modal zero-shot activity recognition methods by a significant margin, such as ERZSAR and JigsawNet, which train on Kinetics-400 and need crawled descriptions of activity classes with manual correction. Moreover, DTS-TPT is superior to recent methods, either adapted directly from CLIP or tuned from CLIP with K400 for zero-shot recognition. As a baseline for our approach, BIKE fine-tunes the pre-trained CLIP model with video attributes and activity categories. Our approach achieves state-of-the-art results on HMDB51, UCF101, Kinetices-600, and ActivityNet in terms of top-1 accuracy, surpassing BIKE by a significant margin. Besides, the proposed framework is compatible with different VLMs with different backbones, as shown in Table 3.

5.4 Ablation Study

Component Analysis. To illustrate the effectiveness of each component of our approach, we conduct ablation experiments on HMDB51 and UCF101 datasets. As shown in Table 2, the proposed modules (TS-TPT and DSC) bring obvious gains respectively, and it is better to use them together, which indicates that they are complementary.

Temporal-Sync Sampling. As shown in Figure 4, we have designed four different temporal sampling strategies for TSTPT. For a fair comparison, all the variants employ Vanilla CLIP as a visual encoder and sample 6 frames out of 8, and perform TS-TPT twice. From Table 4, we find that the results of three non-random sampling methods are slightly better than random sampling. It indicates that TS-TPT is not affected by the temporal sampling strategy.

ComparisonwithTPT. We compare the proposed TS-TPT with the previous standard TPT in Table 5. Based on the results, we can conclude the following: i) Applying simple spatio-temporal augmentation (ST-TPT) cannot bring improvements, compared with TPT. ii) The performance of the TS-TPT is gradually improved with the number of optimization steps increasing from 1 to 3. This indicates that tuning prompts collaboratively with video features at different temporal scales is more likely to approximate the latent semantic distribution of labels. iii) Depending on the original design, TPT can be optimized repeatedly using the same scale video features, but more update steps do not lead to better performance, and sometimes even worse.

Context for Text Prompts. We investigate the effect of different context (class name and explanation) for text prompts in Table 6. The explanation generated through LLM significantly improves performance compared with using class names alone, suggesting that fine-grained text context helps prompt tuning of diverse motion semantics.

Explanation from Different LLMs. We study the effect of description generated by different LLMs in Table 7. We observed that LLM with a larger number of parameters (i.e., Gemini and GPT 3.5) resulted in a more significant performance improvement. Notably, the description generated by GPT-3.5 brings the best overall performance, thus we used it for the final model.

Backward Types of DSC. We have designed three backward types of dual semantic consistency (as shown in Figure 3). As reported in Table 6, Switch Backward cannot combine the benefit from two types of semantic consistency (class name and explanation), but optimizing the Dual Semantic Consistency via Parallel and Hybrid Backward has a better performance compared to single semantic consistency, Therefore, we used Hybrid Backward for the final model.

Visualization. We apply t-SNE to visualize the original features and tuned features of class names and their corresponding explanation generated by GPT 3.5, as shown in Figure 5. At the same time, we calculated the standard deviation of the data points in different feature distributions. We found that the standard variance of the features increases significantly and some confused features (in red circles) no longer exist after tuning, indicating that divergent feature distribution is more conducive to cross-modal alignment.

6 Conclusion

This work has managed to adapt Test-time Prompt Tuning (TPT) to zero-shot activity recognition by fully exploiting the temporal characteristics and semantic diversity of human behaviors. We developed the Dual Temporal-Sync Testtime Prompt Tuning framework which tunes the prompts in multi-steps on different temporal scales video features with dual semantic consistency. Extensive experiments on various benchmarks demonstrated that DTS-TPT can effectively improve the generalization ability of the pre-trained model on video data without training data/annotation and is compatible with various VLMs. In the future, it is critical to improve the efficiency of multi-step tuning and class name explanation in practical applications.

Acknowledgments

The work is supported by the National Natural Science Foundation of China (Grant No. 62302208, 62222207, 62072245, and 61925204), the Natural Science Foundation of Jiangsu Province (Grant No. BK20211520), the Postdoctoral Fellowship Program of CPSF (Grant No. GZB20230302), the China Postdoctoral Science Foundation (Grant No. 2023TQ0151 and 2023M731596), the Jiangsu Funding Program for Excellent Postdoctoral Talent (Grant No. 2023ZB256), and the Fundamental Research Funds for the Central Universities (Grant No. 14380109 and 14913211). Rui Yan and Hongyu Qu equally contributed to this work, and Wenbin Li is the corresponding author.