The Role of Video Generation in Enhancing Data-Limited Action Understanding

1 Introduction

Over the past decade, deep learning [LeCun et al., 2015; Miech et al., 2019] has brought remarkable progress benefiting from large-scale annotated data [Carreira et al., 2018; Sultani et al., 2018], especially in the computer vision community such as CLIP with 400M image-text pairs [Radford et al., 2021]. However, obtaining high-quality datasets is often complex, time-consuming, and costly [Carreira et al., 2018; Grauman et al., 2022], requiring a large amount of manual annotation. In particular in the field of video understanding, human annotations can be inaccurate (50% samples are not aligned in the HowTo100M dataset [Miech et al., 2019]). A lack of a well-annotated dataset with sufficient samples will limit the learning of models and result in poor generalization ability, facing challenges in real-world applications such as abnormal action detection [Sultani et al., 2018; Lv et al., 2023] and long-tail action recognition [Perrett et al., 2023; Grauman et al., 2022].

� Corresponding author.

In order to solve the data-limited problem in video understanding, previous methods often focus on designing data augmentation strategies [Yun et al., 2020; Kim et al., 2020; Li et al., 2022b] and transfer knowledge [Kim et al., 2025; Li et al., 2024; Rasheed et al., 2023; Luo et al., 2023] from other modalities, such as images, where data is easier to collect. However, we argue that these solutions are suboptimal as a) data augmentation does not create novel semantics for general knowledge learning [Trabucco et al., 2024], and b) knowledge learned from other modalities does not provide necessary information for video understanding, specifically, images do not include temporal relations between video frames.

With the development of diffusion models [Yang et al.,

2024; Liu et al., 2024; Esser et al., 2023], existing methods propose to synthesis datasets to improve model learning [He et al., 2023; Tian et al., 2024; Luo et al., 2024; Feng et al., 2023]. There is a problem in the utilization of generated samples for training video understanding models, manifested by the deficient information within the samples, which leads to suboptimal training efficiency. As illustrated in Figure 1 (a) and (b), our finding is that due to the complex and rich information of real videos, real samples have higher intra-class distances and lower inter-class distances than synthetic samples; while the videos generated by the diffusion transformer tend to include information with simple and similar content, which limits the learnable information contained in the synthetic samples . This also makes the classification difficulty of synthetic samples lower than that of real samples.

To solve the problem of deficient information within generated samples, we propose an information enhancement strategy to enhance the effective information of synthetic samples, which refers to context information conducive to action understanding. This strategy injects various character and environment information that conforms to specific human actions into the generated video and alleviates the domain gap between synthetic and real samples. Due to the limited capabilities of the text2video model, some synthetic videos perform unsatisfactorily and contain less effective semantic information, as shown in Figure 1 (c). To alleviate the impact of low-quality synthetic samples on model training, we propose uncertainty-based label smoothing. We calculate the uncertainty of the generated samples with CLIP to measure their quality and adjust the smoothness of label smoothing with uncertainty, which can prevent the model from overfitting lowquality samples and alleviating its impact.

We conducted extensive experiments on four datasets (Kinetics-600, UCF-101, HMDB-51, UCF-Crime) across five tasks (few-shot, zero-shot, base-to-novel, long-tail, abnormal action detection) and demonstrated the effectiveness of our method in data-limited action understanding. We achieved state-of-the-art performance for zero-shot action recognition tasks on Kinetics-600, UCF-101, and HMDB-51.

In summary, our contributions are as follows:

1. To the best of our knowledge, we are the first to investigate the effectiveness of data generated by video diffusion transformers for enhanced data-limited video understanding that includes action recognition and action detection tasks.
2. To solve the problem of deficient information within generated samples, we propose two strategies: the information enhancement strategy and the uncertainty-based label smoothing. These two strategies significantly improve the training efficiency of the generated samples.
3. We conduct extensive experiments on four datasets across five data-limited action understanding tasks to demonstrate the effectiveness of our proposed method, and our method achieves SOTA performance for zero-shot action recognition.

2 RelatedWork

2.1 Training with Synthetic Samples

The methods of training with synthetic samples can be broadly categorized into two main streams: generative-based and graphics engine-based methods. The graphics enginebased methods use real objects' 3D models to arrange, reconstruct, or move them according to certain rules and then render them to generate training samples. It has found usage in a myriad of fields, including object detection [Gaidon et al., 2016; Cabon et al., 2020], optical flow estimation [Dosovitskiy et al., 2015; Kim et al., 2022], auto driving [Dosovitskiy et al., 2017], etc.

With the explosion development of AIGC, image generation models have become of sufficient quality to generate training samples. Earlier works on image understanding have concentrated on the generation of samples or data augmentation with GANs [Baranchuk et al., 2021; Li et al., 2022a]. But with the emergence of the diffusion model, many methods focus on generating samples using diffusion models given its realistic, high-quality and controllable generative ability. [He et al., 2023] found that diffusion model synthetic samples can significantly improve the classification results of zero-shot, few-shot and transfer learning, and designed strategies to filter the noise of synthetic samples and enhance their diversity. [Feng et al., 2023] proposes a novel test-time prompt tuning method which leverages diffusion models to generate augmented data. [Zhou et al., 2023] proposes diffusion inversion, which invert images to the diffusion model's latent space, and generates novel samples by conditioning the generative model. [Feng et al., 2024] fine-tunes a diffusion model to enhance the capability of the object detection model. [Wang et al., 2024] studies the reasons why synthetic samples sometimes harms contrastive learning performance from the perspective of data inflation and data augmentation, and proposes adaptive inflation to improve the contrastive learning. [Trabucco et al., 2024] uses a diffusion model to edit and change the semantics of the image to address the lack of diversity in previous image augmentation methods.

Within the realm of video understanding, most works still rely on graphics engines or GANs [Guo et al., 2022; Kim et al., 2022]. Unlike these previous works, this paper focuses on leveraging text2video diffusion transformers to improve datalimited action understanding that includes action recognition and action detection tasks. To the best of our knowledge, no prior research has delved into this domain.

2.2 Text-to-Video DiffusionModels

In recent years, text-to-video generation has undergone significant advancements, particularly driven by the rapid development of diffusion models. Previous video generation models based on diffusion typically followed a UNet-based architecture. Video Diffusion Models (VDMs) [Ho et al., 2022] extend image synthesis diffusion models to video generation by training jointly on both image and video data. Gen-1 [Esser et al., 2023] presents a structure and contentaware model that modifies videos guided by example images or texts. With the impressive capabilities demonstrated by Sora [Liu et al., 2024], which uses Transformer as the backbone of diffusion models, i.e. Diffusion Transformers (DiT), has gradually become mainstream. A series of DiTbased works have emerged, including CogVideoX [Yang et al., 2024] and Hunyuan Video [Kong et al., 2024]. These works are capable of generating high-resolution videos with coherent actions, which makes it possible to apply the generated long-tail video data for downstream task training. In this paper, we utilize CogVideoX [Yang et al., 2024] to generate all training samples because it performs well in generating human-related videos and includes rich motion information.

3 Method

3.1 Overview

We address the data-limited action understanding problem by bridging the data scarcity. To this end, we adopt the text2video model to generate the required data according to the target task. Our approach consists of two steps: 1. Generate a video dataset with a text2video model based on given labels; 2. Train the action understanding model with the generated video dataset, as shown in Figure 2 (a). This paradigm is concise yet general and can produce infinite-scale annotated data without human intervention.

3.2 Video Sample Generator

We employ the text2video model to solve the problem of datalimited action understanding by generating synthetic samples. Given a data-limited dataset D = {v1, v2, . . . , vn-1, vn} with a class name list C = {c1, c2, . . . , ck-1, ck}, we use GPT-4o to generate text descriptions of humans performing specific actions, then generate the synthetic video dataset D′ based on them:

where DiT represents the text2video diffusion transformer.

Although this paradigm is straightforward and simple, it has the problem of insufficient information in the generated samples, which leads to suboptimal training efficiency. When action descriptions are generated solely on the basis of action names, text-to-video models tend to produce videos with simplistic and repetitive content with these descriptions, leading to lower information richness in the generated videos compared to real videos.

3.3 Information Enhancement Strategy

To enrich the action-related information that serves as a crucial factor in action understanding datasets to train action understanding models effectively, we propose the information Enhancement Strategy towards generated samples. Actionrelevant information, such as the environment in which the action frequently occurs, the objects that often appear when the action is performed, the characteristics of the people who always perform the action, benefits action understanding model training, as it encompasses rich semantics and contextual details that are relevant to human actions. A wellconstructed generated dataset should encompass sufficient action-related information, as this determines the upper limit of knowledge that the model can acquire from it.

Given an action category name c, we construct the contextual information most relevant to this action. We mainly focus on information from two perspectives: the environment where the action occurs and the appearance characteristics of the person performing the action. We generate contextual information through GPT-4o, where we generate 4 different environment descriptions Env = gpt(promptenv, c) and 16 different character descriptions Char = gpt(promptchar, c) for each action. Finally, we use this contextual information to enhance the content of the generated video. The process of generating action description information is shown in Figure 2 (b).

3.4 Uncertainty-Based Label Smoothing

Due to the limited capabilities of the text2video model, generated videos sometimes have suboptimal generation effects and contain less effective semantic information for training. Such samples may be detrimental to the training of the model, as shown in Figure 1.

To address this problem, we propose uncertainty-based label smoothing. Given a synthetic video sample v′ , we first use CLIP [Radford et al., 2021] to calculate its similarity S with all category names C, and then use the entropy of the similarity to measure its uncertainty.

After that, we dynamically adjust the smoothing of label smoothing according to the uncertainty of the sample:

whle w represents the weighting of uncertainty-based label smoothing, y = [q1, q2, ..., qk-1, qk] represents the original one-hot label. As low-quality generated samples often exhibit greater uncertainty, leading to an increased degree of label smoothing in such cases, as shown in Figure 2 (c). In this way, we can prevent the model from overfitting to low-quality samples and mitigate their impact on model training.

4 Experiment

4.1 Implementation Details

The text2video model we use is CogVideoX-2B [Yang et al., 2024]. For each dataset, we generate 128 videos for each category with 50 inference steps. We need to emphasize that the training data of CogVideoX-2B does not contain the annotated UCF-101, HMDB-51 or Kinetics-600 datasets we used in our experiments, so there is no data leakage. For information enhancement strategy, we use GPT-4o to generate multiple different environment descriptions at once, with the following prompt like "Given a phrase describing an action, generate four different scene appearance descriptions that are most suitable for performing the action.". Similarly, the prompt for generating character descriptions is as follows: "Given a phrase describing an action, generate 16 different descriptions of the physical features of a person who is suitable for that action.". In subsequent training, we adopt TC-CLIP [Kim et al., 2025] in most experiments except for abnormal action detection. For abnormal action detection, we use the X-CLIP-B/32 [Ni et al., 2022] following [Lv et al., 2023]. For tasks where real samples are not available such as zero-shot, we train the model with synthetic samples only. For tasks where real samples are available such as few-shot, long-tail, etc., we pre-train the model with synthetic samples and then fine-tune with the real samples. We set the w to 0.3 in uncertainty-based label smoothing. After calculating the uncertainty of all samples in a dataset, normalization is performed on them to constrain the scale.

4.2 Main Results

Zero-shot Action Recognition

In zero-shot action recognition, only the category names of the target dataset are available. The model needs to perform action recognition without training on target dataset. We conduct zero-shot action recognition experiments on the UCF101, HMDB-51, and Kinetics-600 datasets. We report the Top-1 accuracy for UCF-101 and HMDB-51, Top-1 and Top5 accuracy for Kinetics-600.

Table 1 shows the zero-shot action recognition results of the proposed method. The proposed method has significant improvements across all three datasets, especially the HMDB-51, where the Top-1 accuracy of zero-shot action recognition is improved by 5% after combining the generated sample training. It should be noted that when our method is based solely on the VIT-B backbone, our performance on HMDB-51 even surpasses those of methods based on the VIT-L backbone ( [Zhu et al., 2023; Akbari et al., 2023]). The rich contextual information related to actions is introduced by the proposed method, leading to the improvement of zero-shot action recognition.

The proposed method is generic and can be adapted to various models. We train using the generated samples on three zero-shot action recognition models, including ViFi-CLIP, BIKE, and TC-CLIP. The results are shown in Table 1 bottom. The proposed method improves all three models, especially the Top-1 accuracy of BIKE on Kinetics-600 is improved by almost 10%.

Few-shot Action Recognition

The purpose of the few-shot action recognition task is to enable the model to accurately classify action categories using only a few labeled samples per category, thereby alleviating the reliance on data. We conducted the few-shot action recognition experiments with the UCF-101 and HMDB-51 datasets in Table 2. We first pre-train the model with generated samples and then fine-tune it on each dataset with only K samples per category, where K is in 2, 4, 8 and 16.

The results demonstrate that the proposed method achieves improvements in all K-Shot settings, which can be attributed to the effective initialization of the model with the proposed method. Table 2 shows that the improvement gradually decreases as the K increases from 2 to 16. This phenomenon could be attributed to the fact that the information provided by the generated samples partially overlaps with that provided by the real samples. The proposed information enhancement strategy enhances the action-related information in the generated samples, allowing the proposed method to remain effective even at high values of K=16 in HMDB-51 and K=8 in UCF-101.

Base-to-Novel Generalization

The task of base-to-novel generalization is proposed to evaluate the generalization of a model to unseen classes when only samples from half of the classes are available for training. In each dataset, 16 samples per category are selected from half of the categories to construct the base split for training, while the remaining half of the categories serve as the novel split for evaluation.

Table 3 shows the Top-1 accuracy of the base classes, the novel classes, and their harmonic mean (HM) in UCF-101 and HMDB-51. The proposed method shows noticeable gains for base and novel categories.

Long-Tail Action Recognition

The concept of "long-tail problem" refers to the phenomenon where the imbalance in the distribution of samples across different classes leads to a significantly larger number of samples in the minority head classes compared to the tail classes, as models tend to perform well on the head classes but exhibit poorer performance on the tail classes. Following [Perrett et al., 2023], we construct long-tail action recognition datasets based on UCF-101 and HMDB-51.

The results of long-tail action recognition are presented in Table 4. The proposed method can improve the performance of the tail and the few categories while maintaining the performance of the head categories.

Abnormal Action Detection

Video anomaly action detection aims to identify abnormal events or actions within videos. The primary objective is to pinpoint the specific time window when an anomalous activity occurs, such as crimes, traffic accidents, or other illegal activities, etc. This task is challenging because weak labels are provided only at the video level, whereas the model needs to make frame-level predictions for abnormal actions. We employ the MIL (Multiple Instance Learning) baseline method following [Lv et al., 2023] for abnormal action detection. Initially, we first pre-train the model on generated samples and subsequently fine-tune it on real samples. We report the AUC and AUCA [Lv et al., 2021] of the frame-level ROC results on the UCF-Crime dataset.

The results for abnormal action detection are presented in Table 5. The proposed method is effective for abnormal action detection, leading to a 2.23% improvement in AUC and a 2.54% improvement in AUCA compared to MIL baseline. The collection of real-world abnormal action videos is often constrained by real conditions, such as the rarity of the occurrence of anomalous actions. The proposed methods enable the low-cost construction of anomalous action datasets, which effectively enhances the model's performance.

4.3 Ablation Studies

We discuss the effectiveness of the proposed strategies in Table 6 and Table 7. We report the Top-1 accuracy under zeroshot setting for for the ablation study.

The proposed information enhancement strategy ("IE") consists of two components: environmental information enhancement and character information enhancement. We discuss them in Table 6. The "Basic" strategy refers to generating action descriptions solely based on category names and then using them to create videos; The "Env" strategy incorporates environmental information where the action occurs; the "Cha" strategy incorporates character appearance information about the person performing the action.

The results show that incorporating either environmental information or character information in the synthetic samples improves zero-shot accuracy. The information enhancement strategy, which combines environmental information and character information, can lead to a 3.4% improvement compared to the basic strategy. This indicates that enhancing the action-related information of the generated samples is beneficial for training action understanding models. Figure 3 illustrates the visualization of the "brush hair" action samples generated using the four aforementioned strategies. When employing the basic strategy, the video generation model tends to generate similar videos in which the characters consistently appear as Caucasian women with long brown hair. When the "Env" strategy is adopted, various contextual details that assist in understanding the "brush hair" action are included in generated videos, such as full-length mirrors and bathrooms with tiled backgrounds. This information enhances the knowledge encompassed by the generated dataset. We set the information enhancement strategy as the default setting for the remaining experiments.

In Table 7, we discuss three strategies to mitigate the impact of low-quality generated samples on training. "LS" refers to the label smoothing without additional design toward low-quality generated samples. "UW" denotes the uncertainty-based weighting strategy, where we weight the loss based on the uncertainty of each sample. "UF" denotes the uncertainty-based filtering strategy that removes highuncertainty samples from the synthetic dataset. "UL" denotes the uncertainty-based label smoothing strategy, where we adjust the smoothness of the label smoothing based on the uncertainty of each sample.

Based on the experiments, uncertainty-based label smoothing shows the best performance among three strategies, leading to a 1.1% improvement on the UCF-101 dataset. This validates the effectiveness of the proposed uncertainty-based label smoothing strategy, which can prevent the model from overfitting low-quality samples and alleviating its impact.

5 Analysis

We conducted qualitative analysis on synthetic datasets and real datasets, as shown in Figure 1. We extract CLIP features from the HMDB-51 dataset, the synthetic HMDB-51 dataset with the basic strategy, and the synthetic HMDB-51 dataset with the information enhancement strategy. Subsequently, we make the t-SNE visualization of these datasets based on the extracted features.

In Figure 1, it is evident that the boundaries between different categories of synthetic samples are quite distinct. In contrast, the boundaries between samples of different categories in the real dataset are more blurred and lack clarity.

This could be attributed to the higher complexity of real samples compared to synthetic samples. This complexity stems from various aspects such as changes in lighting, camera angles, and background details, among others, leading real samples to contain more information. When generating samples based on the basic strategy, the text2video model tends to produce videos with similar content, resulting in low levels of effective information contained in the generated datasets. This limits the upper bound of knowledge that the synthetic data can contain.

The proposed information enhancement strategy effectively increases the information content in the generated samples. As illustrated in Figure 1, the proposed method improves the information contained in the generated samples, bringing their distribution patterns closer to real samples.

We perform quantitative analyzes in Table 8. DaviesBouldin (DB) is a metric used to assess the effectiveness of clustering algorithms, where a smaller value indicates clearer separation between clusters and tighter intra-cluster cohesion. We employ the Davies-Bouldin score to measure the complexity of the datasets. The first column of Table 8 presents the Davies-Bouldin scores for the generated datasets and real datasets. Real datasets (UCF-101,HMDB-51) exhibit higher DB scores compared to generated samples due to their complexity. The Maximum Mean Discrepancy(MMD) [Gretton et al., 2006] is a commonly used metric for measuring the domain gap between different datasets, where a higher MMD indicates a larger domain gap. In columns 2, 3, and 4 of Table 8, we present the MMD between various datasets and the HMDB-51 dataset. MMD between real datasets, such as UCF-101 and HMDB-51, exhibit lower values, while the MMD between generated datasets and real datasets are higher, indicating a larger domain gap. The proposed strategy reduces the domain gap between generated samples and real samples, which is manifested as lower MMD values.

6 Conclusion

In this paper, we tackle the challenge of data-limited action understanding by introducing a generic method that leverages synthetic video data generated through the text2video diffusion transformer. This approach enables the cost-effective creation of large-scale annotated video datasets, significantly mitigating the data scarcity in action understanding tasks. To enhance the utility of generated samples, we propose the information enhancement strategy and uncertainty-based label smoothing. We validate the proposed method on four datasets across five tasks and achieve SOTA performance for zero-shot action recognition.

Acknowledgements

Supported by the National Natural Science Foundation of China (Grant NO 62376266 and 62406318), Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China.