ExVideo: Extending Video Diffusion Models via Parameter-Efficient Post-Tuning

1 Introduction

In recent years, diffusion models [Sohl-Dickstein et al., 2015; Ho et al., 2020] have achieved outstanding results in image synthesis, significantly surpassing previous GANs [Dhariwal and Nichol, 2021]. These achievements have subsequently fostered a burgeoning interest in the adaptation of diffusion models for video synthesis. The diffusion models such as Stable Video Diffusion [Blattmann et al., 2023], AnimateDiff [Guo et al., 2023], and VideoCrafter2 [Chen et al., 2024a]

1 https://ecnu-cilab.github.io/ExVideoProjectPage/.

epitomize this research trajectory, showcasing the ability to produce frames that are not only coherent but also of high visual quality. These achievements underscore the practicality and potential of employing diffusion models in the field of video synthesis. With the groundbreaking results of SORA [Liu et al., 2024] reported at the beginning of 2024, the research direction of video synthesis has once again attracted widespread attention.

Although current video synthesis models are capable of producing video clips of satisfactory quality, the generated videos are generally short, and extending their duration remains a challenge. Current methodologies can be categorized into three types to generate longer videos. 1) Pre-training using long video datasets [Chen et al., 2024b; Wang et al., 2023b]. Through extensive training with long video samples, it is foreseeable that models can improve their ability to generate longer videos. However, training with such datasets would result in prohibitively escalated costs. Consequently, given the computational constraints, current video generation models are primarily trained on short video clips. 2) Generating videos in a streaming [Kodaira et al., 2023] or sliding window [Duan et al., 2024] manner. Without further training, longer videos can be generated by stitching together several short video segments. However, this approach leads to lower video coherence. In addition, existing video generation models lack the capability for long-term video understanding, making the accumulation of errors inevitable. As a result, during the generation of long videos, the visual quality is prone to deterioration, manifesting as a breakdown in the imagery. 3) Frame interpolation [Huang et al., 2022; Wu et al., 2024]. Video frame interpolation models offer a method to augment the frame count of generated videos. However, this approach is inadequate for extending the narrative timeframe of the video. While it increases the number of frames, maintaining the original frame rate would result in an unnatural slow-motion effect, thereby failing to extend the narrative span of the video content. These challenges underscore the necessity for innovative solutions capable of overcoming the existing hurdles associated with video duration extension, without compromising video quality or coherence.

Recent breakthroughs in the development of LLMs (large language models) [Xiong et al., 2023; Xiao et al., 2023; Chen et al., 2023b] have inspired us. Notably, LLMs, despite being trained on fixed-length data, exhibit remarkable proficiency in understanding contexts of variable lengths. This flexibility is further enhanced through the integration of supplementary components and the application of lightweight training procedures, enabling the processing of exceptionally lengthy texts. Such innovations have motivated us to explore analogous methodologies within video synthesis models. In this paper, we introduce a novel post-tuning strategy, called ExVideo, specifically designed to empower existing video synthesis models to produce extended-duration videos within the constraints of limited computational resources. We have designed an extension structure for mainstream video synthesis model architectures. This framework incorporates adapter components, meticulously engineered to preserve the intrinsic generalization capabilities of the base model. Through post-tuning, we enhance the temporal modules of the model, thereby facilitating the processing of content across longer temporal spans.

In theory, ExVideo is designed to be compatible with the majority of existing video synthesis models. To empirically validate the efficacy of our post-tuning methodology, we applied it to the Stable Video Diffusion model [Blattmann et al., 2023], a popular open-source image-to-video model. Through ExVideo, we can extend the original frame synthesis capacity from a limit of 25 frames to 128 frames. Importantly, this expansion was achieved without compromising the model's distinguished generative capabilities. Additionally, the enhanced model exhibits the versatility to be seamlessly integrated with text-to-image models [Rombach et al., 2022; Li et al., 2024b; Chen et al., 2023a]. This synergistic amalgamation establishes robust and versatile text-tovideo pipelines. This adaptability underscores the potential of our post-training technique, the source code and the extended model will be released publicly. In summary, the contributions of this paper include:

• We present ExVideo, a post-tuning technique for video synthesis models that can extend the temporal scale of existing models to facilitate the generation of long videos.
• Based on Stable Video Diffusion (SVD), we have trained an extended video synthesis model named ExSVD. This model is capable of generating coherent videos of up to 128 frames while preserving the generative capabilities of the original model.
• Through comprehensive empirical experiments, we demonstrate the feasibility of enhancing video synthesis models via post-tuning, thereby presenting an innovative approach to the training of large-scale models for extended video synthesis.

2 RelatedWork

2.1 DiffusionModels

Diffusion models [Sohl-Dickstein et al., 2015; Ho et al., 2020] are a category of generative models that characterize the content generation as a Markov random process. Unlike GANs [Goodfellow et al., 2014], diffusion models do not require adversarial training, thus making their training process more stable. Moreover, through an iterative generation process, diffusion models are capable of producing images with exceptionally high quality. In recent years, image synthesis models based on diffusion, including Pixart [Chen et al., 2023a], Imagen [Saharia et al., 2022], Hunyuan-DiT [Li et al., 2024b], and the Stable Diffusion series [Rombach et al., 2022], have achieved impressive success. Diffusion models have given rise to a vast open-source technology ecosystem. Technologies such as LoRA [Hu et al., 2021], ControlNet [Zhang et al., 2023] and IP-Adapter [Ye et al., 2023] have endowed the generation process of diffusion models with a high degree of controllability, thereby meeting the needs of various application scenarios.

2.2 Video Synthesis

Given the remarkable success of diffusion models in image synthesis, video synthesis approaches based on diffusion have also been proposed in recent years. For example, by adding additional motion modules to the UNet model [Ronneberger et al., 2015] in Stable Diffusion [Rombach et al., 2022], AnimateDiff [Guo et al., 2023] transfers the capabilities of image synthesis to video synthesis. Stable Video Diffusion [Blattmann et al., 2023] is an image-to-video model architecture and can synthesize video clips after end-to-end video synthesis training. Unlike image synthesis models, video synthesis models require substantial computational resources since the model needs to process multiple frames simultaneously. As a result, most existing video generation models [Guo et al., 2023; Wang et al., 2023a] can only produce very short video clips. For instance, AnimateDiff can generate up to 32 frames, while Stable Video Diffusion can generate a maximum of 25 frames. This limitation prompts us to explore the methodology to extend video synthesis models over longer temporal scales.

2.3 Extending Generative Models

Although the existing diffusion models are trained with a fixed scale, such as Stable Diffusion being trained at a fixed resolution of 512×512, some approaches can extend them to larger scales. For instance, in image synthesis, approaches like Mixture of Diffusers [Jim´enez, 2023], MultiDiffusion [Bar-Tal et al., 2023], and ScaleCrafter [He et al., 2023] can increase the resolution of generated images by altering the inference process of the UNet model in Stable Diffusion. Similar techniques have also emerged in the field of large language models. With the help of positional encoding technologies such as RoPE [Su et al., 2024] and ALiBi [Press et al., 2021], large language models can extrapolate to longer text processing tasks under the premise of training with limited-length texts. Post-tuning can further help language models achieve super-long text comprehension and generation [Xiong et al., 2023; Chen et al., 2023b]. These research findings have inspired and motivate us to explore the extension of video synthesis models. We aim to endow existing video synthesis models with the capability to generate longer videos.

3 Methodology

In this section, we first review the architectures of mainstream video diffusion models, then discuss the methodologies we have adopted to extend the temporal modules for long video synthesis, and finally introduce the post-tuning strategy.

3.1 Preliminaries

The huge demands of computational resources for training video synthesis models lead to a prevalent practice of adapting existing image synthesis models for video generation. This adaptation is typically achieved by incorporating temporal modules into the model for generating dynamic content. We provide a comprehensive overview of temporal module architectures as follows:

• 3D convolution [Li et al., 2021]: Convolution layers form the foundational blocks in computer vision. 2D convolution layers have been employed in the UNet [Ronneberger et al., 2015] architecture, which is widely used in diffusion models. By extending 2D convolutions into the third dimension, these layers are seamlessly adapted in video synthesis models. Research indicates that convolution layers in diffusion models exhibit a high degree of adaptability across various resolutions [Bar-Tal et al., 2023], which is a testament to their capacity for generalization.
• Temporal attention [Vaswani et al., 2017]: In image synthesis, the importance of attention mechanisms is underscored by their contribution to the generation of images with remarkable fidelity, as evidenced by the ablation studies in latent diffusion [Rombach et al., 2022]. Transferring spatial attention mechanisms to the video domain raises concerns regarding computational efficiency due to the quadratic time complexity of the attention operators. To circumvent this computational bottleneck, advanced video synthesis models typically adopt temporal attention layers [Guo et al., 2023; Blattmann et al., 2023] that optimize efficiency by curtailing the volume of embeddings processed by each attention operator.
• Positional embedding [Su et al., 2024]: The native attention layers cannot model the positional information in videos. Therefore, video synthesis models typically incorporate positional embeddings to enrich the embedding space with positional information. Positional embeddings can be instantiated through diverse methodologies. For example, AnimateDiff [Guo et al., 2023] opts for learnable parameters to establish positional embeddings, whereas Stable Video Diffusion [Blattmann et al., 2023] utilizes trigonometric functions to generate static positional embeddings.

3.2 Extending Temporal Modules

Most video synthesis models are pre-trained on videos comprising only a constrained number of frames due to limited computational resources. For instance, Stable Video Diffusion [Blattmann et al., 2023] is capable of generating a maximum of 25 frames, while AnimateDiff [Guo et al., 2023] is limited to synthesizing videos of up to 32 frames. To augment these models to produce extended videos, we propose enhancements to the temporal modules within these models.

…

…

Firstly, the inherent functionality of 3D convolution layers to adaptively accommodate various scales has been previously validated through empirical studies [Jim´enez, 2023; Bar-Tal et al., 2023; He et al., 2023], even without necessitating fine-tuning. Consequently, we opt to retain the 3D convolution layers in their original form to preserve these capabilities. Secondly, regarding the temporal attention modules, research on large language models has demonstrated the potential for scaling existing models to accommodate longer contextual sequences [Xiong et al., 2023; Chen et al., 2023b]. Inspired by these findings, we fine-tune the parameters within the temporal attention layers during the training process to enhance their efficacy over extended frame sequences. Thirdly, for the positional embedding layers, either static or trainable embeddings cannot be directly applied to longer videos. To circumvent this pitfall while ensuring compatibility with a wide array of existing video models, we use extended trainable parameters to replace the original positional embeddings. These extended trainable positional embeddings are initialized in a cyclic pattern, drawing upon the configurations of the pre-existing embeddings. Further, drawing inspiration from various adapter models [Hu et al., 2021; Zhang et al., 2023], we incorporate an additional identity 3D convolution layer after the positional embedding layer, aimed at learning long-term information. The central unit of this 3D convolution kernel is initialized as an identity matrix, and the remaining parameters are initialized to zero. The identity 3D convolution layer ensures that, before training, there is no alteration to the video representation, thereby maintaining consistency with the original computational process.

We apply our devised extending approach to Stable Video Diffusion [Blattmann et al., 2023], which is a popular model within open-source communities for video synthesis. The comparative architectures, both pre and post-extension, are illustrated in Figure 1. Because of the fundamental similarities that underpin the construction of temporal blocks within video synthesis models, our extending approach can also be applied to various video synthesis models.

3.3 Post-Tuning

After extending the temporal blocks in the video synthesis models, we enhance the model's abilities to generate extended videos via post-tuning. To circumvent potential copyright concerns with video content, we employed a publicly available dataset OpenSoraPlan 2 , which comprises 40,258 videos. These videos were sourced from copyright-free platforms. The videos in this dataset maintain a resolution of 512×512. ExVideo expands its capacity to 128 frames. Over such extended sequences, full training is deemed impractical because of the substantial computational requirements. Instead, we employed several engineering optimizations aimed at optimizing GPU memory usage. These optimizations are crucial for managing the increased computational load and facilitating efficient training within limited hardware resources:

• Parameter freezing: All parameters except the temporal blocks are frozen.
• Mixed precision training: We deploy a mixed precision training program by converting some parameters to 16bit floating-point format.
• Gradient checkpointing [Feng and Huang, 2021]: Gradient checkpointing is enabled in the model. By storing intermediate states during forward passes and recomputing gradients on-demand during the backward pass, this technique effectively decreases memory usage.
• Flash Attention [Dao, 2023]: We integrate Flash Attention to enhance the computational efficiency of attention mechanisms.
• Shard optimizer states and gradients: We leverage DeepSpeed [Rasley et al., 2020], a library optimized for distributed training, to enable shard optimizer states and gradients across multiple GPUs.

The loss function and the noise scheduler are consistent with the original model. The learning rate is 1×10-5 and the batch size on each GPU is 1. The training was conducted using only 8 NVIDIA A100 GPUs over one week. In order to ensure the stability of the training process, exponential moving averages were employed for the update of weights.

4 Experiments

By integrating extended temporal modules into the original Stable Video Diffusion (SVD) model and performing posttuning, we have developed the Extended Stable Video Diffusion (ExSVD) model. This enhanced model is capable of generating coherent videos with lengths of up to 128 frames. To validate its capabilities, we have conducted comprehensive experiments comprising three components. First, we perform a comparative analysis between our ExSVD and the original SVD model to elucidate the enhancements achieved through post-tuning. Second, we evaluate the performance of ExSVD in comparison to other publicly accessible models. Finally, we present illustrative examples to provide a tangible understanding of the model's performance.

2 https://hf.co/datasets/LanguageBind/Open-Sora-Plan-v1.0.0

4.1 Evaluation on ExtendedModel Performance

To assess the performance of ExVideo, we present the results of our ExSVD model and compare them with those of the original SVD model across two primary dimensions: automatic metrics and human evaluation.

Parameter Settings: The comparative experiments are conducted based on VBench [Huang et al., 2024], which is a comprehensive suite of tools designed to automatically assess the quality of generated videos. Following VBench's prompt sampling methodology, we assigned 5 random seeds to each prompt for the text-to-video generation process. Initially, we employed the Hunyuan-DiT [Li et al., 2024b] text-to-image model to generate 5 images for each prompt. Subsequently, for each image, both our ExSVD and the SVD models were used to generate corresponding videos, resulting in a total of 4720 (944 × 5) videos for each model. We used the DDIM [Song et al., 2020] sampler with 50 steps for sampling.

Evaluation Metrics: VBench assesses video performance from two broad perspectives: video quality and videocondition consistency. Video quality focuses on the perceptual quality of the synthesized video, including temporal quality and frame-wise quality. Video-condition consistency focuses on whether the synthesized video aligns with the userprovided guiding condition (text prompt) and includes metrics from semantic and style dimensions. It is important to note that the dynamic degrees in the original implementation of VBench are sensitive to frames per second (FPS) and the total number of frames. Our findings indicate that this metric can significantly impact the overall score, potentially resulting in biased comparisons between different models. We will address this issue in future research, as there currently are no valid metrics available for evaluation.

Quantitive Results: Based on the generated videos from our ExSVD and the original SVD model, we have calculated all the automatic metrics in VBench. The visualized results of video quality are shown in Figure 2, and the detailed numerical results are presented in Table 1. From the perspective of video quality, most metrics of the ExSVD model are on par

(i) ExSVD

Figure 3: Visual comparisons of text-to-video results from several existing video synthesis models and our Extended model. The prompts are "a boat sailing smoothly on a calm lake" and "an astronaut flying in space, Van Gogh style". In our pipeline, the first frame is generated by Hunyuan-DiT, and ExSVD generates the video according to the first frame. We highly recommend readers watch the videos on our project page: https://ecnu-cilab.github.io/ExVideoProjectPage/.

with those of the SVD model, indicating that ExSVD does not degrade the video quality of the original SVD model. In the temporal flickering dimension, the ExSVD model demonstrates superior performance. This enhancement is primarily attributed to the extended temporal block in the ExSVD model, which bolsters the model's motion prediction capabilities. After post-tuning, the ExSVDmodel exhibits enhanced temporal consistency, resulting in fewer flickering phenomena and thus a higher temporal flickering score. From the perspective of video-condition consistency, the performances of the ExSVD and SVD models are generally consistent. In the spatial relationship and multiple objects metrics, ExSVD achieves higher scores. This indicates that ExSVD is capable of fully leveraging the text-to-image model to synthesize realistic videos based on the generated images. Overall, ExVideo enhances the total number of frames and duration of videos without compromising video quality.

Human Evaluation: In addition to the automatic metrics evaluation, we conducted a human preference experiment involving 30 participants to facilitate a comparative analysis between the SVD and our ExSVD model. In each evaluation session, we randomly selected two videos that both corresponded to the same prompt and presented them to the participants. Participants were instructed to choose from one of three options: "Left is better", "Tie", or "Right is better", without disclosing the names of the models. Each participant evaluated up to 30 randomly selected video pairs. The results indicate that our ExSVD model outperformed the SVD model in terms of human preference, achieving a win rate of 48.35% compared to SVD's win rate of 34.77%.

4.2 Comparison with Publicly Accessible Models

We further evaluated the performance of ExSVD in comparison to other publicly available models. To facilitate a comprehensive analysis that incorporates both text-to-video and image-to-video models, we designed a text-to-video pipeline that integrates ExSVD with Hunyuan-DiT. This allows for a uniform assessment of the models across the text-to-video task. The parameter settings of this evaluation were consistent with those outlined in the previous subsection. For a thorough evaluation of our models, we selected top-performing video synthesis models for comparison. The results are summarized in Table 1, where the metrics of the baseline models are collected from the original VBench leaderboard. Due to the variability of the dynamic degree metric with respect to frame and FPS, we have opted not to include it in the table. In comparison to other models, our ExSVD outperforms the competition, achieving the highest overall score and excelling in the aesthetic quality, temporal style, and overall consistency metrics. Additionally, ExSVD exhibits competitive performance in the dimensions of temporal flickering, motion smoothness, and multiple objects.

4.3 Case Study

Visual Comparison: We present a series of video examples generated by ExSVD and other video synthesis models to facilitate an intuitive comparison of their performance. The illustrative results from these models are displayed in Figure 3. We highly recommend readers watch the videos on our project page. A noteworthy observation from this comparison is that the majority of existing video synthesis models tend to produce videos characterized by limited motion dynamics. In contrast, our extended model, which benefits from posttuning applied over extended temporal durations, exhibits a markedly improved ability to generate videos with significant movement. In the second example, Kling, a competitive close-source model, demonstrates the capability to generate realistic videos, but it is unable to generate the astronaut in the style of Van Gogh. This disparity in performance highlights the advanced generative capabilities of our model.

Generalization Abilities: Although the model is trained at a fixed resolution and exclusively utilizes realistic video

(b) 1024×576

(c) 1024×1024

(b) Pixel art style

Figure 5: Examples in various styles generated by ExSVD, where the first frame is generated by Stale Diffusion 3. The prompt is "A beautiful coastal beach in spring, waves lapping on sand", followed by the description of style.

datasets, the extended variant demonstrates remarkable capabilities, allowing for the generation of videos across a spectrum of resolutions and styles. To rigorously assess the performance of ExSVD, we conduct additional evaluations across various resolutions. Figure 4 illustrates several generated video examples that underscore the model's capacity to generate videos in various resolutions and aspect ratios. This adaptability ensures that the generated videos maintain visual integrity regardless of the resolution parameters. Furthermore, Figure 5 presents an array of stylistic variations, further emphasizing the model's versatility in accommodating diverse artistic expressions. These examples underscore the robustness and generalizability of ExSVD, offering flexibility in video generation across varying contexts.

Visualization of Training Process: We investigated the evolution of the model's capabilities during the training process. Figure 6 presents the generated videos that exemplify the model's performance at three distinct phases of training. It is difficult to present the dynamics using still images, thus we present the optical flow, computed by RAFT [Teed and Deng, 2020], to the right of each example for a clearer demon-

(c) After 64000 steps, complex motion emerges.

Figure 6: Examples in different phases. The prompt is "sunset, mountains, clouds". We show the optical flow to visualize motion, with similar colors indicating similar movement directions.

stration of motion. Initially, before training, the extended model architecture was solely capable of guaranteeing the structural integrity of the video frames, which suffered from pronounced jittering artifacts. Progressing through the training, after 32,000 steps, the model began to produce videos displaying smooth camera movements. With continued training up to 64,000 steps, the model further advanced to create complex motions, such as clouds and mountains moving with nuanced, layered speed. The model effectively understands the depth and spatial relationships within the scene. This example intuitively illustrates the process of the model learning long-term information.

5 Conclusions and FutureWork

In this paper, we delve into the enhancement of video diffusion models through post-tuning. Specifically, we propose a post-tuning approach called ExVideo, which can extend the duration of generated videos and release the potential of video synthesis models. Based on Stable Video Diffusion, our approach achieves a quintupling in the number of frames, while preserving the original generalization abilities. However, the enhanced model remains inherently constrained by the limitations of the base model. Looking ahead, we are committed to furthering our exploration of video synthesis models through post-tuning methodologies.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under grant Number 62202170 and Alibaba Group through Alibaba Innovation Research Program.