FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance

1 Introduction

With the advancement of the diffusion model, the text-toimage (T2I) generative models [Blattmann et al., 2023b; Ho et al., 2022; Luo et al., 2023; Ma et al., 2024; Liu et al., 2025] can produce high-resolution and photo-realistic images by complex text prompts, resulting in various applications. Currently, many studies [Wang et al., 2024; Guo et al., 2023a] explore the text-to-video (T2V) generative model due to the great success of T2I models. However, building a powerful T2V model remains challenging as it requires maintaining temporal consistency while generating coherent motions simultaneously. Moreover, due to limited memory, most diffusion-based T2V models [Wang et al., 2024; Guo et al., 2023a; Zhang et al., 2024a; Guo et al., 2023b; Chen et al., 2023; Menapace et al., 2024] can only produce fewer than 16 frames of video per sampling without extra assistance (i.e., super-resolution).

The existing T2V models [Zhang et al., 2024a; Guo et al., 2023b; Chen et al., 2023; Menapace et al., 2024] typically employ spatial cross-attention between text conditions and latent features for achieving text control generation. However, as shown in Fig. 2(I), this manner shares the same text condition across different frames, thus lacking the specific textual guidance tailored to each frame. Consequently, these T2V models struggle to comprehend the temporal logic of text prompts and produce videos with coherent motion. Taking AnimateDiff [Guo et al., 2023b] as an example, in Fig. 1, we exhibit its generated video and visualize the [verb]- focused region (which is closely associated with the video motion) based on the attention map from the cross-attention module. Ideally, these regions should transition smoothly over time and align with the semantics of motion instructions. However, as observed in the upper right of the figure, the [verb]-focused region remains nearly identical across different frames due to the consistent textual guidance between frames. Meanwhile, the video exhibits poor motion in the upper left of the figure.

Furthermore, we perform a similar visual analysis for the longer video (e.g., 64 frames) generation and find that this problem is more prominent, as illustrated in the lower part of Fig. 1. Therefore, we believe this approach hampers the advancement of video dynamics and consistency and is suboptimal for video generation tasks based on text prompts.

To this end, we present a novel T2V model named FancyVideo, capable of comprehending complex spatialtemporal relationships within text prompts. By employing a cross-frame textual guidance strategy, FancyVideo can generate more dynamic and plausible videos in a sampling process. Specifically, to boost the model's capacity for understanding spatial-temporal information in text prompts, we optimize the spatial cross-attention through the proposed Cross-frame Textual Guidance Module (CTGM), comprising a Temporal Information Injector (TII) and Temporal Affinity Refiner (TAR). As illustrated in Fig. 2(II), TII injects temporal information from latent features into text conditions, building cross-frame textual conditions. Then, TAR refines the affinity between frame-specific text embedding and video along time dimension, adjusting the temporal logic of textual guidance. Through the cooperative interaction between TII and TAR, FancyVideo fully captures the motion logic embedded within images and text. Consequently, its motion token-focused area shifts logically with frames, as illustrated in the lower right part of Fig. 1. This characteristic enables FancyVideo to produce dynamic videos, as displayed in the lower left part of the figure. Experiments demonstrate that FancyVideo successfully generates dynamic and consistent videos, achieving the SOTA results on the EvalCrafter [Liu et al., 2023] benchmark and the competitive performance on UCF-101 [Soomro et al., 2012] and MSR-VTT [Xu et al., 2016]. Additionally, FancyVideo supports generating videos from user-input images, i.e., the image-to-video task. We have also conducted extensive experiments to demonstrate the superiority of our method.

Contributions. 1) We introduce FancyVideo, the pioneering endeavor as far as our knowledge extends, delving into cross-frame textual guidance for the T2V task. This approach offers a fresh perspective to enhance current text-control methodologies. 2) We propose the Cross-frame Textual Guidance Module (CTGM), which constructs cross-frame textual conditions and subsequently guides the modeling of latent features with robust temporal plausibility. It can effectively enhance the motion and consistency of video. 3) We demonstrate that incorporating cross-frame textual guidance represents an effective approach for achieving high-quality video generation. Our experiments showcase that this approach attains state-of-the-art results on both quantitative and qualitative evaluations.

2 RelatedWork

Text to Video Generation. Generative models like GANs [Wang et al., 2020; Munoz et al., 2021; Gur et al., 2020], auto-regressive models [Wang et al., 2019; Yan et al., 2021], and implicit neural representations [De Luigi et al., 2023] have been explored for video generation. Recently, diffusion models [Rombach et al., 2022; Zhang et al., 2024b; Zhang et al., 2024c] have advanced text-to-image quality. Stable Diffusion [Rombach et al., 2022] uses a VAE [Kingma and Welling, 2013] latent space to reduce cost [Jiang et al., 2023]. T2V models [Wu et al., 2023a] add temporal layers to T2I models but often lack frame-to-frame consistency. We propose cross-frame textual guidance to improve temporal coherence.

Image-conditioned Video Generation. To bridge the gap between text and video, recent work leverages images for clearer video generation. SVD [Blattmann et al., 2023a] treats images as noisy latent inputs, while MoonShot [Zhang et al., 2024a] improves semantic consistency using a CLIP encoder. Though effective, these I2V methods rely on input images. Hierarchical approaches [Zeng et al., 2023; Chen et al., 2023] use images as keyframes to extend video length with fewer constraints. These methods, though I2Vcapable, are essentially T2V. FancyVideo adopts a hierarchical design with cross-frame textual guidance, enabling more frames per iteration and faster inference.

3 Method

3.1 Preliminaries

Latent Diffusion Models. LDMs [Sohl-Dickstein et al., 2015; Ho et al., 2020] enhance efficiency by running diffusion in the VAE-compressed latent space [Kingma and Welling, 2013] instead of pixel space. The forward process adds Gaussian noise (ϵ ∼ N(0, I)) to the latent code z, yielding:

where ¯ αt denotes a noise scheduler with timestep t. For the inverse process, it trains a denoising model (fθ) with the objective:

where c represents the condition and target y can be noise ϵ, denoising input z or v-prediction (v = √ ¯ αtϵ - √1- ¯ αtz) in [Salimans and Ho, 2022]. In this paper, we adopt the vprediction as the supervision.

Zero terminal-SNRNoise Schedule. Previous studies proposed zero terminal SNR [Lin et al., 2024] to handle the signal-to-noise ratio (SNR) difference between the testing and training phase, which hinders the generation quality. At training, due to the residual signal left by the noise scheduler, the SNR is still not zero at the terminal timestep T . However, the sampler lacks realistic data when sampling from random gaussian noise during the test, resulting in a zero SNR. This train-test discrepancy is unreasonable and an obstacle to generating high-quality videos. Therefore, following the [Lin et al., 2024; Girdhar et al., 2023], we scale up the noise schedule and set ¯ αT = 0 to fix this problem.

3.2 Model Architecture

Fig. 3 illustrates the overall architecture of FancyVideo. The model is structured as a pseudo-3D UNet, which integrates frozen spatial blocks, sourced from a text-to-image model, along with Cross-frame Textual Guidance Modules (CTGM) and temporal attention blocks. The model takes three features as input: noisy latent Zn ∈ Rf×h×w×c , where h and w indicate the height and width of the latent, f signifies the number of frames, and c denotes the channels of the latent; mask indicator M ∈ Rf×h×w×1 , with elements set to 1 for the first frame and 0 for all other frames; image indicator I ∈ Rf×h×w×c , with initial image as the first frame and 0 for all other frames. The denoising input Z is formed by concatenating Zn, M and I along the channel dimension, represented as Z = [Zn;M; I] ∈ Rf×h×w×(2c+1) . Within each spatial block, we first incorporate prior knowledge of the motion score as embeddings. In each subsequent cross-attention layer, CTGM is employed to capture the intricate dynamics described in the text prompts. Afterward, we apply temporal attention blocks to enhance the temporal relationships across various patches.

Motion Embedding

To achieve more controllable video generation in terms of motion amplitude, we introduce motion score information calculated by the RAFT [Teed and Deng, 2020] alongside the timestep information. Specifically, we calculate a motion score for the training samples in the dataset within a range of 0.1 to 10. The score are then encoded into motion features through a motion embedding layer. By controlling the motion score, we can generate videos with stronger motion. However, simply adjusting the score may lead to unrealistic motion. We use CTGM to prevent these issues.

Cross-frame Textual Guidance Module

CTGM advances the existing text control method through two sub-modules: Temporal Information Injector (TII) and Temporal Affinity Refiner (TAR) as depicted in Fig. 3(III). Before engaging in cross-attention, TII initially extracts temporal latent feature Zt and then incorporates temporal information into text embedding Trep based on Zt, obtaining cross-frame textual condition Tz. Subsequently, TAR refines the affinity between Zt and Tz along the time axis, enhancing the temporal coherence of textual guidance. The computation process of the CTGM can be formalized as:

where Wq, Wk, and Wv represent the linear layers for query, key, and value in original cross-attention, respectively. The hyper-parameter dk is acquired from the query dimensions. TII(·, ·) and TAR(·) denotes the functions of TII and TAR. In the end, we get refined noisy latent feature Zref . A detailed description of these three modules is provided as follows.

Temporal Information Injector. In previous work [Guo et al., 2023b; Girdhar et al., 2023], the text embedding Trep is repeated equally f times, resulting in Trep ∈ Rf×n×c , n denoting the length of the embedding vector. We inject temporal information into the embedding before performing spatial cross-attention, thereby enabling distinct focal points on the text within different frames. In Temporal Information Injector (TII), we initially reshape the noisy latent Z from Rf×h×w×c to R(hw)×f×c and apply temporal self-attention to acquire Zt. Then, we conduct spatial cross-attention, using the repeated text embedding Trep as queries and the noisy latent Zt ∈ Rf×(hw)×c as both keys and values, resulting in the text embedding Tz with frame-specific temporal information. The formalization of the TII module can be expressed

where SelfAttnt denotes temporal self-attention and CrossAttns denotes spatial cross-attention. Through TII, we obtain the noisy latent Zt with temporal information and the latent-aligned text embedding Tz.

Temporal Affinity Refiner. To dynamically allocate attention to text embedding across different frames, we design the Temporal Affinity Refiner (TAR) to refine the attention map of spatial cross-attention. In spatial cross-attention, the noisy latent serves as the query, while the text embedding serves as both the key and value. The attention map A ∈ Rf×(hw)×n , compute as A = (WqZt)(WkTz)T /√dk, reflects the affinity between the text and patches. Then, TAR applies temporal self-attention to the attention map A ∈ R(hw)×f×n , obtaining the refined attention map Aref , which can be represented as:

With the TAR, Aref establishes a more logical temporal connection in the affinity matrix. It can perform more dynamic action while ensuring no additional video distortion occurs. Finally, the cross-attention process is completed with the refined attention map as Zref = Softmax(Aref )(WvTz).

4 Experiments

In the quantitative experiments, FancyVideo utilizes the T2I base model to generate images as the first frame. In the qualitative experiments, for aesthetic purposes and to remove watermarks, an external model is used to generate a beautiful first frame.

4.1 Qualitative Evaluation

We choose AnimateDiff [Guo et al., 2023b], DynamiCrafter [Xing et al., 2023], and two commercialized products, Pika [PikaLabs, 2024] and Gen2 [Runway, 2024], for a composite qualitative analysis. It is worth noting that in the quantitative experiments, the first frame of FancyVideo is generated by SDXL to achieve a more aesthetically pleasing result and to minimize the appearance of watermark (although subsequent frames may still exhibit it).

As shown in Fig. 4, our approach exhibits superior performance, outperforming previous methods regarding temporal consistency and motion richness. In contrast, AnimateDiff, DynamiCrafter, and Gen2 generate videos with less motion. Pika struggles to produce object-consistent and high-quality video frames. Remarkably, our method can accurately understand the motion instructions in the text prompt (e.g., "A teddy bear walking ... beautiful sunset." and "A teddy bear running ... City." case).

4.2 Quantitative Evaluation

For a comprehensive comparison with the SOTA methods, we adopt three popular benchmarks (e.g., EvalCrafter [Liu et al., 2023], UCF-101 [Soomro et al., 2012], and MSRVTT [Xu et al., 2016] ) and human evaluation to evaluate the quality of video generation. Among them, EvalCrafter is a relatively comprehensive benchmark for video generation currently. UCF-101 and MSR-VTT are benchmarks commonly used in previous methods [Girdhar et al., 2023; Zhang et al., 2023]. Meanwhile, human evaluation can compensate for the inaccuracies in existing text-conditioned video generation evaluation systems.

EvalCrafter Benchmark. EvalCrafter [Liu et al., 2023] quantitatively evaluates the quality of text-to-video generation from four aspects (including Video Quality, Text-video Alignment, Motion Quality, and Temporal Consistency). Each dimension contains multiple subcategories of indicators shown in the Table. 1. As discussed in community [Liu and Cun, 2024], the authors acknowledge that the original manner of calculating the comprehensive metric was inappropriate. For a more intuitive comparison, we introduce a comprehensive metric for every aspect by considering each subindicators numerical scale and positive-negative attributes.

In detail, we compare the performance of the previous video generation SOTA methods (e.g., Pika [PikaLabs, 2024], Gen2 [Runway, 2024], Show-1 [Zhang et al., 2023], Lumiere [Bar-Tal et al., 2024], DynamiCrafter [Xing et al., 2023], and AnimateDiff [Guo et al., 2023b]) and exhibit in Table. 1. Our method demonstrates outstanding performance beyond existing methods at the Video Quality and Text-video Alignment aspect. Although Show-1 has the best Motion Quality (81.56), its Video Quality is poor (only 85.08). That indicates that it cannot generate high-quality videos with reasonable motion. However, our method has the second highest Motion Quality (72.99) and the best Video Quality (177.72), achieving the trade-off between quality and motion. The above results indicate the superiority of FancyVideo and its ability to generate temporal-consistent and motion-accurate video.

UCF-101 &MSR-VTT. Following the prior work [Zhang et al., 2023], we evaluate the zero-shot generation performance on UCF-101 [Soomro et al., 2012] and MSR-VTT [Xu et al., 2016] as shown in Table. 2. We use Fr´echet Video Distance (FVD) [Unterthiner et al., 2019], Inception Score (IS) [Wu et al., 2021], Fr´echet Inception Distance (FID) [Heusel et al., 2017], and CLIP similarity (CLIPSIM) as evaluation metrics and compared some current SOTA methods. FancyVideo achieves competitive results, particularly excelling in IS and CLIPSIM with scores of 43.66 and 0.3076, respectively. Besides, previous studies [Ho et al., 2022; Girdhar et al., 2023; Wu et al., 2023b] have pointed out that these metrics do not accurately reflect human perception and are affected by the gap between the distribution of training and test data and the image's low-level detail.

Human Evaluation. Inspired by EvalCrafter [Liu et al., 2023], we introduce a multi-candidate ranking protocol with four aspects: video quality, text-video alignment, motion quality, and temporal consistency. In this protocol, participants rank the results of multiple candidate models for each aspect. Each candidate model receives a score based on its ranking. For instance, if there are N candidate models ranked by video quality, the first model gets N -1 points, the second gets N - 2 points, and so on, with the last model receiving 0 points. Adhering to this protocol, we selected 108 samples from the EvalCrafter validation set and gathered judgments from 100 individuals. As depicted in Fig. 5, our method significantly outperforms text-to-video conversion methods, including AnimateDiff [Guo et al., 2023b], Pika [PikaLabs, 2024], and Gen2 [Runway, 2024], across all four aspects. FancyVideo demonstrates exceptional motion quality while preserving superior text-video consistency. Additionally, we conducted a similar comparison of four image-to-video methods, including DynamiCrafter [Xing et al., 2023], Pika, and Gen2, as shown in Fig. 5.

4.3 Ablation Studies

In this section, we conduct extensive experiments and exhibit detailed visual comparisons on the EvalCrafter benchmark [Liu et al., 2023] to thoroughly explore the effect of critical designs in CTGM. The ablation includes three key modules

"A teddy bear walking down the street during

a beautiful sunset."

"Waves crashing against a lone lighthouse,

ominous lighting."

Figure 4: Qualititive analysis. We compare the video generation results from AnimateDiff [Guo et al., 2023b], DynamiCrafter [Xing et al., 2023], Pika [PikaLabs, 2024], Gen-2 [Runway, 2024], and our FancyVideo.

(TII and TAR), each boosting video quality. As shown in Table 3, TAR significantly improves both metrics, highlighting the importance of temporal attention refinement. Adding TII further enhances performance by refining latent features and enabling frame-level text control.

5 Conclusion

In this work, we present a novel video-generation method named FancyVideo, which optimizes common text control mechanisms (e.g., spatial cross-attention) from the cross-frame textual guidance. It improves cross-attention with a well-designed Cross-frame Textual Guidance Module (CTGM), implementing the temporal-specific textual condition guidance for video generation. A comprehensive qualitative and quantitative analysis shows it can produce more dynamic and consistent videos. This characteristic becomes more noticeable as the number of frames increases. Our method achieves state-of-the-art results on the EvalCrafter benchmark and human evaluations.

Contribution Statement

Jiasong Feng, Ao Ma, Jing Wang, and Ke Cao contributed equally to this research. Ao Ma served as the project leader.