Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing

1 Introduction

Text-driven video editing, powered by generative diffusion models [Ho et al., 2020], [Song et al., 2020], [Rombach et al., 2021], has emerged as a transformative technology with broad applications in film, art, and advertising [Ho et al., 2022], [Hong et al., 2022], [Blattmann et al., 2023]. Recent advancements, such as Tune-A-Video [Wu et al., 2023], FateZero [Qi et al., 2023], and VideoComposer [Wang et al., 2024], have significantly enhanced the ability to edit objects, backgrounds, and styles in video while preserving overall scene consistency through optimized attention mechanisms and spatiotemporal continuity. Despite these successes, existing methods are constrained by the limited expressiveness of CLIP [Radford et al., 2021] word embeddings, which restricts their ability to perform nuanced edits on targets with specific attributes. Moreover, modifications to the target prompt often disrupt attention mechanisms, leading to inconsistencies in non-target areas before and after editing.

Inspired by Textual Inversion [Gal et al., 2022], a feasible approach is to leverage external concept word embeddings, which are optimized within CLIP text encoder [Radford et al., 2021] while keeping the diffusion model's parameters frozen. This technique allows the model to incorporate user-provided custom images for guided editing. However, the conventional Textual Inversion faces significant limitations when applied to video editing. Specifically, it lacks the ability to capture novel object information from arbitrary concept videos, resulting in word embeddings with insufficient fidelity to accurately describe target objects. Consequently, directly applying Textual Inversion to one-shot video editing often fails to generate satisfactory results for novel concept pairs, highlighting the need for a more robust and adaptive solution.

To this end, we propose Concept-Augmented Textual Inversion to enable one-shot flexible video editing based on external word embedding and target video. Specifically, we employ cutting-edge LoRA (Low-Rank Adaptation) modules to fine-tune attention value weights, focusing exclusively on the V (Value) projection (we elaborate on the rationale for tuning only V in subsequent sections). This approach effectively maintains the advantages of low VRAM overhead during tuning while preserving the plug-and-play capabilities of the model. In the context of V projection LoRA, our primary objective is to perform inversion while integrating novel object information from arbitrary concept videos for one-shot video editing. The textual inversion process relies on the pretrained denoising network's established text-image attention probability distribution to achieve accurate target representation. By fine-tuning only the V weights-rather than both Q (Query) and K (Key)-we enable the direct integration of new feature representations while minimizing disruptions to the pre-trained attention distribution. This selective finetuning strategy ensures stable training during the early stages, as it suppresses unnecessary changes to the model's foundational attention mechanisms. In addition, we introduce a Dual Prior Supervision (DPS) mechanism, designed to stabilize the generated video by supervising the cross-attention between the source and target prompts. This mechanism addresses the issue of attention dispersion, which often arises when modifications are made to the target prompt. By effectively controlling the attention distribution, DPS significantly enhances the consistency of non-target areas before and after video editing. Furthermore, it enriches the fidelity of the concepts in the edited results, ensuring that the final output maintains both spatial and temporal coherence.

• We propose a novel Concept-Augmented Textual Inversion (CATI) approach that reliably captures target attributes from user-provided concept videos, improving the fidelity and flexibility of video editing.
• We introduce a Dual Prior Supervision (DPS) mechanism that stabilizes video generation by supervising cross-attention between source and target prompts. DPS prevents attention dispersion caused by target prompt modifications, significantly improving the consistency of non-target areas before and after editing.
• We orchestrate a framework that allows users to extract concepts from custom videos and generate diverse edited videos through concept templates. This approach supports plug-and-play integration with stable diffusion models, enabling efficient and stable video editing.

2 RelatedWork

Text-Driven Video Editing. Current approaches for textdriven video editing mainly fall into three categories: finetuning video generation models [Zhao et al., 2023], [Wang et al., 2024], fine-tuning image generation models extended with temporal modules [Wu et al., 2023], [Qi et al., 2023], and combining NLA [Kasten et al., 2021] with pre-trained image generation models [Bar-Tal et al., 2022], [Lee et al., 2023], [Chai et al., 2023]. Recent advances have demonstrated various innovative approaches in these categories. For example, [Ku et al., 2024] employs a pretrained model for diverse video editing tasks, while GenVideo [Singer et al., 2025] utilizes a target-image-aware approach with InvEdit masks to overcome text-prompt limitations. [Bar-Tal et al., 2022], [Lee et al., 2023], [Chai et al., 2023] extract layered neural atlases from video to edit atlases which are further processed to synthesize videos; however, generating a neural atlas demands considerable computational time. Recently, Tune-A-Video [Wu et al., 2023] achieves one-shot video editing with improved inter-frame coherency by updating selfattention with sparse causal attention. FateZero [Qi et al., 2023] further proposes self-attention blending and incorporates attention control [Hertz et al., 2023] to enhance the ability of editing objects, background, and styles while maintaining scene consistency. For temporal consistency specifically, VidToMe merges self-attention tokens across frames, while [Geyer et al., 2023] leverages inter-frame correspondences to propagate features. In spatial editing, approaches like [Ceylan et al., 2023], [Cohen et al., 2024], [Liu et al.,

2024] improve results using spatial or temporal attention features in diffusion models. For editing targets with specific attributes, it becomes necessary to introduce external word embeddings. Our method supports the incorporation of external concept word embeddings. Furthermore, inspired by TuneA-Video [Wu et al., 2023] and FateZero [Qi et al., 2023], we introduce a dual prior supervision mechanism between video latents and word embeddings to enhance scene consistency before and after video editing based on attention control methods. Compared to existing approaches, our method focuses on attention supervision and control mechanisms and operates on a one-shot video editing paradigm.

Textual Inversion. [Gal et al., 2022] proposes a textual inversion method that optimizes newly added concept word embeddings in the CLIP [Radford et al., 2021] text encoder, supervised by the latent variable distribution of specific images in the diffusion model. However, using a pre-trained diffusion model for self-supervised text inversion may lead to under-fitting for some specific images due to the finite latent space. Although it's feasible to optimize full parameters of the denosing network in diffusion model with a smaller learning rate simultaneously, or to train it with frozen concept embeddings in the next stage, this process faces issues of easy over-fitting and high storage costs. Our method, building upon textual inversion [Gal et al., 2022], attempts to add LoRA [Hu et al., 2022] to the denosing network, optimizing them simultaneously with concept words at a smaller learning rate, to enhance the text editing capabilities of concept words.

Cross Attention Control and Supervision. Prompt-toPrompt [Hertz et al., 2023] proposes three attention control methods for stable text-driven image editing based on diffusion models: word swap, refinement, and reweighting. By applying the cross-attention probability map recorded from the original image latent variables and text to the denoising process of original image latent variables and edited text, it has achieved significant success in stable text-driven image editing [Avrahami et al., 2022], [Avrahami et al., 2023]. Additionally, [Qi et al., 2023] proposed self-attention blend effectively transfers the stability of text-driven image editing to video editing. Our method, built upon this foundation, introduces external concept words to support editing with higher degrees of freedom. Inspired by the work of [Yang and Tang, 2022], we introduce an attention supervision mechanism to address the issue of dispersed attention in editing words.

3 Method

3.1 Preliminaries

Textual Inversion. Textual inversion [Gal et al., 2022] learns new embeddings for user-provided visual concepts in the textual embedding space, associating them with pseudowords for use in new sentences for text-to-vision editing. The process uses a latent diffusion model, typically with a pretrained autoencoder and a noise prediction network: the encoder E maps image x to latent z = E(x), and the decoder D reconstructs x ≈ D(z). Textual inversion employs a CLIP [Radford et al., 2021] text encoder cθ with added concept words to encode conditional text y. The optimization objective is:

where zt is the noised latent at time step t, ϵ is the noise, and ϵθ is the noise prediction network.

Low-Rank Adaption. [Hu et al., 2022] proposes an efficient fine-tuning scheme based on matrix low-rank decomposition. For the pre-trained weight W0 ∈ Rd×k in the original model, it updates the weight as W = W0 + ∆W, where ∆W = BA, B ∈ Rd×r , A ∈ Rr×k , and r ≪ min(d, k). During the fine-tuning process, the pre-trained weight W0 is frozen, while A and B are trainable parameters. For the forward computation of the original weight h = W0x, the updated forward computation becomes:

Video Diffusion Models with Temporal Extensions. Tune-A-Video [Wu et al., 2023] introduces SpatioTemporal Attention (ST-Attn) to replace the original SelfAttention [Vaswani, 2017] in the 2D UNet. When calculating the keys K and values V , ST-Attn concatenates latent variables of the first and former frames of the video, leading to the attention result where the current frame attends to both the first and former frames. The specific operations for replacing K, V in Self-Attention are as follows:

where W K and W V are projection matrices for key and value respectively, zvi denotes the latent variable of the i-th frame of the video to the current attention layer, and [·] denotes concatenation.

3.2 Stabilized Text-Driven Video Editing

Our training and inference pipelines are shown in Fig. 1. We use a UNet initialized from Stable Diffusion's pre-trained 2D

UNet [Rombach et al., 2021] as the noise predictor. To handle 3D video inputs, we replace spatial self-attention layers with ST-Attn (Eq.3). Following FateZero [Qi et al., 2023], we add LoRA-based temporal convolution layers after spatial convolutions, and temporal self-attention with zero-initialized linear output after cross-attention. These new modules are residually connected to the originals.

Our approach for stabilized text-driven video editing has two learning phases. In the first phase, we introduce ConceptAugmented Textual Inversion (CATI) to adapt the diffusion model to new visual concepts. In the second phase, we tune partial parameters of the temporally extended diffusion model to suppress unintended changes in edited videos by calibrating cross-attention results.

Concept-Augmented Textual Inversion. Textual inversion [Gal et al., 2022] learns to represent a specific set of user-provided images with pseudo-words in the latent space, offering an intuitive way for natural language-guided image editing. We incorporate this technique into our framework to facilitate video editing. However, due to the self-supervised nature within the limited latent space of the pre-trained diffusion model, the vanilla textual inversion often results in varied performance in terms of quality and efficiency for different image sets, requiring meticulous adjustments for learning rates and iteration counts.

To alleviate this issue, we draw inspiration from existing parameter-efficient fine-tuning techniques and propose adding LoRA modules [Hu et al., 2022] to the value projection parameters in the cross-attention layers of the UNet. Consequently, the values V are updated to LoRA(V ) according to Eq. (2). The rationale behind our approach is that we aim to enhance the expressiveness of the pre-trained diffusion model by slightly adjusting its capacity to accommodate new visual concepts while preserving its original generation capability. Besides, inserting LoRA modules not only augments textual inversion with low storage overhead but also maintains a plug-and-play characteristic during inference. Textual inversion process relies on the pre-trained denoising network established text-image attention probability distribution to achieve accurate target representation. In this context, fine-tuning only the V weights instead of Q and K allows new feature representations to be directly integrated while suppressing changes towards the pre-trained attention distribution to enable stable training during the early stages.

We train the concept-word embeddings of textual inversion and the weight parameters of LoRA modules in an end-toend manner (see orange blocks in Fig. 1), where the learning rate for LoRA parameters is relatively smaller than that for concept-word embeddings to avoid over-fitting. Denote the noise prediction network with LoRA modules loaded on value projection parameters as ϵθL , the optimization objective of concept-augmented textual inversion is then updated from Eq. (1) to the following:

Model Tuning with Dual Prior Supervision. After learning concept-augmented textual inversion, we adapt and tune the video diffusion model for text-driven video editing in line with the paradigm of few-shot learning. Specifically, we learn the LoRA-structured temporal convolution layers, the query projection weights within spatio-temporal attention layers and cross-attention layers, and the temporal selfattention layers (see red blocks in Fig. 1). These parameters are selected for updates during training due to their strong relevance to the temporal modeling of 3D videos. To attain more stable and higher quality editing results, we tried directly integrating existing attention control techniques [Hertz et al., 2023] in an early attempt; however, we found that when applying text-driven video editing types such as word swap, the dispersion phenomenon of cross-attention between text embeddings and video latents leads to reduced stability in editing results, which is shown in Fig 5. To address this challenge, we propose a dual prior supervision mechanism, which includes a source cross-attention mask (scam) loss and a target cross-attention mask (tcam) loss.

The scam loss is designed to reduce the attention influence of the words to be replaced in the source prompt on irrelevant frame areas (see the first row in Fig. 2). It is also applied to modulate attention between concept words and concept videos (see the second row in Fig. 2). Specifically, for K cross-attention layers in the UNet, we record crossattention matrices Ms between the words and the video frame latents in each cross-attention layer. To obtain ground truth for optimization, we use an off-the-shelf object detection network OWL-ViT [Minderer et al., 2022] to localize objects in video frames and generate corresponding binary pseudolabels Mgt s . We further apply max pooling to generate K pseudo-labels, each with a designated resolution Pk. The loss is then calculated as the mean absolute loss on irrelevant areas:

The tcam loss is introduced to diminish the attention influence of the target words in the edited prompt to further promote the consistency of irrelevant areas before and after video editing (see the third row in Fig. 2). Similar to the scam loss, we obtain cross-attention matrices Mt and pseudo-labels Mgt t between the target words in the edited prompt and the video frame latents. The loss is computed as:

Let the trainable parameters during the model tuning phase be denoted as ϵθT . The noise prediction loss Lnoise is then obtained by substituting ϵθ in Eq. (1) with ϵθT . Given α and β as the weighting coefficients for our proposed scam loss and tcam loss, respectively, the total loss for model tuning with dual prior supervision is formulated as:

Inference. As shown in Fig.1, the pipeline consists of an inversion stage with the source prompt and an editing stage with the modified prompt. During inversion, we cache selfand cross-attention matrices at each step, which are later used to control attention in editing. Specifically, we blend self-attention matrices to preserve semantic layout [Qi et al., 2023], and swap cross-attention matrices for changed words and video latents [Hertz et al., 2023].

4 Experiments

4.1 Settings and Datasets

Our experiments are conducted on a machine equipped with an NVIDIA GeForce RTX 4090. During the concept augmented textual inversion stage, we set the learning rate for CLIP [Radford et al., 2021] word embeddings to 1 × 10-3 , and the learning rate for LoRA modules inserted into the UNet to 1 × 10-5 , with the number of training steps set to Additionally, we randomly sample frame numbers within the range [4, 8] from the concept video during training, to prevent the inversion process from over-fitting to a fixed frame number. For the video diffusion model fine-tuning stage, we empirically set α = 0.1 and β = 0.1 in Eq. (7). The training steps above all use AdamW [Loshchilov and Hutter, 2017] optimizer. In the inference stage of video editing, the guidance scale is set to 12.5, the number of DDIM Inversion steps is T = 50, and the self-attention blending and cross-attention swap steps are within the interval [0, 0.7T]. To evaluate our proposed method, we used a portion of the DAVIS [Caelles et al., 2019] dataset and clip videos from the internet to construct video editing pairs, either with or without concept videos.

4.2 Metrics

Frame Consistency. To compare the coherence of the video frames F, we refer to the metric used in [Wu et al., 2023], [Hessel et al., 2021], which calculates the average cosine distance d between features (vi, vj ) of each two different frames (fi, fj ) encoded by the CLIP visual encoder [Radford et al., 2021], as Eq. (8). Here, fi, fj ∈ F, fi ̸= fj, and D denotes the set of the vector pairs (vi, vj).

Masked Peek-Signal-Noise Ratio. To compare the stability of the video non-target areas before and after target editing, we design a Masked Peak Signal-to-Noise Ratio (MPSNR) metric. We use the OWL-ViT [Minderer et al., 2022] open-vocabulary object detection model with text pseudolabels to estimate the bounding box mask M of the edited target. We then compare the average peek-signal-noise ratio of the original video frames and the edited video frames after applying this mask. The calculation formula for the specific function f for the Mean Squared Error (MSE) used as input is as follows, where M ∈ RH×W , I s ∈ RH×W×C , and I e ∈ RH×W×C refer to the mask value, the frame pixel value of video before and after editing, respectively.

Concept Consistency. We employ a multi-step approach to evaluate the correlation between the video editing results guided by the concept video and the concept video itself while minimizing interference in non-target areas. First, we use a pre-trained OWL-ViT [Minderer et al., 2022] model in conjunction with pseudo-label prediction to generate object masks for both videos. We then extract pixel segments of the target objects from both the edited video and the concept video. Finally, we leverage the CLIP model to predict visual encoding vectors for these extracted segments and calculate the average cosine similarity between them.

4.3 Comparisons with ExistingMethods

Quantitative Evaluation. As illustrated in Tab. 1 and Tab. 2, we assess text-driven video editing results in three aspects. Compared with existing methods that extend and fine-tune the Stable Diffusion model, including TuneA-Video [Wu et al., 2023], FateZero [Qi et al., 2023], RAVE [Kara et al., 2024], and MotionDirector [Zhao et al., 2023], our approach demonstrates superior inter-frame coherence in terms of the Frame Consistency Metric. To evaluate the consistency of unrelated areas before and after video editing, we employ M-PSNR as a reference metric, and our method achieves the highest score by a large margin. Concretely, our method outperforms MotionDirector [Zhao et al., 2023] by a noticeable 6.98 M-PSNR in editing with concept video. This is attributed to our proposed prior supervision mechanism, which effectively reduces the editing noise in non-target areas for both source and concept videos. Furthermore, to evaluate the target fidelity in concept and edited videos, we utilize Concept Consistency as a reference metric, and our method demonstrates greater fidelity compared to others.

Qualitative Evaluation. Fig.3 shows visual comparison results of video editing with and without concept video guidance. Our method maintains content consistency in nontarget areas before and after editing. With concept videos, it effectively introduces visual concepts from the concept video into the edited result. For instance, in Fig.3 (Setting I), our method successfully replaces man with '$OPTIMUS', while others fail to preserve background or transfer the complete target shape. Other approaches commonly face instability in non-target areas. Tune-A-Video [Wu et al., 2023] encounters dispersed cross-attention issues due to fine-tuning with only one video-text pair. While FateZero [Qi et al., 2023] and RAVE [Kara et al., 2024] mitigate this through cross-attention manipulation or noise shuffling, their direct concept-driven editing compromises non-target consistency and concept fidelity. MotionDirector [Zhao et al., 2023] extracts targets via its trainable spatial path, but coupled spatialtemporal paths provide unstable guidance, causing non-target inconsistencies. Our CATI and DPS effectively maintain nontarget content consistency while accurately capturing userprovided concept attributes.

4.4 Ablation Study

Concept Augmentation Alleviates Under-Fitting of Textual Inversion. In this work, we draw on the idea of Textual Inversion (TI) from text-to-image generation and apply

it to text-driven video editing to address the embedding of external concept words. However, simply applying TI may lead to under-fitting, resulting in a lack of realism. For instance, in the results shown in Fig. 4(a) and Fig. 4(c), where the keywords 'jeep' are altered to '$LAMBO' and '$CYBERTRUCK', although some attributes (e.g., shape) of the target concepts are partially retained, the results appear to "drift" due to insufficient inductive bias. In contrast, the concept-augmented textual inversion (CATI) can effectively capture the color, shape, and other attributes, as demonstrated in Fig. 4(b) and Fig. 4(d). CATI provides more detailed features for editing, significantly improving inversion fidelity.

Dual Prior Supervision Improves Stability and Fidelity. In this work, we propose a Dual Prior Supervision (DPS) strategy, which consists of two main components (See Sec. 3.2): scam loss and tcam loss. Both components play crucial roles in maintaining the stability of the target generation. By comparing the attention regions in Fig. 5 (a) (w/o tcam, w/o scam), Fig. 5 (b) (w/o tcam, w/ scam), and Fig. 5 (c) (w/ tcam, w/o scam), we can conclude that both scam and tcam (Fig. 5 (d)) significantly reduce background disturbances and improve stability. However, the generated video results reveal that using either component alone cannot effectively capture attributes of the target object, such as the color of the car. DPS combines both components, not only enhancing the stability of the background in the target results but also capturing the target object's attributes more accurately, thereby improving the fidelity of the edited concept target.

Tuningw/ Concept Video Produces StylizedResults. Recall that we construct the target videos in this work by templating the concept pairs to make the editing process more flexible. To explore the impact of the concept video in Setting I (Fig. 3), we conduct a simple experiment as shown in Fig. 6. As shown in Fig. 6(a) and Fig. 6(b), tuning models with both concept video and concept prompt produces more stylized videos. The possible explanation lies in that the concept video alleviates the overfitting issue.

5 Limitations and FutureWork

Mismatch when Significant Deformation. Our method effectively mitigates the inconsistency in non-target areas caused by attention dispersion in video editing methods using attention replacement mechanisms, it may struggle when a single concept video guides target replacement in cases of significant deformation in the source video, such as running people. For instance, there may be insufficient detailed correspondences between the internal parts of the replacing and replaced targets during deformation, such as moving arms and legs. Potential solutions include ControlNet [Zhang et al.,

2023], OpenPose [Cao et al., 2019] and Sign-D2C [Tang et al., 2025] which utilize motion conditions, like human pose or sign language, to guide the video editing process.

6 Conclusion

In this paper, we present an improved concept-augmented video editing approach that flexibly produces diverse, stable target videos by leveraging abstract conceptual pairs. Specifically, we introduce Concept-Augmented Textual Inversion (CATI) to capture user-defined target concepts, enabling a plug-and-play, stable diffusion pipeline for more stylized editing. We further propose a Dual Prior Supervision (DPS) mechanism to align cross-attention between source and target prompts, preventing unintended changes in non-target regions. Experimental results show that our method significantly enhances flexibility, consistency, and stability in textdriven video editing.

Acknowledgments

This work is supported by the National Natural Science Foundation of China under Grant No. 62472139, by the Anhui Provincial Natural Science Foundation, China (Grant No. 2408085QF191), the Fundamental Research Funds for the Central Universities (Grants No. JZ2024HGTA0178, JZ2023HGQA0097), by the Open Project Program of the State Key Laboratory of CAD&CG (Grant No. A2403), Zhejiang University.