Drafting and Revision: Advancing High-Fidelity Video Inpainting

1 Introduction

Video inpainting aims to fill the missing regions of a video with spatial-temporally coherent contents, which is a fundamental visual restoration task. High-quality video inpainting can benefit general users in various applications, such as object removal [Wu et al., 2023c], video restoration [Wang et al., 2024], autonomous driving [Zhang et al., 2023], and so on. Unlike image inpainting [Liu et al., 2024a; Zhuang et al., 2024], which primarily focuses on the spatial dimension, video inpainting pays more attention to the temporal information. Directly using image inpainting methods to individual frames for video inpainting will neglect the motion continuity between frames, resulting in flicker artifacts.

†Corresponding author.

Recently, several deep learning-based video inpainting methods [Li et al., 2020; Liu et al., 2021; Zhang et al., 2022c; Wang et al., 2023; Wu et al., 2021] have been proposed and have achieved significant results. Nevertheless, these methods always treat the missing regions as a whole and employ a hybrid objective consisting of a reconstruction loss (L1/L2 norm) and an adversarial loss to train the model, resulting in over-smooth generated missing contents compared to reveal realistic detail, as illustrated in Fig. 1. On the one hand, these methods treat the missing regions as a whole, i.e., all pixels are viewed equally. They do not distinguish between flat regions and texture details, which are contained in lowfrequency and high-frequency components respectively. In this way, the trained models will be easily dominated by flat regions which are the most common [Wu et al., 2023b]. On the other hand, the reconstruction loss and the adversarial loss tend to synthesis contents at different frequencies, i.e., the former focuses on recovering the low-frequency global structures [Pathak et al., 2016], while the latter prefers to generate the high-frequency texture details [Yu et al., 2021]. Simply combining these two losses may cause inter-frequency conflicts, leading to much less favourable results.

Recall the process of a painter inpainting a corrupted painting, we can find that a common practice, especially for a beginner, is to draw a draft first to capture the global structure of the painting, and then gradually revise the local details based on the global structure, rather than directly completing the final inpainting part-by-part. Inspired by such a "drawing a draft first and revising the details later" manner [Lin et al., 2021], we propose a novel Drafting-andRevision Completion Network (DRCN) for video inpainting. DRCN decomposes the video frame into low-frequency and high-frequency components, and designs the a Drafting Network and a Revision Network to complete them respectively. In this way, we can not only avoid inter-frequency conflicts by applying adversarial loss and L1 loss to the high-frequency and low-frequency branches separately, but also solve the problem of varying difficulties in generating low-frequency semantics and high-frequency details.

Specifically, we first adopt Laplacian transform to decompose the frames into low-frequency and high-frequency components. By doing this, flat regions common in frame are recorded in low-resolution low-frequency components, while texture details are mainly concentrated in high-resolution high-frequency components. Next, we develop a Drafting Network to complete the semantic structure of missing regions in low-frequency components at low resolution, benefiting from its larger receptive field and less local details. Thereafter, a Revision Network is designed to revise the highfrequency local details of the missing contents at 2× resolution. The Revision Network utilizes the draft generated by Drafting Network to guide the high-frequency components in generating the missing contents. Finally, the completed lowfrequency and high-frequency components are aggregated to yield the final inpainting result by inverse Laplacian transform. Notably, our Revision Network can be stacked in a pyramid manner to complete high-frequency details at higher resolution, which can provide a feasible solution for the highresolution video inpainting. Extensive experimental results demonstrate that DRCN can generate the missing contents with richer textures compared to baselines.

To sum up, our contributions are summarized as follows:

• A novel Drafting-and-Revision Completion Network (DRCN) is designed to effectively mitigate the interfrequency conflict in video inpainting.
• A feasible high-resolution video inpainting solution is first attempted. Our network can handle higher resolution videos on limited GPU memory.
• Extensive experiments on two benchmark datasets, including Youtube-vos [Xu et al., 2018] and DAVIS [Perazzi et al., 2016], demonstrate the superiority of our DRCN in both quantitative and qualitative evaluations.

2 RelatedWork

Video Inpainting. Recently, several deep learning based video inpainting methods have been proposed and achieved great progress. According to the network architectures involved, these methods can be summarised into three groups.

3D CNNs-based Methods: Some researchers [Chang et al., 2019; Kim et al., 2019] utilize 3D CNNs to integrate spatial-temporal information and fill in missing regions. Although they have produced promising results, the computational complexity of 3D CNNs is relatively higher, which limits their practical application [Wu et al., 2023a].

Optical Flow-based Methods: Unlike 3D CNNs-based methods, optical flow-based methods [Xu et al., 2019; Gao et al., 2020; Zhang et al., 2024] formulated the video inpainting as a flow-guided pixel propagation task. They first completed the optical flow by a flow completion network, and then propagated the relevant pixels using the completed flow into missing regions. Despite achieving encouraging results, they still suffer from challenges in propagating valid pixels from distant frames. In a sense, their performance significantly decrease when the missing regions are large and slowmoving [Zhou et al., 2023].

Attention-based Methods: Attention [Li et al., 2023; Li et al., 2025a; Wang et al., 2025; Liu et al., 2024b; Li et al., 2024; Li et al., 2025b] has been proven to model long-distance dependencies, some methods [Liu et al., 2021; Zhang et al., 2022b; Zhou et al., 2023; Wu et al., 2024] incorporated attention mechanism to extend the limited temporal receptive field. These methods retrieve relevant information from long-distance frames by this mechanism and adopted weighted operation to generate missing contents. Among these methods, Zeng et al. [Zeng et al., 2020], Liu et al. [Liu et al., 2021], Zhang et al. [Zhang et al., 2024], and Wu et al. [Wu et al., 2024] employed transformers to retrieve similar features in a considerable temporal receptive field, resulting in high-quality video inpainting.

In spite of the promising results shown by these methods, over-smoothed missing contents are generated, failing to infer realistic details. Meanwhile, these methods can only handle low resolution videos (typically smaller than 1K) due to constraints in GPU memory and computation time, and are ineffective for high-resolution videos in real-world scenarios. Coarse-to-Fine Strategy. In the field of visual restoration, the coarse-to-fine strategy has been proposed and utilized in various tasks, such as image super-resolution [Tian et al., 2021], video super-resolution [Xiao et al., 2023], and so on. Although these methods have achieved remarkable results by coarse-to-fine strategy, they usually treat the content as a whole and adopt a hybrid objective consisting of reconstruction loss and adversarial loss to train the network, leading the trained model to generate compromised results.

Unlike the coarse-to-fine approach, our drafting-andrevision strategy employs the Laplace transform to decompose the frame into low-resolution low-frequency components and high-resolution high-frequency ones, and processes them separately in the drafting and revision stages. Such a design not only avoids inter-frequency conflicts by applying adversarial loss and L1 loss to different frequency branches separately, but also enables stacking the Revision Networks in a pyramid manner to process high-resolution video.

3 ProposedMethod

3.1 Formulation and Overview

Problem Formulation. Assume X = {xi ∈ Rh×w×3 }T1 is a corrupted video with length T. The binary mask M =

{mi ∈ Rh×w×1 }T1 denotes the missing regions of corresponding frames. For each mask mi, "0" indicates that the valid region of xi, and "1" stands for the missing regions. The goal of video inpainting is to generate a completed video ̂Y = {̂yi ∈ Rh×w×3 }T1 , which should be consistent with ground truth video Y = {yi ∈ Rh×w×3 }T1 in both spatial and temporal dimensions.

In practice, we usually use a deep neural network DNN (·) to predict ̂yt frame by frame i.e., ̂yt = DNN(xt, Xt+n t-n , mt). Here, xt is the current frame that needs to be inpainted, called target frame. Xt+n t-n = {xt-n, . . . , xt-1, xt+1, . . . , xt+n} denotes a short clip of neighboring frames with a center moment t and a temporal radius n, namely reference frames. Existing methods [Yu et al., 2019; Zeng et al., 2020; Liu et al., 2021; Zhang et al., 2022b; Zhou et al., 2023; Wu et al., 2024] always treat the missing contents as a whole and employ a hybrid objective Lhy to train the DNN(·),

where Lre is the reconstruction (L1 or L2) loss, Ladv is the adversarial loss, and λre is the trade-off parameter.

We argue that these methods treat all pixels equally, resulting in the completed regions being too smooth and lacking realistic details. On the one hand, the reconstruction objectives of the missing regions are not consistent regarding different low-level frame elements, e.g., smoothness preserving for flat regions, sharpening for edges and textures. On the other hand, the reconstruction loss Lre prefers to focus on low-frequency global structures [Deng et al., 2019; Yu et al., 2021], while the adversarial loss Ladv tends to concentrate on high-frequency texture details [Pathak et al., 2016]. Simply combining them like Eq.(1) will lead to inter-frequency conflicts. To alleviate such conflicts, existing methods [Yu et al., 2019; Zeng et al., 2020; Liu et al., 2021; Zhang et al., 2022b; Zhou et al., 2023; Wu et al., 2024] attempt to balance the Lre and Ladv by adjusting the parameter λre. However, the missing contents in spatial domain are still generated with mixed frequency [Yu et al., 2021]. Not only is this strategy sub-optimal, but adjusting hyper-parameters λre is trivial. Therefore, separate treatment of low-frequency and high-frequency of missing regions and explicitly applying the Ladv and Lre losses to different branches, is necessary to generate the missing contents with more realistic details.

Network Design. In this paper, we propose a Draftingand-Revision Completion Network, named as DRCN. The pipeline of our DRCN is illustrated in Fig. 2. Given the target frame xt and the reference frame xr ∈ Xt+n t-n , we first decompose them into low-frequency component lt, lr ∈ R h2 × w2 ×3 and high-frequency component ht, hr ∈ Rh×w×3 by Laplacian transform. Notably, lt and lr mainly record the global semantic structure, while ht and hr generally contain their corresponding detailed information, e.g., edges and textures. Then, the low-frequency component lt and high-frequency component ht are fed into Drafting Network and Revision Network to conduct the completion, respectively. Finally, the completed low-frequency and high-frequency components are aggregated to generate final inpainting result ̂yt ∈ Rh×w×3 by the inverse Laplace transform. In this way, L1 loss Lre and adversarial loss Ladv can be applied to lowfrequency and high-frequency components independently, effectively mitigating inter-frequency conflicts. Furthermore, Revision Network can be stacked in a pyramid manner to inpaint high-resolution video. In the following, we will introduce the Drafting Network and Revision Network in detail.

3.2 Drafting Network

At low resolution, the semantic structure is easier to complete due to the large receptive field and less local details [Wu et al., 2023b]. Based on this fact, we design a Drafting Network to complete the semantic structures of the missing regions at low resolution. As shown in Fig. 2, our Drafting Network is built upon an encoder-decoder architecture, which consists of a frame-level encoder, a temporal alignment module, a feature aggregation module and a frame-level decoder. Temporal alignment and feature aggregation modules are the core components of the Drafting Network. The former performs the feature alignment to eliminate image variations between the reference frame and the target frame, while the latter aggregates the aligned features of the reference frame to generate the missing contents of the target frame.

Self-Supervised Flow-Guided Temporal Alignment. Since the movement of the camera or object cause the image variation, it is difficult to directly utilize the reference frames to complete the the missing regions of target frame. Therefore, an extra alignment module is necessary to eliminate the image variation between the reference frame and the target frame. Benefit from the capability of deformable convolution (DCN) [Dai et al., 2017] to handle complex geometric transformations, some works [Wang et al., 2019; Tian et al., 2020; Wu et al., 2023b] have proposed various forms of DCNbased temporal alignment module to achieve alignment between reference frame and target frame. Although these modules achieve excellent alignment results, they often suffer from offset overflow during training, thereby having a negative impact on model performance. To alleviate this problem, researchers [Chan et al., 2022; Zhang et al., 2022a; Liang et al., 2022; Wu et al., 2023a] used the optical flow between reference frame and target frame as base offset of DCN to train the network. However, these alignment module still face the following challenges:

• They typically relied on a heavyweight pre-trained DNN (e.g., PWC-Net [Sun et al., 2018]) to generate the optical flow, which significantly increases the computational cost, thus limiting their practical application.
• They were difficult to converge due to the lack of constraints during the training.

In fact, the optical flow in these alignment module only serves as a base offset to guide the training of DCN, which means that a coarse-grained optical flow is sufficient for the requirements [Zhang et al., 2022a; Wu et al., 2023a]. In other words, precise optical flow generated by the heavyweight DNN is redundant for these alignment module. Furthermore, target frames can act as labels to force the reference frames closer to them in a self-supervised manner during alignment, achieving fast convergence of the model.

For this purpose, we design a self-supervised flow-guided temporal alignment module. Specifically, we estimate the optical flow using a lightweight motion estimator that stacks 3 convolutional layers. Such a design not only significantly reduces the computational cost, but also allows the network to be trained from scratch to generate more suitable optical flows for video inpainting. Besides, we introduce an alignment loss La as a self-supervised constraint.

As shown in Fig. 3, for the features f r and features f t corresponding to lr and lt are extracted by the frame-level encoder, we first estimate the optical flow or→t by a lightweight motion estimator M(·), and generate warped features f r by a warping operation [Wang et al., 2019] W(·),

Then, the warped features f r and the features f t are used to compute the offsets θr and modulation masks ωr,

where Cθ(·) and Cω(·) presents 2D convolution, [·, ·] denotes the concatenation, and σ(·) indicates sigmoid function. Notably, when calculating the offsets θr based on Eq.(4), we consider the residuals of optical flow or→t as a base offset of DCN instead of directly computing the offsets. Such a strategy can effectively mitigate the offset overflow [Chan et al., 2022; Wu et al., 2023d] of DCN during the training.

Next, the aligned reference frame features ˜f r can be acquired by a DCN layer DCN(·):

Finally, to obtain a more robust alignment feature, we fuse f r calculated by Eq.(3) and ˜f r calculated by Eq.(6) to generate the final aligned features f ar ,

where Cf (·) denotes feature-level fusion operation consisting of convolutional layers.

Temporal-Adaptive Feature Aggregation. Due to occlusion, blurry regions and parallax problems, different reference frames are not equally beneficial for reconstructing the missing contents [Wu et al., 2023a; Chen et al., 2025]. To solve this issue, we construct a temporal-adaptive feature aggregation module to generate the missing semantic structures.

Specifically, we first compute the attention weight sr of the aligned reference frame features f ar by a softmax function:

where Cq(·) and Ck(·) denote 1 × 1 2D convolution. Then, the attention-modulated features qr can be obtained by qr =

Cv(f ar ) ⊙ sr, where Cv(·) indicates 1 × 1 2D convolution and ⊙ presents the element-wise multiplication.

After obtaining the all attention-modulated features qr , r ∈ {t-n, . . . , t-1, t+1, . . . , t+n}, the completed features ̂f t can be generated by a aggregation convolutional layer Ca(·),

Here, the size of the mask mt is resized to fit the size of the features f t . The final completed low-frequency component̂lt can be obtained by decoding ̂f t with the frame-level decoder.

3.3 Revision Network

The low-frequency component ̂lt generated by Drafting Network contains the complete semantic structure but lacks detailed information, e.g., edges and textures. Adding the highfrequency information to the ̂lt will produce clearer result with richer details. Therefore, it is quite necessary to design a Revision Network to complete the high-frequency component ht. A naive design of Revision Network is to directly replicate our Drafting Network. Nevertheless, such a design will consume a lot of GPU memory since Revision Network needs to search for relevant information from multiple reference frames simultaneously, which is extremely detrimental to high-resolution video inpainting. For this purpose, we develop a Revision Network that exploits the inpainted lowfrequency component̂lt to guide the completion of the highfrequency component ht. In the real implementation, due to the high sparsity of the ht, directly summing or concatenating ̂lt and ht to generate ̂ht will greatly suppress the highfrequency information. Therefore, aligning thêlt with the ht is a crucial step in generating consistent and realistic highfrequency missing contents.

Specifically, we first encode the ht into the feature gt by the frame-level encoder, where gt = {g1t , . . . , get } ∈ R h2 × w2 ×c and e = h2 × w2 . Then, the self-attention score wi,j of the feature ̂f t obtained by Eq.(9) is calculated as follows,

where 1 ≤ i, j ≤ e, ̂ Ck(·) and ̂ Cq(·) denote 1 × 1 2D convolutions. The acquired attention map w depicts the correlation among the completed low-frequency feature. We aggregate the high-frequency feature of the valid regions to reconstruct the missing contents of gt by

where zi is i-th aggregation feature and ̂ Cv(·) is a 1 × 1 2D convolution layer.

Due to the sparseness of high-frequency feature, the magnitude of aggregation feature in Eq.(11) is relatively small. Inspired by frequency region attentive normalization [Yu et al., 2021], we employ the parameter-free positional normalization [Li et al., 2019] to normalize z while the preserving structural information. Similarly, the parameter-free positional normalization is also applied to ̂f t . The aligned lowfrequency feature et is computed as follows,

where µf and σf are the mean and standard deviation of ̂f t along the channel dimension, respectively. Cγ(·) and Cβ(·) denote convolution operation. Finally, the completed highfrequency feature ̂gt is generated by a 1 × 1 2D convolution layer Cg(·), i.e., ̂gt = Cg(et, gt ).

4 Experiments

4.1 Experimental Setting

Datasets. Following previous works [Ren et al., 2022; Zhou et al., 2023; Wu et al., 2024], two most commonly used datasets (Youtube-vos [Xu et al., 2018] and DAVIS [Perazzi et al., 2016]) are considered to verify the effectiveness of our method. The Youtube-vos [Xu et al., 2018] dataset contains 4453 videos with various scenes, and is split into three parts containing 3471, 474 and 508 videos for training, validation and testing, respectively. As for the DAVIS [Perazzi et al., 2016] dataset, it contains 150 high-quality videos of challenging motion-blur and appearance motions. Consistent with existing studies [Zhou et al., 2023; Yu et al., 2023; Zhang et al., 2024; Wu et al., 2024], 60 videos are used for training and 90 videos are utilized for testing.

Baselines and Evaluation Metrics. We select nine recently video inpainting methods as our baselines, including VINet [Kim et al., 2019], CPVINet [Lee et al., 2019], FGVC [Gao et al., 2020], STTN [Zeng et al., 2020], FuseFormer [Liu et al., 2021], E2FGVI [Li et al., 2022], FGT [Zhang et al., 2022b], ProPainter [Zhou et al., 2023], and WaveFormer [Wu et al., 2024]. To ensure the fairness of the results, these baselines are fine-tuned using their released models and codes, and we report best results. Furthermore, PSNR [Haotian et al., 2019], SSIM [Lin et al., 2021], LPIPS [Zhang et al., 2018], and Ewarp [Lai et al., 2018] are used to evaluate inpainting quality.

4.2 Experimental Results and Analysis

Quantitative Results. Tab. 1 shows quantitative results on Youtube-vos and DAVIS datasets under 256×256 resolution. As can be seen from Tab. 1, our method significantly outperforms all competitive baselines in four metrics. In particular, our method achieves 1.18%, 1.03%, 7.43% and 12.64% relative improvements on Youtube-vos dataset and 1.48%, 1.13%, 14.43%, and 8.57% relative improvements on DAVIS dataset regarding the PSNR, SSIM, Ewarp, and LPIPS. These quantitative results validate that our proposed method can generate results with more visually realistic (PSNR, SSIM, and LPIPS), and more temporally consistent (Ewarp).

Qualitative Results. In Fig. 4, we visually compare the qualitative results of our method with five baselines (FuseFormer [Liu et al., 2021], E2FGVI [Li et al., 2022], FGT [Zhang et al., 2022b], ProPainter [Zhou et al., 2023], and WaveFormer [Wu et al., 2024]) under three different setting: (a) curve mask, (b) stationary mask, and (c) object mask. As can be seen, frames inpainted by our method have more realistic details which are significantly better than baselines. Our inpainted results not only have complete semantic structure, but also their details are more vivid and clear. Furthermore, to verify the effectiveness of "Drafting and Revision" framework, we compare the inpainting results of our method with ProPainter and WaveFormer on different frequencies in Fig. 5. As can be seen, the low-frequency semantics among these three methods exhibit negligible differences, whereas our method significantly outperforms ProPainter and WaveFormer in capturing high-frequency details. This indicates that our proposed "Drafting and Revision" framework can generate richer high-frequency details.

4.3 High Resolution Video inpainting

Recently, significant progress has been made by deep learning based video inpainting methods. However, due to the memory limitations of hardware devices, these methods can only complete videos with resolutions smaller than 1K. Naively using the"down-sampling-inpainting-up-sampling" technical pipeline to complete high-resolution video merely yield blurry results, which is disadvantageous in real-world applications. In our framework, Revision Network can be stacked more layers to handle high-resolution video. Tab. 2 shows the quantitative results at three different resolutions on Youtubevos [Xu et al., 2018] dataset, which were tested under a RTX 2080 Ti GPU. As shown in Tab. 2, as the video frame resolution increases, the baseline models suffer from GPU memory overflow. In contrast, our DRCN constructed by stacking three layers of Revision Network can effectively inpaint videos at a resolution of 2048 × 2048. Furthermore, Fig. 6 illustrates inpainted examples of our method at 2048 × 2048 resolution. It can be observed that DRCN generates missing contents with rich details for high-resolution videos.

4.4 Ablation Study

Drafting Network. To demonstrate the effectiveness of Drafting Network, we replaced the Drafting Network in our framework with two baseline models (STTN [Zeng et al., 2020], and E2FGVI [Li et al., 2022]) and compared their results with our full model. As shown in Tab. 3, our model outperforms STTN+ Revision and E2FGVI+Revision on four metrics. These results demonstrate that the proposed Drafting Network is beneficial and necessary for completing the global semantic structure of the video frame.

Revision Network. In Fig. 7, we visually compared the inpainting results of E2FGVI+Revision and our full model. As observed from Fig. 7, the E2FGVI+Revision and our full model generate more reasonable and detailed missing contents than the E2FGVI [Li et al., 2022] and Drafting Network. These results indicate that Revision Network has significant benefits for the final video inpainting results, which further verifies the necessity of the proposed "Drafting and Revision" framework in video inpainting task.

Alignment Manner. This section compares different temporal alignment manner. From Tab. 4, the following conclusions are confirmed: i) Performing temporal alignment can improve the performance (2nd-6th rows). ii) The reference frame alignment effect of our alignment module (6th row) outperforms traditional flow-based warping alignment methods (2nd row) and traditional deformable convolution alignment methods (3rd row). iii) Utilizing a lightweight estimator to calculate optical flow between frames does not significantly reduce the performance of the temporal alignment module (5th row). iv) The strategy of aggregating the aligned reference feature ˜f r and warped reference feature f r by Eq.(7) can obtain more a robust alignment feature (6th row). v) Using the alignment loss La to train the temporal alignment module in a self-supervised manner can further improve the alignment performance of reference frame (7th row).

5 Conclusion

This paper propose a novel Drafting-and-Revision Completion Network (DRCN) for video inpainting, which contains two main sub-networks, i.e., Drafting Network and Revision Network, where the former learns to complete the semantic information at low resolution, and the latter aims to generate the detailed information at high resolution. Such a design can flexibly supervise the inpainting of high-frequency and low-frequency component separately to effectively mitigate the inter-frequency conflicts. Furthermore, our DRCN can provide a feasible solution for high resolution video inpainting by stacking the Revision Network in a pyramid manner. Comprehensive experiments demonstrate the effectiveness of our model in both quantitative and qualitative evaluations.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (U2336212), the Natural Science Foundation of Zhejiang Province (LDT23F02023F02) and the Earth System Big Data Platform of the School of Earth Sciences, Zhejiang University.