Beyond Alignment: Blind Video Face Restoration via Parsing-Guided Temporal-Coherent Transformer

1 Introduction

Deep learning technology has developed rapidly recently. Researchers have expanded deep learning technology into different fields such as target recognition [Zhang et al., 2024], natural language processing[Chen et al., 2024], and video enhancement[Xu et al., 2022b; He et al., 2022; Xu et al., 2023; Li et al., 2020; Xu et al., 2022a], video restoration[Wang et al., 2024a; Wang et al., 2024c; Wang et al., 2024b], etc. The human face, traditionally considered a focal point in the video, presents an intriguing opportunity - leveraging limited computational resources to enhance areas of interest (face area). Currently, researchers have pivoted towards real-world face restoration, conceptualizing the task as blind face restoration to restore high-quality faces from real face suffering from multiple degradations.

∗Corresponding Author

The amalgamation of various degradations (including noise, blur, and coding artifacts) poses significant challenges in achieving high-quality face restoration. Earlier methodologies directly incorporated face geometry [Chen et al., 2021; Chen et al., 2018] as a prior into the model. Despite their ability to perceive global structure, as depicted in Fig.1 (b), these methods fall short in generating high-quality texture details. Techniques [Yang et al., 2021; Wang et al., 2021] grounded in generative priors have shown potential in producing highquality texture details. However, generative models suffer from fixed spatial bias, leading to various erroneous illusions and artifacts when processing low-quality faces with complex poses, as shown in Fig.1 (c).

Furthermore, previous works [Wang et al., 2021; Zhou et al., 2022; Gu et al., 2022] have primarily focused on imagebased face restoration. However, in real-world scenarios, face restoration is not limited to images alone; it is also crucial in videos. In the context of video face restoration, there are unique challenges and unresolved issues that need to be addressed. One such challenge is the variability in face pose across frames. When using existing prior generation methods, the workflow involves 1. Pre-aligning low-quality face images to standard poses. 2. Face restoration. 3. Reverse warping to the input face, as shown in the Fig.2 shown in (a). Unfortunately, this approach introduces additional degradation problems during frame-by-frame processing. These problems arise from errors in low-quality face keypoint detection, which in turn affect the accuracy of warping. Consequently, facial structure generation errors occur, and the accumulation of such errors can amplify inter-frame differences, leading to video jitter. These issues remain unsolved by prior methods relying on face-generation priors.

To tackle the aforementioned challenges, we propose a novel video face restoration approach to overcome the limitations of previous techniques by addressing the issues associated with changing face poses, error propagation, and video jitter. In contrast to previous image-based face restoration approaches, our proposed method aims to restore high-quality face texture details consistently across frames, while avoiding the generation of undesirable artifacts, hallucinations, and frame jitters. To achieve video face restoration, we introduce a temporal-spatial interaction model that facilitates the enhancement of facial information. Moreover, we leverage a face parse-guided module to restore faces from videos without the need for pre-alignment.

Our approach introduces a Temporal-Spatial VQGAN (TSVQGAN) based on the Swin3D [Liu et al., 2022] Transformer, enabling the extraction of precise timing information from video frames to obtain an accurate representation of video face feature. The trained codebook and decoder encompass a rich set of high-quality video face priors.

Furthermore, we propose a codebook prediction mechanism guided by temporal context parsing. Specifically, we fuse the low-quality (LQ) features from consecutive frames and input them into a 3D Transformer, referred to as Temporal Parsing-guide Code predictor (TPCP), to enhance the accuracy of codebook predictions using contextual information from the video. Notably, we incorporate face parsing features as position encoding inputs to the 3D Transformer, addressing the issue of artifact hallucinations caused by fixed spatial biases in global face representations. By adopting this approach, we mitigate the generation of unnatural artifacts, even when dealing with complex face poses. This novel technique significantly expands the applicability of generative prior-based methods in video face restoration.

Finally, we introduce a novel module called the Temporal Fidelity Regulator (TFR), which plays a crucial role in integrating multi-scale low-quality features with the decoder stage. To ensure the temporal consistency of video faces, our TFR extracts temporally consistent face features, which are then utilized to modulate the HQ features. This enables the decoder to achieve coherent recovery of high-fidelity video faces, promoting visual consistency throughout the sequences.

Contribution: 1) Our work is pioneering as the first end-to-end method for video face restoration; 2) We introduce a tailored temporal-spatial interactive network structure that effectively utilizes multi-frame face information, yielding superior restoration results; 3) Furthermore, we present a novel approach for high-quality face restoration without pre-alignment, significantly broadening the scenarios of face restoration applications; 4) Our proposed scheme showcases state-of-the-art performance on video face restoration tasks, as validated through extensive experiments.

2 RelatedWork

2.1 Blind Image Face Restoration

Current methods based on generator priors can recover highquality texture details. For example, [Gu et al., 2020; Menon et al., 2020] use the face generation prior from StyleGAN for blind face restoration, but the degraded LQ face is illposed when converting to HQ face, and the fidelity of these methods is difficult to be guaranteed. In order to solve the problem of too low fidelity, GLEAN[Chan et al., 2021a] and GFPGAN[Wang et al., 2021] directly introduce the generator prior into the decoder, which can improve the fidelity to a certain extent. Despite the increased fidelity, when the input is heavily corrupted, artifact artifacts can be produced.

Another method, VQFR[Gu et al., 2022] and CodeFormer[Zhou et al., 2022], directly uses the codebook dictionary information in the VQGAN series of methods to model face restoration as a codebook selection problem, and improves the fidelity through encoder information fusion. These methods all use the face prior information of the generator, but highly aligned face data is required to train the generator, and only the aligned face can be restored during use. If the input face is not pre-aligned, the resulting face image will have unacceptable artifacts.

In the video face, there will be structural changes in the faces of adjacent frames, and this structural change is more prominent in low-quality faces. The difference between two adjacent frames of faces is detected by key points, prealigned, face restoration, and objection After aligning the four passes were zoomed in, which caused major problems with discontinuity between frames.

Therefore, although these methods based on generative priors have achieved good performance in image face restoration tasks, they cannot effectively restore low-quality faces in videos.

2.2 Video Restoration

Video restoration is to reduce or eliminate the quality loss of video in the process of lossy compression, so as to generate a high-quality image video that is closer to lossless. MFQE[Guan et al., 2021; Yang et al., 2018] adds multi-scale information extraction and dense connection mapping structure to the quality enhancement network to improve the enhancement effect. In EDVR[Wang et al., 2019a], time information is gathered through space-time deformable convolution. According to different image frames, the spacetime sampling position of convolution can be deformed adaptively, expanding the receptive field to a certain extent, and obtaining the most relevant context. information, exclude noisy content and improve the quality of image frames. The BasicVSR[Chan et al., 2021b; Chan et al., 2022] series methods restore high-quality videos through feature propagation and alignment.

3 Methodology

The primary objective of our research is to achieve superior restoration of the face in video. However, traditional imagelevel face restoration methods encounter several limitations when applied to videos. These limitations include the inability to leverage temporal information to enhance restoration quality, the lack of continuity in restored faces, and the introduction of additional degradation due to frame-based prealignment.

To address these challenges, we propose a comprehensive framework. The workflow of the framework is illustrated in Fig. 3. It comprises two main parts. The first part is a spatio-temporal quantized autoencoder (TS-VQGAN) based on Swin3D, as depicted in Fig. 3 (a). This component is trained on high-quality face videos and captures high-quality texture priors. The second part is the temporal parse-guided codebook selection face restoration network, as shown in Fig. 3 (b). This component utilizes temporal parse information to guide the codebook selection process, thereby facilitating accurate restoration of facial features.

TS-VQGAN allows us to extract facial information from videos and generate a quantized codebook representation of high-quality facial video sequences. After obtaining a codebook with high-quality video face priors, the face restoration problem is modeled as a codebook prediction problem. Subsequently, our focus shifts towards designing the corresponding codebook predictor. To accomplish this, we employ a 3D transformer as the codebook selector. This codebook selector utilizes the facial features extracted from the video sequence to accurately select the appropriate high-quality codebook. The selected codebook is then fed into the decoder Dh to recover a high-quality face video sequence. To effectively handle complex facial poses in the video, which may lead to erroneous codebook selection by the codebook selector, we introduce the incorporation of face parse features as conditional position codes. This ensures accurate codebook selection even in the presence of atypical facial poses, thereby avoiding codebook selection errors caused by fixed face space priors. Finally, we introduce a temporal feature regulator to enhance the temporal continuity of the recovered video face.

3.1 Temporal-Spatial VQGAN

To obtain a high-quality representation of video faces, we adopt a pre-training strategy using the temporal-spatial quantized autoencoder (TS-VQGAN). This approach aims to learn a codebook that encompasses spatio-temporal context, thereby enhancing the network's expressiveness and robustness against degradation.

As depicted in Fig. 3 (a), the input consists of a sequence of high-quality face video frames denoted as xh ∈ RT×H×W×3 . These frames are fed into the spatio-temporal interactive encoder Eh, which generates a potential feature representation zh ∈ RT×H×W×D . Subsequently, the nearest neighbor matching technique is used to find the closest codebook vector Ck in codebook C for each vector in zh, and the index S of the codebook is obtained. Get the quantized representation latent zq through S. For the formal expression, please refer to Equation 1. Finally, the quantized representation zq is input into the decoder Dh, yielding the reconstructed video sequence xr.

θEh , θDh are the parameters of the encoder and decoder, respectively. Here, our encoder Eh and decoder Dh are comprised of 8 residual blocks and 4 resizing layers for downsampling and upsampling. To ensure accurate face reconstruction, we set the number of codebooks N to 1024, which provides ample capacity. Furthermore, we set the code dimension d to 512. To extract video sequence features effectively, we integrate Swin3D Transformer layers during the 2nd, 3rd, and 4th downsampling and upsampling stages.

3.2 Temporal Parse-guided Codebook Predictor

Utilizing the trained TS-VQGAN's codebook and decoder to generate high-quality faces, we formulate the face restoration problem as a controllable face generation process.

During the face restoration process, we input the lowquality face xl into the low-quality encoder El, generating a low-quality latent representation zl. Our goal is to use the function T to predict the codebook index ˆ S based on zl, and obtain the corresponding high-quality features ˆ zq from Codebook C according to the index ˆ S. By inputting ˆ zq into the decoder Dh, we obtain the restored output xo.

This process can be framed as a maximum likelihood estimation, where we maximize the conditional likelihood of the predicted codebook ˆ zq with respect to the low-quality input image xl, enabling optimal parameter estimation for El and T to achieve accurate code prediction. Due to the loss of face texture details in the low-quality input, the accuracy of codebook selection using the nearest neighbor method is limited. To address this, previous methods employ a simple Transformer to predict a single frame codebook.

The process can be formally defined as follows.

where θEl , θT , and θDh represent the parameters of the encoder, Transformer, and decoder, respectively. By optimizing these parameters, we aim to enhance the face restoration performance and generate visually appealing results.

However, for video face restoration, straightforward using the Transformer prediction codebook cannot effectively utilize temporal context face information, and artifacts are prone to appear on unaligned low-quality video faces (such as the Fig. 1).

Based on this, we design a temporal parse-guided NonAutoregressive Transformer for codebook prediction. This design pattern can predict the codebook more accurately and restore high-quality face videos. Specifically, we input the video sequence into the face parsing model Ep, and embed the obtained intermediate feature xp into the Transformer as a relative position code. The network structure of the face parsing model Ep is consistent with [Yu et al., 2021], and we use the CelebAMask-HQ [Lee et al., 2020] dataset to train Ep. At this time, the process of our temporal parse-guided codebook predictor can be defined in the Eq. (3).

Compared with previous codebook predictior, our proposed codebook predictor TPCP has two advantages: 1) It can utilize multi-frame information for more accurate codebook prediction. 2) Face parsing is used as position coding to realize codebook selection for face parsing perception.

3.3 Temporal Fidelity Regulator

Although it is already possible to obtain high-quality face videos using codebook prediction, the fidelity and coherence of faces are also important when recovering faces from videos. To this end, this paper designs a Temporal Fidelity Regulator (TFR) that uses the encoder feature xe to modulate the decoder feature xd. Specifically, our TFR is divided into two processes. The first is time feature compression, which performs dimensionality reduction (squeeze) on the input xe and xd. Followed by time feature aggregation, temporal fusion is performed on xe and xd along the time dimension to get xTe and xTd through convolution. Next, concatenate the interacted xTe , xTd with xe and xd to get xconcat. After two convolutions respectively, affine parameters α and β are obtained, and α and β are used to modulate the multi-level decoder feature xd to obtain the output feature ˆ xd. The TFR will be adopted in each decoder level to enhance the final temporal refined output xo. The detail of TFR is shown in Eq.4.

TFR performs feature aggregation in the time dimension during the modulation process, thereby improving the consistency of restored video faces and reducing jitter artifacts.

3.4 Training Objectives

Based on our sophisticated architecture, the training of the model is divided into four stages.

Stage I. In the training process of the quantized autoencoder, three loss functions (L1, perceptual loss, adversarial loss) are used to jointly optimize TS-VQGAN, and the specific form is shown in the Eq. 5, Φ is the VGG19 network.

The weights of the three losses are set to 1.0, 1.0, and 0.75, respectively.

Stage II. The goal of the second stage is to fine-tune the encoder El and train the parsing-guided codebook predictor (TPCP). Only code-level loss is needed at this stage. The codebook prediction loss comprises two distinct components. The primary component, denoted as Lcode, involves computing the cross-entropy loss between the predicted and actual codebook indices. The secondary component pertains to the codebook's feature loss, where we measure the L2 loss between the encoder's output feature zl and the ground truth codebook feature zq. Where s is the codebook selected by the HQ video input to the HQ encoder, and ˆ s is the codebook predicted by the LQ video input to the LQ encoder.

Stage III. The third stage is fidelity training for PGTFormer. To optimize the TFR module and fine-tune the encoder El, we use code loss Lcode, pixel reconstruction loss L1, perceptual loss Lper, adversarial loss Ladv, and set the weights to 0.5, 1.0, 1.0, 1.0, and 0.75, respectively.

Stage IV. The fourth stage is complex pose face finetuning training, here we use unaligned face video sequences to fine-tune the model, and the loss is consistent with the third stage. We added LMSRL[Dai et al., 2022] as a loss to increase video coherence, defined in Eq. (7).

4 Experiments

4.1 Datasets andMetrics

Datasets. We train the model on the VFHQ [Xie et al., 2022] dataset, which contains 16,000 video sequences, the resolution of the video is 512 × 512. To make training pairs, we follow [Xie et al., 2022] synthesize LQ videos x from HQ video counterparts y with the following degradation model (in Eq.8):

(8)

where the HQ video y is first convolved with a Gaussian kernel kσ), and then downsampled to scale r. After that, additive Gaussian noise nδ is added to the video, and then the video coding degradation is implemented by FFMPEG(x264[Merritt and Vanam, 2006]) with a constant rate factor (CRF). Finally, the LQ video is resized to 512×512. The procedure for generating complex video bind face degradation is provided by VFHQ.

Metrics. We first use PSNR, SSIM, LPIPS [Zhang et al., 2018] to evaluate the quality of generated faces. In addition, we also evaluate identity preservation by the cosine similarity of the features of ArcFace [Deng et al., 2021; Sun et al., 2020], denote as Deg. To better measure the fidelity of accurate facial expressions and expressions, we further employ Landmark Distance (LMD) as a fidelity metric. Specifically, we use the method proposed by [Wang et al., 2019b] to obtain 98 landmarks from recovered faces and real faces. We then compute the L2 distance for each landmark and average the distances into the final score for the LMD metric. Since we are a video face restoration task, it is necessary to evaluate the continuity of the face, we calculate the L2 distance between the real face landmark displacement and the recovered face landmark displacement, which can evaluate the continuity of the face components in the video. We further calculate the peak signal-to-noise ratio of MSRL [Dai et al., 2022] to evaluate the continuity of the restored face video.

(7)

4.2 Comparisons with State-of-the-Art Methods

Quantitative Results

We compare the proposed PGTFormer with state-of-the-art

methods, including HiFaceGAN, GFPGAN, VQFR, Code-

Former. We also compare with the current state-of-the-art

method BasicVSR++ for video restoration. To verify the ef-

fectiveness of different methods, we train and test models on

aligned and unaligned data.

We show the quantitative comparison of VFHQ-Test in Table 1(tests on aligned and unaligned). In terms of image quality metrics PSNR, SSIM, LPIPS, our method achieves the best scores than existing methods. Furthermore, it faithfully preserves the identity, which is reflected in the highest Deg score. We calculated the landmark distance LMD between the restored face and the ground truth, which can evaluate the structural difference of the face, and our method achieved best performance on the LMD index. We evaluate the temporal continuity by tracking the offset error of the face landmarks between frames, which can evaluate the temporal continuity of different methods. In Table 1, our method achieves the best performance on TLME and MSRL metrics, proving that our method has higher video continuity and can restore more natural continuous face video.

In addition, we calculated the FVD metric, which can evaluate the subjective quality of the recovered face video. Table

Table 4: Ablation for Parsing-Guided Codebook Predictor(Stage II). Baseline, Face parsing guide, and 3D codebook predictor are abbreviated as B, M0, and M1 respectively.

2 shows the FVD score. Compared with the past state-of-theart methods, our proposed method has significant advantages in the FVD metric.

Qualitative Results

Furthermore, we give a qualitative comparison in Fig. 4. Other comparison methods did not yield pleasant recovery results. Especially in the restoration quality of eyes and teeth, other methods are prone to disturbing hallucinatory artifacts, which our method can overcome. Since our face prior comes from spatio-temporal interaction features, it can better characterize face components. In areas prone to artifacts such as eyes and teeth, our method can recover natural face components. The advantages of our method in the quality of the eye region are demonstrated in Fig.5.

4.3 Ablation Studies

Importance of TS-VQGAN

We do ablation experiments to study the superiority of TSVQGAN. As shown in Fig.6. The left side is the ordinary VQGAN reconstruction result, and the right side is the reconstruction result of our proposed TS-VQGAN. We can see that our Temporal has higher fidelity and texture details. This is because in the process of expressing facial features, spatiotemporal feature interaction can further complement the missing details of a single frame, and implicitly align features through Swin3D. Quantified ablation results such as Table 3, compared with ordinary VQGAN, we have advantages in evaluation indicators such as PSNR, SSIM, and Lpips. This proves that our TS-VQGAN can better reconstruct video faces.

Superiority of Temporal Parse-guided Codebook Predictor (TPCP)

In order to ablate the effectiveness of the proposed analytically guided 3D codebook predictor, we add analytically guided and 3D codebook predictors on the basis of Baseline respectively. From the Table 4 and Table 6 we can see that with the introduction of face parse guidance and temporal codebook predictor, indicators such as PSNR have been improved. In particular, we have further measured the accuracy of codebook selection, and we can see that the introduction of the two components has improved Top 1 Acc and Top 5 Acc.

Effectiveness of Temporal Fidelity Regulator (TFR)

We ablate temporal fidelity regulator (TFR), and show the results of ablation experiments in Table 5. After adding TFR, Landmark Continuity Error (TLME) is reduced and Multiscale Continuity (MSRL) is improved. It is proved that the proposed TFR can utilize the temporal features to enhance the continuity of the restored video face.

Inference Time

We analyze the inference speed of different methods. The previous method applied to the video face restoration task requires additional pre-alignment operations. We calculated the total inference speed (keypoint detection, warp, restoration, inverse warp), in the table The time overhead of different methods is shown in Table 2. Compared with previous face restoration method our method achieves the best performance on inference time.

5 Conclusion

This paper tackles the fundamental challenge of blind videobased face restoration, enabling high-quality results without pre-alignment. To enhance video face representation, we propose TS-VQGAN, which achieves higher-quality face codebook representation. Our design includes a temporal parseguide codebook predictor that accurately selects the codebook from the continuous video sequence representation. Additionally, we leverage the temporal fidelity regulator to enhance the overall continuity of video face restoration. Notably, our approach is the first to specifically target video face restoration, and our experimental results showcase its superior effectiveness.

Acknowledgments

This Research is Supported by the National Key Research and Development Program from the Ministry of Science and Technology of the PRC (No.2021ZD0110600), Sichuan Science and Technology Program (No.2022ZYD0116), Sichuan Provincial M. C. Integration Office Program, and IEDA Laboratory of SWUST, Grant CEIEC-2022-ZM02-0247, the 2023 Research Project of Shaanxi Provincial Department of Transportation, No. 23-58X.