Delocate: Detection and Localization for Deepfake Videos with Randomly-Located Tampered Traces

1 Introduction

Deepfakes, AI-generated videos of people, pose serious threats to society [Chesney and Citron, 2019; Wang et al., 2022], emphasizing the need for reliable detection methods. By reliable, we believe the following three characteristics are necessary: First, the method should be robust to unseen forgery patterns (Fig. 3(a), [Rossler et al., 2019; Zi et al., 2020; Dolhansky et al., 2020; Li et al., 2020b]) with randomly located forgery traces (Fig. 3(b)), calling for crossdomain or cross-dataset evaluation. Second, a truly reliable method should convincingly explain the underlying reasons behind the model's decision by pointing to the manipulated part of a face. Unfortunately, we are not aware of works that satisfy all these criteria, so this paper develops a method that meets them.

∗Corresponding author.

Recently, reconstruction-prediction-based methods have achieved relatively high detection performance. These methods typically involve the encoder and decoder that encode the input data into a low-dimensional representation and subsequently decode the original inputs from that representation. For example, [Khalid and Woo, 2020] uses reconstruction scores to classify real and fake videos. Moreover, reconstructing and predicting future frame representations [Hu et al., 2022], forgery configurations [Chen et al., 2022a], pseudo training samples [Chen et al., 2022b], artifact representations [Dong et al., 2022], the whole faces [Cao et al., 2022; Shi et al., 2023], the masked relation [Yang et al., 2023], and mask regions [Chen et al., 2023] can boost the detection performance. As illustrated in the first row of Fig. 1, since these methods typically train the reconstruction model solely with real images without specifically targeting any fake patterns, they have relatively good cross-domain performance. However, these methods overlook the importance of locating the forgery areas. Meanwhile, Deepfake localization methods[He et al., 2021; Guo et al., 2023; Kong et al., 2022; Lai et al., 2023; Huang et al., 2022; Zhao et al., 2023; T, ˆant,aru et al., 2024] can locate forgery areas, but they learn the representation directly using real and fake videos, leading to a performance drop in detecting unseen types of fakes.

In this paper, we design a method that (1) can robust to unseen forgery patterns with randomly located forgery traces and (2) can locate the manipulated parts of faces. We name our method as Delocate, which, in essence, works as follows. The first stage, Recovering for Consistency Learning (Fig 2-top), pretrains a masked autoencoder using real faces only. Training on real faces ensures that the method does not overfit to any Deepfake patterns, enabling better generalization to unseen generation techniques, addressing point (1). Moreover, to detect tampered traces that appear randomly on a face, we design a unique masking strategy guided by facial parts. Subsequently, the masked autoencoder predicts the masked regions of interest (ROIs) based on the unmasked facial parts and interframes. This strengthens the understanding of relationships between facial parts and their temporal consistency, addressing point (1). Furthermore, the second stage, Localization for Discrepancy Learning (Fig 2-bottom), combines meta-learning with localization supervision to explicitly enhance cross-dataset generalization performance while simultaneously localizing the tampered regions of fake faces, addressing points (1) and (2).

Contributions. (1) We propose Delocate to learn representations guided by facial parts, enabling the detection of Deepfake videos in unknown domains.

(2) Unlike most detection methods that simply predict real or fake, Delocate can precisely localize tampered regions on faces. Learning to localize actually enhances the model's ability to detect fake videos.

(3) Extensive experiments on benchmark datasets, including FaceForensics++ (FF++) [Rossler et al., 2019], Celeb-DF (CDF) [Li et al., 2020b], DeeperForensics-1.0 (DFo) [Jiang et al., 2020], DFDC [Dolhansky et al., 2020] show that Delocate achieves effective performance under various metrics.

2 RelatedWork

Deepfake detection. Detection methods that focus on classifying real and fake videos can be broadly divided into two types: classification based on generalized methods (CLSGen) and classification based on reconstruction-prediction methods (CLS-Rec). The generalized methods contain methods based on implicit clues, explicit clues, and both implicit and explicit clues. Methods that explore implicit clues [Rossler et al., 2019; Li et al., 2020a; Sun et al., 2021; Sabir et al., 2019; Zhao et al., 2021a; Sun et al., 2022; Chen et al., 2021; Zhao et al., 2021b; Dong et al., 2023] use supervised learning to distinguish genuine and fake videos without explicitly incorporating clues to detect Deepfake videos, making it challenging to understand the underlying detection clues. Methods that employ explicit clues [Li et al., 2018;

Yang et al., 2019; Mittal et al., 2020; Luo et al., 2021; Nadimpalli and Rattani, 2022; Haliassos et al., 2021; Hu et al., 2022; Chai et al., 2020; Zheng et al., 2021; Guan et al., 2022; Hu et al., 2022; Shiohara and Yamasaki, 2022; Wang and Chow, 2023] have achieved more promising performance. Furthermore, Huang et al. [Huang et al., 2023] explore explicit and implicit embeddings for Deepfake detection. However, given the rapid advancement of Deepfake technology, various falsification traces can be left behind, rendering detection methods that rely on explicit features vulnerable to attack.

The reconstruction-prediction-based methods [Khalid and Woo, 2020; Hu et al., 2022; Chen et al., 2022a; Chen et al., 2022b; Dong et al., 2022; Cao et al., 2022; Yang et al., 2023; Chen et al., 2023; Shi et al., 2023] are explained in Sec. 1. Though these methods achieve promising detection performance, they do not focus on forgery localization.

Deepfake localization. Kindly note that while there is a vast array of research papers on image localization, our discussion here is specifically focused on papers related to Deepfake classification and localization (CLS & Localize). There are few works that focus on Deepfake localization. Kong et al. [Kong et al., 2022] use the noise map and semantic map to predict the forgery regions. Lai et al. [Lai et al., 2023] use the mask decoder to locate forgery areas and classify videos. Zhao et al. [Zhao et al., 2023] proposed RGB-Noise correlation to obtain the predicted manipulation regions. A recent paper [Shuai et al., 2023] proposes a two-stream network for improving detection performance. These methods push the Deepfake forensic one step further to forgery localization, but they struggle to classify cross-domain Deepfake videos.

3 Method

This section presents the details of Delocate for Deepfake video detection. Specifically, the proposed method is composed of two stages: (1) Recovering for Consistency Learning, and (2) Localization for Discrepancy Learning stage, as shown in Fig. 2. We demonstrate the logic design in Algorithm 1.

Notations. Let Aor , Aof , Arr , Arf , Amr , Amf , Aolr , Aolf , Aplr , Aplf be original real faces, original fake faces, recovered real faces, recovered fake faces, masked real faces, masked fake faces, original real face localization, original fake face localization, predicted real face localization, and predicted fake face localization.

3.1 Recovering for Consistency Learning

In this stage, we perform self-supervised learning of real faces to learn generic facial part consistency features. As a result, the unspecific inconsistencies of fake faces with randomly-located tampered traces are exposed. Furthermore, we finetune the model with real faces and fake faces.

Masking strategy tailored to learn the consistent face representation. We design a facial part masking strategy to ensure that the model can learn the consistencies of all facial parts. The designed facial part masking strategy is different from the frame masking strategy of VideoMAE [Tong et al., 2022].

First, as shown in Fig. 3(b), the tampered traces may only be sporadically present in one part and not related to other facial parts. Hence, we devise the masking strategy by considering Deepfake's domain knowledge. Specifically, we split the faces into different facial parts, i.e., eyes, cheek & nose, and lips, enabling the model to focus on both local and global consistencies among all facial parts. We choose region-specific masking strategy instead of a haphazard approach because random masks can fail to maintain the crucial global consistency among various facial regions. Neglecting such global facial part consistency could impede the model's ability to learn accurate facial part consistency features, making it challenging to distinguish real from fake videos based on reconstructed faces.

Second, the original masking strategy of VideoMAE [Tong et al., 2022], with a high masking ratio, would make it too challenging to restore the original appearance without any artifacts or distortions. If reconstruction artifacts occur, real faces will contain them, and fake faces will display both reconstructed artifacts and tampering artifacts. This makes it difficult to distinguish real videos from fake videos since both have artifacts. Therefore, we propose a masking strategy that focuses on ROIs and utilizes a relatively low masking ratio to enable the model to reconstruct the original faces more accurately.

The ROIs extraction is partially inspired by Facial Action Coding System (FACS) [Friesen and Ekman, 1978], which considers the action units of FACS as fundamental elements. Drawing from psychology studies [Friesen and Ekman, 1978; Wang et al., 2014; Wang et al., 2015; Li and Deng, 2020; Russell and Fernandez-Dols, 1997], it is well-known that real faces exhibit inherent consistency in these elements. Consequently, when we mask ROIs, it becomes more challenging to reconstruct these regions for fake faces compared to real faces.

Figure 3: The significance of the randomly-located traces. Different forgery patterns employ different shapes to alter the face area, rendering random tampered traces across different frames, which cannot be predicted based on the current frame, resulting in strong unpredictability. (I) Face2Face in FF++. (II) FSGAN in DFDC (III) DeepFakes in FF++. (IV) Deepfake in Celeb-DF.

We reference the action units of eyebrows, lower eyelid, nose root, cheeks, mouth corner, side of the chin, and chin to calculate bounding box coordinates according to facial key-

Algorithm 1: The algorithm process of Delocate.

1 Input:

• 2 The original real faces Aor , the original fake faces
• Aof . The original localization Aol ={Aolr ,Aolf }. The batch size bs = 8. The number of iterations num iter. The learning rate α.

3 Output:

• 4 Trained model in recovery θTed Rec , finetuning θTed Fin , and localization θTed Loc process.
• 1: while (Recovery process) θRec have not converged do
• 2: for i = 1 → num iterRec do
• 3: Arr = θRec(Aor )
• 4: θgrad Rec ←∇θRec ( 1b ∑i=bs i=1 Lmae mse (Aor , Arr ))
• 5: θT ed Rec ← θRec - αRec · AdamW (θRec , θgrad Rec )
• 6: end for
• 7: end while
• 8: while (Finetunig process) θFin have not converged do
• 9: for i = 1 → num iterF in do

• 11: bs )) 12: θT ed F in ← θF in - αFin · AdamW (θFin , θgrad Fin )

• 13: end for
• 14: end while
• 15: while (Localization process) θLoc have not converged do
• 16: for i = 1 → num iterLoc do

∇θLoc ( 1

∑i=bs

i

i=1 Lbcls (pAo Loc

, yAo Loc

i

))

bs

• 23: θMetagrad Loc ←∇θLoc ( 1 bs ∑i=bs i=1 Lb )

• 25: end for
• 26: end while

points. We discuss more about the masking strategy in the ablation study.

Network architecture. Our masked autoencoder is based on an asymmetric encoder-decoder architecture [He et al., 2022]. To consider temporal correlation, the vanilla Vision Transformers (ViT) and joint space-time attention [Tong et al., 2022] are adopted for recovering.

Recover masked faces. The masked patches of faces are dropped in the processing of the encoder, leaving the unmasked areas. In this way, the decoder predicts the missing facial part based on the unmasked areas. The reconstruction quality of masked patches is calculated with the MSE loss function Lmae mse .

If the model learns consistencies among facial parts, the loss between the reconstructed patches and the input patches should decrease. Our facial part masking strategy makes each part selected randomly, which enforces the model to learn the representation unspecific to any facial part. Furthermore, since this phase solely utilizes authentic videos and excludes any Deepfake content, it helps prevent the model from overfitting to particular Deepfake tampering patterns. In this way, the pretrained recovery model is obtained. Let Arr = REr × Aor , subject to 0 < REr < 1, where REr represents the recovery quality of Aor . A higher score of REr indicates better reconstruction quality.

Finetuning the recovery model. We discard the decoder and apply the encoder to uncorrupted Aor and Aof for finetuning. The finetuning process uses a cross-entropy loss for detection.

(2)

where pAo i is the predicted label of original faces, yAo i is the groud truth label of original faces. Since the recovery model learns the facial part consistency of real videos, the welltrained encoder can extract the consistency features of real videos. For fake videos, as shown in Fig. 3, they are generated by different forgery patterns and tampered with different areas, and the tampered traces can show up in random regions. Consequently, the tampered traces can not be predicted. If the masked areas contain tampered traces, the recovery process would be affected. If there are no tampered traces in the masked area, the tampered traces in unmasked areas can not be recovered. That is, regardless of whether the tampering traces are covered, the video with randomlylocated tampering traces will influence the recovery process, which makes the features extracted from the encoder different from those of the original videos.

3.2 Localization for Discrepancy Learning

In this stage, we leverage the well-trained recovery model from the first stage and map the recovery result to enlarge the discrepancy between real and fake videos.

Input data. We load the trained recovery model to obtain Aor , and input Aof , Aor , Arf , and Arr into the Localization stage. Let Arf = REf × Aof , where REf represents the recovery quality of Aof .

Data split strategy. To avoid over-fitting to specific Deepfake patterns, we use meta-learning [Jia et al., 2021] and randomly divide the training data into Meta-train set and Metatest set, where fake faces in Meta-train and Meta-test have different manipulated patterns.

Network architecture. We utilize the first convolutional layer of ResNet-18 [He et al., 2016]. Instead of directly utilizing ResNet-18, we employ the first three residual blocks of ResNet-18 and concat the outputs of these residual blocks. Second, the concatenated outputs are fed into three convolutional layers for face mapping. The dimensions of the mapped faces are 56×56×3. In this way, Amr and Amf can be represented as, Amr = MAr × Arr , subject to 0 < MAr < 1, Amf = MAr × Arf , subject to 0 < MAf < 1, where MAr and MAf represent the mapping quality of Arr and Arf . Third, the extracted mapping features are leveraged for classification purposes. Lastly, these features, alongside the original faces and localization labels, are fed into an encoderdecoder framework.

The encoder incorporates the SENet architecture [Hu et al., 2018], while the decoder adopts the UNet framework [Ronneberger et al., 2015]. To enhance the network's focus on pivotal regions, the SCSE Module [Wu et al., 2022] is integrated into the decoder. The classification outcomes derived from the mapping features are governed by the constraint Lbmse map , establishing a link with the encoder-decoder's localization component. This localization module is similarly regulated by Lbmse loc . Both the mapping and localization results collectively contribute to the overall classification constraint Lbcls . Instead of focusing on one task of classification and localization, our classification and localization results are mutually constrained and mutually promoted to facilitate us to complete multi-tasks of classification and localization.

Detection loss. To amplify the differences between Aor and Aof , we should satisfy:

Combine the analyses of the Amr , Amf , Aor , and Aof , Eq. (3) can be represented as:

Since the recovery model is trained on real data Aor and the randomly-located traces of Aof could influence the recovery process, we have Arr > Arf . Moreover, the recovery quality of Aof can be smaller than that of Aor . That is, 0 < REf < REr < 1. To satisfy Eq. (4), it is necessary to ensure that MAr ≫ MAf . Therefore, we minimize the MSE loss Lbmse map between the mapped faces and the reconstructed faces. Consequently, Lbmse map allows the Amr to be constrained by the consistency of Arr , while the Arf are constrained by the inconsistency. In this way, the model is able to recover Arr but fails to recover Arf , ensuring that MAr ≫MAf . These discrepancies enable the model to detect the failed reconstructed faces and better locate the tampered areas without misjudging the real faces.

For each pixel value in predicted localization masks Aplr pix , we normalize it and process it as follows.

The primary objective of localization is to minimize the MSE loss Lbmse loc between the ground truth localization mask and predicted localization mask.

Moreover, we also minimize the binary cross-entropy Lbcls between the video labels and the combined outputs of the mapping and localization features.

For each epoch, a sample batch is formed with the same number of fake videos and real videos to construct the binary detection task. To simulate unknown domain detection during training, the Meta-train phase performs training by sampling many detection tasks, and is validated by sampling many similar detection tasks from the Meta-test. Thereafter, the parameters of Meta-train phase can be updated. The goal of Meta-test phase is to enforce a classifier that performs well on Meta-train and can quickly generalize to the unseen domains of Meta-test, so as to improve the cross-domain detection performance.

The final loss function of the Localization stage is:

which combines the Meta-test loss of Lbcls , Lbmse map , and Lbmse loc and Meta-train loss of Lbcls , Lbmse map , and Lbmse loc to achieve joint optimization.

Detection results. We average the output of Recovering stage and Localization stage to get the final detection score.

4 Experiment

4.1 Experimental Setup

Datasets. Four public Deepfake video datasets, i.e., FF++ [Rossler et al., 2019], CDF [Li et al., 2020b], DFo [Jiang et al., 2020], DFDC [Dolhansky et al., 2020] are utilized to evaluate the proposed method and existing methods. FF++ is made up of 4 types manipulated algorithms: DeepFakes (DF) [DeepFakes, 2018], Face2Face (F2F) [Thies et al., 2018], FaceSwap (FS) [FaceSwap, 2018], NeuralTextures (NT) [Thies et al., 2019]. Moreover, 4000 videos are synthesized based on the 4 algorithms. These videos are widely used in various Deepfake detection scenarios. Celeb-DF contains 5639 videos that are generated by an improved DeepFakes algorithm [Li et al., 2020b]. The tampered traces in some inchoate datasets are relieved in Celeb-DF. DeeperForensics1.0 dataset is published for real-world Deepfake detection. DFDC is a large-scale Deepfake detection dataset published by Facebook.

Implementation details. In the Recovering stage, the masking ratio, batch size, patch size, and input size are set as 0.75, 8, 16, 224, respectively. The AdamW [Loshchilov and Hutter, 2017] optimizer with an initial learning rate 1.5 × 10-4 , momentum of 0.9 and a weight decay 0.05 is utilized to train the recovery model. The finetuning of the Recovering stage utilizes the AdamW optimizer with an initial learning rate 1 × 10-3 to detect videos. The SGD optimizer is used for optimizing the Localization stage with the initial learning rate 0.1, momentum of 0.9, and weight decay of 5 × 10-4 . We use FFmpege [Lei et al., 2013] to extract 30 frames from each video. The dlib [Sharma et al., 2016] is utilized to extract faces and detect 68 facial landmarks. We follow Kong et al. [Kong et al., 2022] to extract the ground truth of forgery localization.

Comparison methods. We compare Delocate with the CLS-Gen methods that are representative of implicit methods, explicit methods, and explicit and implicit combined methods, i.e., MultiAtt [Zhao et al., 2021a], LipForensics [Haliassos et al., 2021], Huang et al. [Huang et al., 2023]. We also compare Delocate with CLS-Rec methods, i.e., OST [Chen et al., 2022b], RECCE [Cao et al., 2022], MRL [Yang et al., 2023], and DisGRL [Shi et al., 2023]. Furthermore, we compare the CLS & Localize methods, i.e., Kong et al. [Kong et al., 2022], Zhao et al. [Zhao et al., 2023], Chao et al. [Shuai et al., 2023] .

4.2 Generalization to Unknown Domains

We enforce Delocate to learn unspecific features for Deepfake video detection with randomly-located tampered traces. The unknown domain detection is precisely the scenario where tampered traces are often randomly-located. To test the performance of Delocate, we simulate unknown domain Deepfake detection in multiple scenarios.

Comparisons of classification. First, we conduct experiments by training the model on FF++ with all 4 types of videos, but testing on other datasets, i.e., CDF, DFo, DFDC, and we use Area Under Curve (AUC) and Equal Error Rate (EER) to evaluate the classification performance. The enormous differences between the training domain and the testing domain make it challenging to improve unknown domain detection performance. Nonetheless, the results in Table 1 show that Delocate manages to improve the classification performance and achieve comparable localization performance at the same time. For example, Delocate improves the AUC on CDF from 86.2% (the localization method: Chao et al. [Shuai et al., 2023]) to 91.3%.

Second, to avoid performing experiments on a particular training mode, we change the training mode and conduct other unknown domain detection experiments. Specifically, we implement experiments by selecting one type of FF++ for training, but testing on other datasets, i.e., CDF, DFo, DFDC. Since there is only one type of video for training in experiments, we randomly split the training data into Metatrain and Meta-test with 7 : 3. Results in Table 2 illustrate that Delocate outperforms previous methods in many scenarios. Compared with classification methods, OST [Chen et al., 2022b] performs better than Delocate in 3 scenarios. Despite these results, it is worth noting that Delocate achieves better classification performance, especially with a 2.4% improvement over OST [Chen et al., 2022b] when training on FS and testing on CDF. We also observe that Delocate performs better AUC performance than that of lo- Train Test calization methods [Kong et al., 2022; Zhao et al., 2023; Shuai et al., 2023]. For instance, when training on DF and testing on CDF, Delocate achieves a 5.8% AUC improvement over Chao et al. [Shuai et al., 2023].

Figure 4: Comparisons of predicted forgery regions on CDF, DFo, and DFDC datasets when trained on 4 types of videos of FF++.

Comparisons of localization. We use Intersection over Union (IoU) and Pixel-wise Binary Classification Accuracy (PBCA) [Kong et al., 2022] to evaluate the localization performance. We train the model on FF++ and test it in other datasets. Table 3 shows that Chao et al. [Shuai et al., 2023] the best IoU results in testing DFDC. Delocate performs best results in other scenarios.

We also conduct forgery localization analyses for the CLS & Localize methods and show the results in Fig. 4. It shows that the localization area identified by Kong et al. [Kong et al., 2022], Zhao et al. [Zhao et al., 2023] and Chao et al. [Shuai et al., 2023] exhibits sporadic mismatches across various regions when compared to the ground truth. For the CDF, DFo, and DFDC datasets, Delocate aligns more closely with the ground truth region compared to the area localized by Zhao et al. [Zhao et al., 2023] and Chao et al. [Shuai et al., 2023]. It may be because Delocate focuses on unspecific features during the reconstruction stage, thereby revealing inconsistencies in the synthetic faces. In the localization stage, it maps the outcomes of the reconstruction, where the classification and localization results mutually influence and enhance each other. This process leads to the extraction of more generalized features, consequently improving the crossdomain performance.

4.3 Intra-dataset Detection Performance

To provide a comprehensive assessment of the proposed Delocate, we compare Delocate with the state-of-the-art methods in the scenario of intra-dataset detection. Specifically, we conduct experiments on 4 subsets of FF++ (C23). The training data and testing data of intra-dataset experiments are from the same subset of FF++. Table 4 shows that most methods perform well in intra-dataset detection. Chao et al. [Shuai et al., 2023] achieves the highest intra-dataset detection score while Delocate has a slight decrease of 0.2% in average accuracy compared. This drop may be due to the fact that the model improves the unknown domain performance while sacrificing a little bit of intra-domain performance to fit the unseen domain.

4.4 Ablation Study

We conduct ablation study experiments by training on FF++ but testing on CDF, DFo, and DFDC datasets.

Influence of the masking ratio. We trained models on the FF++ dataset with different masking ratios. Note that instead of defining the masking ratio as the ratio of masked area to the entire face, we define the masking ratio as the ratio of masked area to the corresponding ROIs facial parts. We choose not to use the original definition of mask ratio, which measures the ratio of the mask area to the entire face. Instead, we focus on specific regions of interest (ROIs) and divide the face into three parts. Then, we randomly mask only one part at a time. Our attention is directed towards the specific masked ROIs during the masking procedure, rather than considering the entire face as a whole.

In Table 5, we observe that Delocate scales well with the masking ratio of 75%. The performance gets a slight drop in the masking ratio of 55% and 65% indicating that low masking ratios may hinder learning robust features. When the mask rate is 85% and 95%, the detection performance is also degraded. That may be because that high masking ratio can raise the difficulty of reconstructing faces. If both real faces and fake faces are not reconstructed well, the distinction between them can be reduced. Therefore, we set the masking ratio as 75% in the experiments.

Influence of the masking strategy. We modify the masking strategies of MAE [He et al., 2022] to improve the generalization. To evaluate the effectiveness of the improved masking strategy, we compare the proposed masking strategy with masking strategies of MAE and VideoMAE. Furthermore, since the modified strategy randomly selects parts to mask, evaluating the effects of different masked parts is important. To analyze the effectiveness of the ROIs, we compare the proposed strategy with the masking strategy that does not focus on ROIs. We trained models on the FF++ dataset with different masking strategies.

The results of 1st , 2nd , and 7th lines in Table 6 demonstrate that modifying the masking strategies of MAE [He et al., 2022] and VideoMAE [Tong et al., 2022] can improve the detection performance. The results in the 3rd , 4th and 5th lines, which represent methods that mask eye areas, cheek and nose areas, and lip areas, respectively, show a performance degradation compared to the proposed strategy. That is, randommasking a part of all facial parts is more conducive to extracting robust features than masking a certain part only. Moreover, the results of the 6th line and 7th lines show that the proposed masking strategy that focuses on ROIs achieves better performance than the masking strategy without ROIs. The reason is that the model can better capture the differences between real and fake videos by masking patches in these ROIs, as fake videos typically lack consistency. Therefore, the proposed masking strategy is effective in detecting Deepfake videos.

Influence of MAE and VideoMAE. We compare the detection performance of the Delocate with the original MAE and VideoMAE methods for Deepfake detection. The results are shown in the 1st and 2nd line of Table 7. The detection performance of the original MAE and VideoMAE is lower than that of Delocate, demonstrating the effectiveness of the modifications in Delocate.

Influence of Recovering stage and Localization stage. To validate the performance of each stage, we compare the performance of a single stage with that of both stages combined. The results are shown in the 3rd and 4th lines of Table 7. We can see that removing either the Recovering stage or the Localization stage degraded the detection performance, as each stage plays a crucial role in Deepfake detection. Combining both stages improves the performance by magnifying the distinction between real and fake videos.

Influence of Meta-learning, Mapping, and EncoderDecoder. We remove the meta-learning, mapping, and Encoder-Decoder module to carry out experiments, respectively, and the results are shown in the 5th , 6th , 7th line of Table 7. Compared with results of 11th line, the method without meta-learning, mapping, and Encoder-Decoder module achieves worse results than the proposed Delocate with these modules. The meta-learning approach simulates crossdomain detection in the training phase, improving detection performance. The mapping module can reveal the inconsistencies by developing the autoencoder of the Recovering stage, which facilitates the Encoder-Decoder module to locate the forgery regions. The Encoder-Decoder module achieves the forgery localization, providing a guidance for the classification results.

5 Conclusion

This paper focuses on the detection and localization of Deepfakes, particularly in identifying Deepfake videos with randomly-located tampered traces. By focusing equally on all facial parts rather than relying on specific facial parts, our two-stage model can learn unspecific facial consistencies and general representations. In the Recovering stage, the model is trained to recover faces from partially masked ROIs on the face, which facilitates the model in learning the facial part consistencies of real videos. In the Localization stage, the model enforces a mapping and an encoder-decoder strategy to expose the forgery areas in synthetic ones. Extensive experiments illustrate the generalizability of Delocate in detection and localization.

Acknowledgments

This work is supported by National Natural Science Foundation of China (Grant Nos. U22A2030, U20A20174), National Key R& D Program of China (Grant No. 2022YFB3103500), Hunan Provincial Funds for Distinguished Young Scholars (Grant No. 2024JJ2025). Juan Hu thanks the support of the Ministry of Education, Singapore, under its MOE AcRF TIER 3 (Grant MOE-MOET32022-0001). Satoshi Tsutsui is supported by the NTU-PKU Joint Research Institute (a collaboration between the Nanyang Technological University and Peking University that is sponsored by a donation from the Ng Teng Fong Charitable Foundation). Mike Zheng Shou does not receive any funding for this work.