Learning Real Facial Concepts for Independent Deepfake Detection

1 Introduction

With the rapid development of generative algorithms, the barrier to the synthesis of facial forgeries has decreased significantly, fostering online fraud, opinion manipulation, and pornographic content dissemination. To mitigate the abuse of face forgeries, a considerable number of effective deepfake detection methods [Xia et al., 2024b; Sun et al., 2024] have been proposed, and they have achieved significant success when training and testing on the same dataset, i.e., evaluating under the within-dataset setting. Most of these methods treat deepfake detection as a binary classification problem, aiming to capture various artifacts in facial images, such as inter-frame correlation [Qiao et al., 2024] and style latent flow [Choi et al., 2024] to distinguish between real and fake.

∗Corresponding author.

However, when these models are applied to unseen datasets, their performance often degrades significantly. This issue, i.e., the lack of generalization, has garnered widespread attention in recent years. The solutions for it can be broadly categorized into two approaches: 1) Expanding the scale or diversity of the training dataset to include more complex forgery traces; 2) Optimizing the model architecture to extract general forgery artifacts that are consistent across various deepfake techniques. Based on the aforementioned approaches, many studies achieve remarkable detection performance. However, as illustrated in Figure 1(a), the existing models counterintuitively misclassify real instances as fake ones, rather than maintaining a balanced misclassification rate between real and fake instances (red circles and green triangles represent real and fake samples, respectively).

We attribute this phenomenon to two key factors: 1) Data. As shown in Figure 1(b), real facial features exhibit a clustered distribution, while the distribution of fake ones is relatively uniform. This will cause the model to overfit these homogeneous real samples and limit the feature distribution of the real class. Meanwhile, in Figure 1(c), we find test samples that shift beyond the original training distribution are likely to be misclassified. This indicates that the narrow distribution of the real class makes it difficult to cover the real samples in the test set and classify the extra-distributed real samples accurately. 2) Classifier. Deepfake detectors rely on basic binary classifiers to differentiate between real and fake instances. However, because of the difficulty in obtaining the concept of real faces, the classifier would overly rely on forgery traces to make judgments. In other words, the decision-making process has a built-in priority: the model could first determine whether an image is fake according to forgery artifacts, and only if it is not classified as fake will it be recognized as real. Moreover, due to the limited diversity of the training dataset, forgery-irrelevant patterns, such as noise and blur, will also be entangled with forgery artifacts. When these so-called 'mis-artifacts' appear in real samples during the testing phase, they can easily lead to an incorrect judgment. Given the insufficient exploration of real samples and the overlook of real facial concepts, a natural question motivates us - Could we improve the generalization by learning a more comprehensive concept for real faces and inducing the model to make judgments based on both the real facial concepts and the presence of forgery artifacts?

We address the aforementioned issues by proposing the Real Facial Concepts based Independent Deepfake Detection (RealID) approach. Our approach focuses on exploring and leveraging real facial concepts, enabling the model to make judgments based on both real facial concepts and forgery artifacts. Specifically, we design two novel modules: 1) Real Concept Capture Module (RealC2 ), which employs a Multi-Real Memory mechanism to store various real prototypes. It operates over the entire training set, learning a more comprehensive real facial concept with specially designed Prototype Distinction Loss and Prototype Diversity Loss. In this way, the model can focus more on the differences of the real samples and generate a more robust distribution for the real class. 2) Independent Dual-Decision Classification strategy (IDC). We reformulate the binary classification strategy to a novel independent dual-decision strategy. By adding a regularization term and two auxiliary classes, the probability of belonging to the incorrect classes can further decrease after being optimized with the cross-entropy loss. This means that our IDC can adaptively search for an alternative optimization path, i.e., utilizing the real facial concepts learned from RealC2 , and mitigate the misguidance of the overfitting misartifacts. Using previously overlooked real facial concepts, our method enables the model to make robust decisions from both real and fake perspectives, thereby enhancing its generalization capability in target domains. We conduct extensive experiments on five widely used datasets and the experimental results demonstrate a 1.74% average accuracy improvement compared to existing state-of-the-art methods.

In summary, our contributions are three-fold:

• We attribute the issue of limited model generalization to the inadequacy in learning comprehensive real facial concepts. Our observations indicate that existing detectors tend to overly rely on forgery artifacts, which often results in the misclassification of real instances as fake.
• We propose the Real Facial Concepts based Independent Deepfake Detection approach, which combines two special modules to facilitate the learning of real facial concepts and enable the classifier to independently assess the probabilities of belonging to real and fake classes.
• Extensive experiments are conducted to demonstrate the

effectiveness of our approach which achieves state-ofthe-art generalization performance on several datasets.

2 RelatedWork

2.1 Deepfake Generation

The rapid development of portrait synthesis has propelled deepfake technology [Xu et al., 2022b] into a prominent research area. Autoencoders [Kingma and Welling, 2014] form the foundational architecture for many early deepfake approaches [Thies et al., 2016; Suwajanakorn et al., 2017]. Typically, these methods involve training two separate models on a reconstruction task and then swapping their decoders to change the identities of source faces. While these techniques can produce realistic face-swapped images, they are limited to one-to-one face swapping. To enable more versatile synthesis, GANs [Goodfellow et al., 2014] have become increasingly popular due to their ability to generate arbitrary high-quality facial images. For example, StyleGAN [Karras et al., 2019] allows for the manipulation of high-level facial attributes by using a progressively growing structure combined with adaptive instance normalization. IPGAN [Bao et al., 2018] disentangles the identity and attributes of the source and target faces for synthesizing new faces. Recently, identity-relevant features have been integrated into deepfake generation to enhance identity consistency. Xu et al. [Xu et al., 2022a] refined identity consistency by augmenting both local and global identity-relevant features through cross-scale semantic interaction modeling, achieving more coherent face swapping that maintains identity integrity.

2.2 Deepfake Detection

Deepfake detection [Wang et al., 2024; Hong et al., 2024; Yan et al., 2024; Xia et al., 2024a; Guan et al., 2024] is generally cast as a binary classification task. Preliminary efforts often endeavor to detect the specific manipulation traces [Jia et al., 2022]. Masi et al. [Masi et al., 2020] utilized optical and frequency artifacts separately. SSTNet [Wu et al., 2020] detects edited faces through spatial, steganalysis, and temporal features. These models have shown certain improvements on some datasets. However, they often encounter inferior performance when applied to different data distributions. To improve generalization [Tan et al., 2024; Li et al., 2023], one manner is to construct datasets that encompass a wider range of forgery methods [Khalid et al., 2021; Le et al., 2021; Cheng et al., 2024a]. For instance, the KoDF dataset [Kwon et al., 2021] comprises over 200,000 videos generated by six distinct algorithms, while DF-Platter [Narayan et al., 2023] involves over 130,000 multi-face forgery videos. These highquality datasets encompass diverse data sources. However, it is crucial to note that these datasets often place greater emphasis on the fake side, while the distribution of real images tends to be more homogeneous. This imbalance makes it easier for a model to overfit on real features, which in turn leads to misclassification. Another approach is to learn more generalized features of forgeries [Wang and Deng, 2021; Luo et al., 2021; Zhao et al., 2021; Li et al., 2020a; Nirkin et al., 2022]. For instance, RealForensics [Haliassos et al., 2022] exploits the visual and auditory [Cheng et al., Diversity

Figure 2: Overall architecture of the RealID framework. RealID consists of two main modules: (i) the Real Concept Capture(RealC2 ) module, which uses a multi-real memory mechanism to learn a more comprehensive real facial concept, and (ii) the Independent Dual-Decision(IDC) module, which leverages regularization terms to independently optimize the decision-making process for different categories.

2023] correspondence in real videos. Chen et al. [Chen et al., 2022] specified the blending regions and facial attributes to enrich the deepfake dataset with more manipulation types. Shiohara et al. [Shiohara and Yamasaki, 2022] introduce a self-blended framework to capture boundary-fusion features. These methods have achieved notable success in capturing forgery artifacts, reaching performance plateaus on multiple datasets. However, they tend to lack sufficient focus on real instances, leading to an imbalanced decision-making process.

3 Methodology

As illustrated in Figure 2, our RealID framework improves generalization through the combination of two novel modules: the Real Concept Capture (RealC2 ) module and the Independent Dual-Decision (IDC) module. Specifically, RealC2 uses a Multi-Real memory mechanism [Park et al., 2020] to maintain several prototypes of real faces, enabling the model to extract more comprehensive features from real samples and preventing the model from overfitting to the homogeneous training sets. IDC modifies the decision logic of the naive deepfake classifier, which makes decisions based solely on the presence or absence of forgery artifacts. By incorporating an additional regularization term and auxiliary classes, it can independently reduce the probability of belonging to the false classes through different optimization paths. This means IDC mitigates the influence of mis-artifacts and improves the generalization of the deepfake detector.

During the training process, we specifically construct mini-batches to ensure the inclusion of both N real samples Xreal (with the labels of Y = 0) and N fake samples Xfake (with the labels of Y = 1). Then, we obtain the feature maps M∈ R2N×H×W via a feature extractor E(·) and transform them to feature vectors F ∈ R2N×C by a series of pooling and flattening operations. This feature extraction process can be formalized as follows:

3.1 Real Concept Capture Module

As shown in Figure 1(b), the homogeneous real samples in the training set may mislead the detection models into overfitting to a specific cluster while neglecting the concept of real faces. This limitation makes it difficult to address the distribution shift in the testing set. To overcome this issue, we design the RealC2 module to extract more comprehensive real facial features. In particular, instead of treating the real class as a single homogeneous entity, our model subdivides real facial features with the assistance of diverse real prototypes. In this way, the model can distinguish the fine-grained differences between real instances and acquire a more comprehensive understanding of real faces.

Prototype Initialization. Before training begins, we randomly initialize K vectors MR = {m1,m2, · · · ,mK} to construct the Multi-Real Memory, and each mi ∈ MR can be considered as a real facial prototype. During the training, these prototypes are designed to record real facial patterns and explicitly enforced to maintain diversity. This prevents the model from focusing solely on the commonalities of real faces (specific implementation will be discussed below).

Prototype Updating. In each training iteration, we update the real facial prototypes MR with total real facial features in the mini-batch. Specifically, we first calculate the similarity between each prototype mi and specific real facial features, resulting in a correlation map of size K×N. Then, we normalize the correlation map using Softmax function along both horizontal and vertical directions. For the vertical one, we perform feature normalization among the K prototypes:

where fj is the feature of a specific real instance. Similarly, the horizontal one is performed on the N real features:

The results in Equation (2) and (3), i.e., wi,j and vi,j, represent the vertical and horizontal normalized matching probabilities and guide the selection of the most suitable real facial features for prototypes update. Firstly, each sample fj is assigned to its nearest prototype mp1 . And we select the prototype mp1 corresponding to the highest vertical normalized matching probabilities between {w1,j,w2,j, · · · ,wK,j}:

This allocation process means that each sample has exactly one nearest prototype, while the number of samples assigned to each prototype is not fixed. Then, all samples assigned to the same prototype mk can be considered as an update set Uk. Based on corresponding horizontal normalized matching probabilities {vk,1, vk,2, · · · , vk,n}, the prototype mk will be updated using a weighted sum of update set Uk:

where Normalize(·) represents the L2 norm function, and v′ k,j is the variant of horizontal normalized matching probabilities vk,j, which is further regularized with the max value in its respective update set Uk as follows:

To have a more clear perspective, we illustrate the computation and update process related to mk in Figure 3.

Prototypes Distinction Loss and Diversity Loss. We utilize two specially designed prototype losses to achieve precise prototype learning. First, we employ the Prototypes Distinction Loss (LDistinction) to align the distribution within the subclass. Specifically, we encourage each real facial feature fj to move toward its nearest prototype mp1 and the mean real facial prototype ¯MR. To enhance the distinction between the real and fake classes, we introduce a set of fake facial prototypes 1 MF . Then, we further push the real facial feature fj away from the fake prototypes and the mean fake prototype ¯MF . This process can be expressed as:

1 The fake prototypes are updated using the same method as the real ones and relying on the fake instances in the mini-batch.

where the mean prototype ¯Mcan be calculated as follows:

In addition, due to the lack of explicit supervision for subclasses, the Prototypes Distinction Loss may mislead different real subclasses to collapse into an undesirable whole. To address this, we design a Prototypes Diversity Loss to maintain the diversity of real facial prototypes:

where mp2 is the second nearest prototype of fj . α is a predefined margin. In this way, the features of different sub-classes can maintain a proper distance during training.

3.2 Independent Dual-DecisionModule

Because of the difficulty in learning and leveraging real facial concepts, deepfake detection models would classify samples with forgery-related features as fake and classify samples without those features as real. Traditionally, deepfake detectors incorporate a binary classifier and are updated based on a naive cross-entropy loss:

where 2N is the total number of samples in a mini-batch. yi is the ground-truth of feature fi. ρ(·) is the classifier that outputs the probability of belonging to the real and fake classes.

Due to the inherent properties of binary cross-entropy loss, as ρ(fj), the predicted probability of belonging to the fake class increases, 1 - ρ(fj), the probability of belonging to the real class will correspondingly decrease, and vice versa. If the detector can abstract accurate criteria for identifying forgery faces, this 'one rises as the other falls' single-decision logic will achieve a satisfactory optimization result. However, in real-world scenarios, the forgery features are often tangled with overfitted 'mis-artifacts' and have poor generalization ability in the target domain. Therefore, we introduce a more reliable Independent Dual-Decision Classification (IDC) strategy and produce the final classification probability based on both real facial concepts and forgery artifacts. Independent Dual-Decision Classifier. As for a sample fj, to sever the one-to-one correspondence between the fake probability ρ(fj) and the real probability 1-ρ(fj), we extend the output dimensions of the naive binary classifier to twice, and the classification process is as follows:

where the Independent Dual-Decision Classifier ˆ ρ(·) consists of the fully connected layer and the normalization layer. The output vector ˆfj ∈ R1×4 can be further represented as:

where the first and second dimensions of the output vector ˆfj represent the probabilities of belonging to the real and fake classes, respectively. The third and fourth dimensions serve as the auxiliary components for our IDC strategy, whose function will be elaborated in the next section. In this setup, the constraints are imposed on all four dimensions, so that ˆf 0 j no longer entirely follows the variations of ˆf 1 j .

Independent Dual-Decision Regularization. After obtaining the relative outputs from ˆ ρ(·), we try to further reduce the probabilities of belonging to the incorrect classes by appending an extra regularization term R to the classification loss:

where 2N and yj are consistent with that in Equation (10). β is a predefined value to control the optimization intensity. dj is used to represent the discrepancy between the incorrect probabilities and their corresponding auxiliary components:

This way, the probability of belonging to the incorrect class will progressively approach the lower auxiliary component without being influenced by the correct probability. For instance, for a real sample ˆfj (with the label of yj = 0), R balances the value of ˆf 1 j and ˆf 3 j which consistently maintains a small value. This means combining the classifier ˆ ρ(·) and regularization term R, our Independent Dual-Decision strategy can rely on another optimization path (i.e., leveraging the real facial concepts) to make robust judgments and implicitly reduces the reliance on forgery artifacts.

3.3 Training Strategy

Combining all modules, the network framework is optimized in an end-to-end manner based on the following loss function:

Ltotal = LCE +λ1LDiversity +λ2LDistinction +λ3R, (15) where LCE is a binary cross-entropy loss. LDiversity and LDistinction are used to learn comprehensive real prototypes. R is a regularization term for achieving the independent dualdecision strategy. λ1, λ2, and λ3 are the hyperparameters that balance the contribution of each term. Through this holistic loss function, the model optimizes classification results based on both comprehensive real concepts and forgery artifacts, thereby enhancing generalization in unseen target domains.

4 Experiment

4.1 Implementation

We utilized Dlib 2 to extract faces and resize them to 256 × 256 pixels for both the training and testing sets. We conducted the experiments on a single RTX 3090 GPU with a batch size of 16. The backbone we employed is EfficientNet [Tan and Le, 2019] and we also switched the backbone to other networks, e.g., ViT [Dosovitskiy et al., 2021], to demonstrate the robustness of our proposed method in Section 4.2. The hyperparameters λ1, λ2, and λ3 in Equation (15) are selected via grid search and set to 0.6, 1.0, and 1.0, respectively. Similar to the common setup for generalizable deepfake detection [Wang and Deng, 2021; Fei et al., 2022; Cao et al., 2022], we trained our model with the FF++ dataset [R¨ossler et al., 2019]. This dataset includes 1,000 real videos from YouTube, as well as five types of manipulated videos yielding a total of 6,000 videos. Finally, to evaluate the generalization capability of our model, we performed the cross-dataset testing on five widely used deepfake datasets, i.e., Celeb-DF [Li et al., 2020b], DFD [Dufour and Gully, 2020], DFDC [Dolhansky et al., 2020], DFDCp [Dolhansky et al., 2020], and UADFV [Li et al., 2018].

4.2 Main Experimental Results

Performance on Cross-Dataset Evaluation. We reported the generalization capability of several SoTA baselines and our RealID in Table 1. All of them are trained on FF++ and evaluated on other five testing datasets. This cross-dataset setup is challenging since neither the testing pristine/forged videos nor the manipulated techniques are visible in the training dataset. We utilized the AUC (area under the receiver operating characteristic curve) metric to quantify the performance. In addition to reporting the AUCs on each individual dataset, we also calculated the average AUC over the five testing datasets. From Table 1, we have three main observations: 1) Our method RealID significantly improves the generalization ability compared to several SoTA baselines. For instance, our method achieves an AUC improvement of nearly 20% on the Celeb-DF dataset compared to FoCus [Tian et al., 2024] and an average AUC improvement of approximately 18% across five datasets compared to EfficientNet. 2) Our method is more robust. Most of the baselines are only effective on specific datasets, whereas our method achieves significant improvements across all datasets. For instance, RECCE, which demonstrates competitive performance on other datasets, unexpectedly suffers a significant performance drop on CelebDF. In contrast, our RealID approach consistently achieves effective improvements. 3) Exploring detailed image features can enhance performance. For instance, training with selfblended images enables SBI to achieve an impressive average AUC of 89%, and UCF leverages the common forgery features to achieve an AUC of 97% on the UAVDF dataset, outperforming other baselines.

Performance with Different Backbones. By switching the backbone, we further demonstrated the generalizability of our method. Specifically, we selected three widely employed backbones: Xception [R¨ossler et al., 2019], ViT-L-32 [Dosovitskiy et al., 2021], and ViT-B-16 [Dosovitskiy et al., 2021] to evaluate the performance of combining them with our RealID. The training details remain consistent with those that utilize EfficientNet. It is worth noting that RealID can be seamlessly integrated into any deepfake backbone since it does not require modifications to the original architecture.

2 http://dlib.net/.

We have reported the experimental results in Table 2. From this table, we have the following observations: 1) Our RealID remains effective across different backbones. For instance, Xception+RealID achieves a 21.15% improvement in average AUC compared to the Xception alone, while ViTL+RealID and ViT-B+RealID achieve 12.73% and 3.93% improvements on the Celeb-DF dataset, respectively. The above results fully demonstrate the generalization of our approach. 2) Stronger backbone architectures yield better performance but are harder to improve. For instance, the ViT-B-based model outperforms both EfficientNet and Xception in terms of average AUC. However, the performance improvement ratio of ViT-B+RealID is the smallest. 3) Appropriate patch size is more important than the model scale. For instance, ViT-B-16 with a patch size of 16 outperforms ViT-L-32 with a patch size of 32. This may also indicates the importance of local features in deepfake detection.

4.3 Ablation Studies

Module Analysis. In Table 3, we presented the results of ablation studies on two key modules in RealID, i.e., RealC2 and IDC. From the results, it can be observed that both components effectively enhance the generalizability of the model. For instance, compared with the backbone on the Celeb-DF dataset, RealC2 and IDC improve the AUC by 30% and 29%, respectively. Combining the two components, the best result, a 30.57% AUC improvement, can be achieved.

Loss Analysis. To ensure the diversity and distinction of the real prototypes used in RealC2 , we employ two relative loss functions, i.e., LDiversity and LDistinction, to supervise the training process. Specifically, LDistinction ensures the consistency of features within a subclass while maintaining the separability between the real and fake classes. LDiversity enforces diversity among prototypes, preventing them from converging into the same distribution. In Table 4, we reported the impacts of these two loss functions. It can be seen that, between the two loss functions, LDistinction plays a more significant role. For instance, the AUC on the Celeb-DF dataset will drop by 3.3% when LDistinction is removed. One critical reason for this is that LDistinction updates the real prototypes based on real instances. Without it, the real prototypes cannot undergo global optimization. Furthermore, LDiversity is also crucial, contributing approximately a 1.2% improvement. In summary, the combination of these two loss functions in RealC2 delivers the best performance gains.

Parameter Analysis. In Equation (15), the three parameters, i.e., λ1, λ2, and λ3, control the influence of the loss functions. Figure 4 illustrates the impacts of these parameters on the performance. Specifically, we progressively increase a certain hyperparameter within the range of [0, 2], while keeping the other two parameters fixed, to retrain the model. The performance is then evaluated on DFDC and Celeb-DF. From the figure, we observed that the impact of the three hyperparameters on the model's performance on both datasets follows the same pattern: as the corresponding λ value increases, the model's performance first improves and then drops sharply. Based on the combined performance of the three parameters, we empirically selected λ1 = 0.6, λ2 = 1.0, and λ3 = 1.0 as the optimal configuration.

4.4 Qualititive Studies

In Figure 5, we presented heatmaps of no-cherry-pick instances from different datasets. It can be observed that, compared to SBI [Shiohara and Yamasaki, 2022], the baseline model, our approach achieves a more evenly distributed attention across the entire face. For example, on the DFDC dataset, SBI focuses primarily on the left cheek, whereas our approach distributes attention more evenly across the lips, nose, and eyes. This indicates that our RealID makes decisions based on more comprehensive facial concepts rather than the presence or absence of local forgery traces.

In Figure 6, we reported the t-SNE visualization for features in testing datasets. For each subfigure, the left side represents the distribution of test samples under SBI, while the right side shows the distribution under our RealID. We can observe that our RealID significantly reduces the misclassified real samples (marked as large red circles). For instance, a notable number of outliers are corrected on the CDF dataset.

5 Conclusion

In this work, we address the critical challenge of generalization obstacle of deepfake detection models, particularly their tendency to misclassify real instances as fake when applied to unseen datasets. We identify that this issue stems from an over-reliance on forgery artifacts and a limited understanding of 'real'. To overcome these limitations, we propose RealID, a novel approach comprising the Real Concept Capture Module and the Independent Decision Classification Module. Extensive experiments demonstrated that RealID significantly outperforms state-of-the-art baselines in cross-dataset detection scenarios. It not only advances the field of deepfake detection but also offers a solution for future research on balanced and robust detection strategies.

Acknowledgements

This project is supported in part by National Natural Science Foundation of China (Grant No. 62172256, No. 62202278, and No. 62202272); in part by the Ministry of Education, Singapore, under its MOE AcRF TIER 3 Grant (MOEMOET32022-0001).