CorrDetail: Visual Detail Enhanced Self-Correction for Face Forgery Detection

1 Introduction

The rapid advancement of generative technologies, exemplified by Artificial Intelligence Generated Content (AIGC) [Cao et al., 2023], has profoundly liberated societal productivity, enabling innovative applications across diverse sectors such as news, literature, entertainment, education, and industry. However, the capability of AIGC to produce highquality content from simple prompts and guidance has simultaneously lowered the barriers for malicious actors to fabricate deceptive information [Zhao et al., 2023]. This ease of content falsification has introduced significant challenges in verifying identity information, resulting in an increase in financial fraud, misinformation dissemination, and identity theft, thereby posing severe threats to societal security.

* Corresponding author. Email: beiyj@zju.edu.cn

To address the security threats posed by the facial image forgery technologies, researchers have focused on identifying visual artifacts inherent in forged facial images by analyzing the images themselves. Current approaches [Cao et al., 2022a; Dong et al., 2022; Qian et al., 2020; Wang et al., 2022; Corvi et al., 2023] involve forensic detection from various perspectives, including the network architectures of forgery methods (such as GANs, autoregressive models, and diffusion models) and the domain-specific features of these methods (encompassing both frequency and spatial domains). These techniques have demonstrated commendable performance in facial authentication tasks. However, concentrating solely on forgery detection within a single visual modality encapsulates the forgery process as a black box. Models that rely on a single visual modality typically yield only binary forgery classification results, lacking the provision of critical visual cues necessary for understanding the basis of the discrimination. Consequently, this leads to a significant deficiency in the interpretability of the forgery detection process.

To enhance the interpretability and informational depth of facial image forgery detection models, researchers have endeavored to incorporate textual information, thereby developing multimodal deepfake detection frameworks. Leveraging established text-image encoding technologies such as CLIP [Radford et al., 2021] and BLIP [Li et al., 2022], a bridge for feature interaction between the image modality and the text modality has been constructed. Within the field, two primary approaches have emerged: text-image information interaction and the application of large-scale visual language models (VLMs). The first approach involves integrating corresponding textual information with forged facial image data to augment the dimensionality of feature extraction for forgery detection. The second approach builds upon this foundation by replacing traditional neural network models with more extensive and comprehensively pre-trained VLMs. Through a question-and-answer dialogue mechanism, VLMs can provide detailed semantic information during the deepfake detection process, including specifics about the forged regions, methods employed, and the overall forgery framework [Chang et al., 2024; Zhang et al., 2025]. However, within the domain, the training and inference mechanisms of VLMs are susceptible to hallucination phenomena, which not only degrade model performance but also risk supplying erroneous forgery cues. This poses significant challenges to the reliability and accuracy of multimodal deepfake detection.

To address these challenges, we propose a visually enhanced self-correction framework, denoted as CorrDetail. By providing error-guided questions, CorrDetail establishes a corrective question-and-answer mechanism that cultivates the ability to identify forged details and rectifies training environments prone to hallucinations. In addition, a visual fine-grained detail enhancement module is integrated to furnish CorrDetail with more precise visual forgery cues. Finally, we devise a fused decision strategy that combines visual information compensation with model bias reduction, further strengthening the model's ability to discriminate extreme samples. Our specific contributions are as follows:

• We propose a multimodal forgery detail self-correction framework, alongside the creation of a Self-Correction Visual Question Answering (SCVQA) dataset tailored for facial forgery detection. This framework seamlessly combines a self-correction strategy with a cross-model forgery detail enhancement module, aimed at identifying authentic forgery characteristics and producing intelligible, explanatory text-based forgery clues.
• We introduce a novel inference decision mechanism that merges a dual-round question-answering process with multi-scale visual feature interactions, effectively correcting biases inherent in large models. This mechanism entails the construction of multi-expert reasoning decisions, thereby bolstering the reliability and precision of the forgery detection process.
• Extensive comparative, ablation, and visualization experiments demonstrate that our approach not only achieves state-of-the-art performance but also delivers precise and accurate forgery details, showcasing remarkable generalization capabilities.

2 RelatedWorks

2.1 Face Forgery Technology

GAN-based and autoregressive-based models were pioneers in the field of image generation. Driven by these models, realistic facial modification techniques such as BigGAN [Brock et al., 2018] and StarGAN [Choi et al., 2017] emerged, enabling traditional facial forgery methods to be categorized into face swapping, face editing, and face reenactment. With the rapid advancement of AIGC technologies, prompt-guided facial image generation methods have become widely accessible, allowing for the creation of diverse facial images with minimal input, such as textual descriptions or sketches. Notable products like Midjourney [Midjourney, 2022] and Stable Diffusion [Stability.ai, 2023] have surged in popularity by integrating prompts into the AIGC generation process (e.g., diffusion models), thereby facilitating guided and procedural content generation. This prompt-guided AIGC generation approach starkly contrasts with traditional task-oriented methods [Bei et al., 2024], significantly increasing the difficulty of developing generalized facial forgery detection models.

2.2 Vision-based Face Forgery Detection

Vision-based Face Forgery Detection is the predominant approach in the field, concentrating on the forgery artifacts present in manipulated facial images. This methodology can be specifically categorized into frequency-domain-based and spatial-domain-based techniques [Bei et al., 2024]. The core principle involves amplifying forgery traces through transformations in different domains, thereby capturing clearer and more precise artifacts.

In the frequency domain, forgery features are extracted at a fine-grained level by mapping forged images into high-frequency, mid-frequency, and low-frequency dimensions [Qian et al., 2020]. This allows for the detailed identification of subtle manipulation signs that may not be easily discernible in the spatial domain. Conversely, in the spatial domain, the focus is on identifying inconsistencies introduced during the face modification process, such as artifacts, glare, and blurred regions [McCloskey and Albright, 2018; McCloskey and Albright, 2019; Li et al., 2020a]. To address the novel generation methods introduced by AIGC, new approaches have emerged that rely on reconstruction errors for forgery detection in the spatial domain. These methods analyze discrepancies between the original and reconstructed images to identify potential manipulations.

While the aforementioned methods effectively accomplish facial image forgery detection, single-modality approaches fail to provide specific semantic information regarding the forgeries. The absence of disclosed forgery cues results in insufficient interpretability of these methods, limiting their ability to offer comprehensive insights into the detection.

2.3 Multimodal-Based Face Forgery Detection

Recent advancements in image-text pre-trained models [Liu et al., 2023; Radford et al., 2021; Li et al., 2022] have driven researchers to adopt multimodal approaches to enhance face forgery detection. Currently, these approaches can be broadly categorized into contrastive learning-based methods and VLM-based methods.

The contrastive learning-based methods leverage the comparison of encoded image-text features to align and reinforce discriminative features or utilize textual information to amplify forgery traces within the image modality [Wu et al., 2023; Khan and Dang-Nguyen, 2024]. In contrast, VLMbased methods utilize VLMs for forgery detection, leveraging their powerful performance for tasks like data processing, feature extraction, and decision-making.

While both categories have demonstrated performance improvements, they exhibit significant areas requiring enhancement. Firstly, current methodologies primarily involve simple encoding and fusion or alignment of multimodal features. This coarse-grained discrimination approach impedes the construction of comprehensive and deeply nuanced features. Furthermore, VLMs are often treated as black-box tools or auxiliary aids within the domain, lacking an indepth development of feature extraction and discrimination decision models grounded in the fundamental principles of VLMs. More critically, this coarse-grained approach induces hallucination phenomena within VLMs, hindering their ability to deliver specific and accurate discrimination results and analyses. As a result, the models are unable to provide detailed and reliable insights into the nature and extent of the forgeries, significantly compromising the overall effectiveness and trustworthiness of the detection system.

2.4 Critique and Refine Mechanism

In the course of in-depth research and application of large language models (LLMs), these models have demonstrated remarkable potential in critique and refine. Researchers have leveraged LLMs to assess the results of specific models, facilitating their self-improvement. In the domain of VLMs, critique and refinement mechanisms have also been introduced to perform logical corrections [Saunders et al., 2022; Madaan et al., 2023; Gou et al., 2024]. Specifically, through querying of results and multi-round interactions, LLMs are endowed with self-improvement mechanisms that allow them to critique and refine their reasoning capabilities [Lin et al., 2024b; Yao, 2024; Li et al., 2024b; Wu et al., 2024]. Following these advancements, the critique and refine Mechanism has garnered significant attention, and in the benchmark domain, there has been a growing trend to use critique mechanisms to assess the critique-correction reasoning ability of LLMs [Lightman et al., 2023; Li et al., 2024a; Luo et al., 2024; Huang et al., 2024].

The aforementioned work on the critique and refine mechanism in text or multimodal LLMs primarily focuses on correcting logical reasoning capabilities. Inspired by this, we propose a novel method that employs error-agnostic questionguided mechanisms, enabling the model to learn how to identify visual forgery features and provide accurate responses. This method does not focus on correcting logical reasoning abilities but rather aims to prevent hallucinated responses, ensuring more reliable and precise detection of visual forgeries.

3 Method

Fig. 2 shows the overall framework of CorrDetail. In this section, we present the various modules of CorrDetail. In Section 3.1, we introduce an innovative self-correction VQA training strategy designed to augment the forgery clue extraction capability of the VLM. Section 3.2 unveils the crossmodel forgery detail enhancement module, which provides intricate visual features for facial forgery detection. In Section 3.3, we introduce a multi-expert ensemble decisionmaking strategy, which strengthens the reliability and precision of the final outcome.

3.1 Self-Correction Visual Question Answering

In the field of face forgery detection, the VQA training of multimodal large models for related detection tasks is essentially a form of knowledge infusion. Although it can continuously improve model performance through large amounts of data, inspired by [Li et al., 2024b] masking strategy and the fact that images dominate the dual-modal interactions with text in the current task, we aim to enable the model to capture the forgery visual features in images to the greatest extent during training. To this end, we propose a novel training method, as shown in Fig. 1.

Based on the VQA format provided by DD-VQA [Zhang et al., 2025] in the FF++ dataset [Rossler et al., 2019a], we enhance the training process by introducing a new prompt in the question. During training, we introduce incorrect and authentic forgery details into the question. There is a 70% probability of incorporating erroneous details into the original prompt (Q1 in Fig. 2 is an example), a 15% probability of adding genuine forgery details through synonym substitution (Q2 in Fig. 2 is an example), while the remaining questions remain unaltered. We deliberately refrain from entirely inserting incorrect details, as this could prompt the model to exploit the prompt directly, deriving answers opportunistically. This Self-Correction Visual Question Answering (SCVQA) dataset equips the VLM to accurately identify authentic forgery details, rather than producing hallucinated responses during the training process.

3.2 Cross-model Forgery Detail Enhancement

In commonly employed VLM, the intricate visual tokens and succinct text tokens are aligned and mapped into a shared latent space. During this alignment and mapping process, the sophisticated visual tokens inevitably lose some of their detailed visual features. However, face forgery detection must prioritize the capture of these forged visual nuances. To address this, we have designed a Cross-model Forgery Detail Enhancement (CFDE) module, which enriches the VLM with additional visual details, enabling it to learn to discern and identify forged visual features.

Specifically, as depicted in Fig. 2, once the image passes through the CLIP and Projector modules, its embedding is devoid of forgery-specific visual details, such as the intrinsic consistency of surrounding elements and the relational information of relevant positions. Therefore, we introduce an additional vision transformer branch to extract the image's intrinsic visual features, providing supplementary information for face forgery detection.

Formally, the extra text prompt Pextra is processed through the same tokenizer and fused with the image tokens Itoken and the image visual feature Ivit (processed through the Vision Transformer) in a cross-modal attention mechanism. The extra text prompt is used as the query Q, the image tokens as the value V , and the image the visual feature as the key K to generate the target attention map. Finally, this attentionenhanced image token ˜ Itoken is obtained by adding the resulting attention map to the original image token in a residual manner. The specific formula is as follows:

where CLIP(·) represents LLaVA's own visual encoder, and Projector(·) represents the pre-trained image token mapping module, A(Q,K, V ) denotes the self-attention operation in transformer. The final obtained ˜ Itoken will serve as the new image token input to the VLM.

3.3 Decision Fusion

This module is primarily subdivided into two segments. The first segment is the Prompt Enhancement Mechanism, which is chiefly conceived to post-process the VLM, thereby liberating the output from the biases and fairness challenges stemming from the tendentious misjudgment of faces in images with small proportions. The second segment is the Dual-Expert Model Decision, in which an independent visual branch is trained separately to alleviate the information loss induced by image embedding within the VLM. Through collaborative decision-making, the two branches harmonize, resulting in superior performance.

Prompt Enhancement Mechanism. During the testing phase, the same image is provided with a basic question and an additional guiding prompt, such as "Does the image look real or fake?" and "Does the image look real or fake? Pay more attention to the face/background." Variations in the text input will inevitably lead to changes in the model's binary classification probability output.

Therefore, the Prompt Enhancement Mechanism is designed to address the issue where the detection probability diminishes when the model places excessive focus on the face or background with the added guiding prompt, suggesting that the initial detection probability is dubious. Fig. 3 illustrates the scenario in which the model's predicted probability for "fake" decreases as more attention is directed towards the face, indicating that the initial "fake" prediction is unreliable. A possible explanation is that the model's initial prediction relies on the background features for challenging samples with smaller facial proportions.

For the initial prediction probability [pf , pr] and secondround prediction [p⋆f , p⋆r ] with additional guiding prompt, the final fusion prediction Pfus is adjusted as follows:

when the initial "fake" prediction pf satisfy the condition (1λ) ≥ pf ≥ λ, where λ is set as a hyperparameter, serving as the exclusion threshold. For pf > (1 - λ), pf < λ or any other cases, final fusion prediction Pfus will simply match the initial prediction [pf , pr].

Dual-Expert Model Decision. As depicted in Fig. 2, an additional purely visual branch is trained independently using cross-entropy loss, enabling it to perform forgery detection autonomously. Its output is subsequently integrated with the VLM branch's output, enhancing the probability for discriminative consistency. The specific formula is as follows:

where Pfus represents the output of the VLM branch, Pvis represents the output of the traditional visual branch, and finally Pfinal is the final discriminative output.

4 Experiments

In this section, we introduce the experimental setup and environment, present a performance comparison between CorrDetail and recent approaches across various datasets, and highlight the advantages of our approach. Finally, we conduct ablation studies to further validate the effectiveness of each module.

Architecture and Implementation Details. The vision branch in 3.3 consists of three components: CLIP, VIT, and Multi-scale. The CLIP module corresponds to the corresponding part in the VLM's architecture, VIT primarily uses vit-L/16, and Multi-scale extracts features at different scales by progressively decomposing through multi-level sampling. More details on the structure of each module can be found in the supplementary materials. We fine-tuned LLaVA-1.5-7B on the SCVQA dataset using a learning rate of 5e-5 , training for 3 epochs with a batch size of 64 on three Nvidia A100 GPUs. The training process employed cosine annealing learning rate decay and the AdamW optimizer. The threshold λ is set as 0.1. More details are given in the supplementary materials.

SCVQA Dataset. Based on the original data from the FF++ dataset, we have constructed the SCVQA dataset, which consists of 19,797 [image, question, answer] triples to support the self-correction mechanism. The dataset includes three types of data formats: (1) 70% of the data is in the form of [image, question, incorrect detail description as the answer], (2) 15% is in the form of [image, question, null answer], and (3) 15% is in the form of [image, question, correct detail description as the answer].

Evaluation Dataset and Metrics. We evaluate the proposed method using the FF++ [Rossler et al., 2019b], CelebDF [Li et al., 2020b], WildDeepfake (WDF) [Zi et al., 2020], DeepForensics-1.0 (DFR) [Jiang et al., 2020], and DeepFaceGen [Bei et al., 2024] datasets. The WDF contains 7,314 facial action sequences extracted from 707 DeepFake videos, and the DFR represents the largest face forgery detection dataset by far, with 60,000 videos constituted by a total of 17.6 million frames. CelebDF employs face swapping as the primary forgery technique, encompassing a total of 5,639 videos. FF++ includes four forgery methods-Deepfake, Face2Face, FaceSwap, and NeuralTextures-comprising a total of 4,000 videos. The videos from both FF++ and CelebDF are cropped into images for facial forgery detection, establishing them as widely adopted evaluation datasets. DeepFaceGen is a facial forgery dataset that includes both video and image modalities. From this dataset, we selected facial forgery images using both traditional taskoriented and novel prompt-guided forgery methods, encompassing a total of 27 approaches. Consistent with the evaluation metrics established in the benchmark papers, we adopt Accuracy (ACC), Equal Error Rate (EER), and Area Under the Curve (AUC) as metrics.

4.1 Comparison with SOTAs

In this section, We compare CorrDetail with several mainstream methods, including XceptionNet, HiFi-Net, RECCE, CD-Net, Local-relation, RFM, RECCE (DD-VQA), EN-B4, GS, and UIA-VIT. We fine-tune CorrDetail on SCVQA constructed with FF++. As presented in Table 2, CorrDetail achieves over 94% in both ACC and AUC in the intra-testing evaluation on FF++, maintains over 72% in ACC and AUC in cross-testing on CelebDF, and reduces the EER to below 34%, attaining the best performance in six out of seven evaluation metrics across intra-testing and cross-testing. The remaining metric also attains the second-best performance. Notably, CorrDetail demonstrates a significant advantage over existing SOTA models in the field, specifically RECCE (DDVQA), which is based on VLM, and GS, which is based on feature decoupling. Fig. 4 further illustrates CorrDetail's ability to accurately detect instances of facial forgery and provide precise forgery cues.

4.2 Generalization on Different Datasets

To further validate the effectiveness of CorrDetail, we conducted generalization performance experiments. Specifically, on CelebDF, we trained CorrDetail on the SCVQA and performed cross-dataset evaluation by testing it on the CelebDF. We compared the performance of CorrDetail against other compared methods that were also trained on FF++. On DeepFaceGen, we evaluated CorrDetail against six SOTA detectionmodels. It is important to note that the six detectionmodels were subjected to intra-testing, meaning they were both trained and tested on DeepFaceGen. In contrast, CorrDetail was trained on SCVQA and tested on DeepFaceGen, representing a cross-testing scenario.

The experimental results for the CelebDF are illustrated in Table 2. In the cross-testing experiments presented in the right column, our method achieved the best performance, surpassing all comparison methods. For the more challenging DeepFaceGen benchmark, the results depicted in Fig. 5 demonstrate that, even under the less favorable experimental setup where CorrDetail was trained on SCVQA and tested on DeepFaceGen while the comparison algorithms were trained and tested on DeepFaceGen, CorrDetail still outperformed the SOTA model RECCE.

In these unseen data scenarios, encompassing both traditional forged facial data and novel AIGC-generated forged data, CorrDetail consistently delivered outstanding performance. These results conclusively demonstrate the robustness and effectiveness of CorrDetail in diverse and challenging forgery detection environments.

4.3 Ablation Study

In this section, we discuss practical effectiveness of each module in CorrDetail, including the enhancement of learning in SCVQA compared to VQA, the effectiveness of CFDE in accurately enhancing target semantics within images, and, finally, whether the decision enhancement module can effectively address certain challenging samples that VLMs struggle to handle. Detailed results are listed in Table 3.

Illustrating the Effectiveness of SCVQA. To demonstrate the performance of SCVQA, we compared models trained using SCVQA with the GPT-4o mini and visualized the outputs on several face-forged images, as shown in Fig. 4. It is worth noting that during this ablation process, no additional innovative modules were incorporated-only prompt guidance was added. Clearly, for some images that 4o mini fails to classify correctly, our model accurately identifies the reasonable or unrealistic regions and details.

Discussing the influence of CFDE. We compared the traditional prompt-based attention enhancement method with CFDE through heatmap visualizations, as shown in Fig. 6. It is evident that feature-level processing yields a more focused overall effect compared to traditional methods. For instance, at the skin level, our method produces a broader heat distribution compared to the prompt-based method, while at the nose level, it exhibits more concentrated heat around the corresponding parts of the image. This comprehensive comparison demonstrates that enhancing from a cross-modal semantic perspective is more effective in guiding the model's recognition than relying solely on text-based approaches.

Discussing the influence of decision fusion. In this section, we focus on addressing native negative sample issues that can be resolved through integrated decision-making, as shown in Fig. 3. For certain images where the background occupies a large proportion of the frame or where the face is slightly blurred, the VLM's attention to the facial region becomes weaker, leading to incorrect classifications. By visualizing the outputs, we further demonstrate the effectiveness of post-processing. Meanwhile, the enhancements from the visual module are more reflected in the overall improvement of classification performance, as shown in Table 3.

5 Conclusion

In this paper, we present CorrDetail, a novel face forgery detection framework that enhances both performance and interpretability through its key modules. The self-correction mechanism utilizes error-guided questions to improve the VLM's efficiency and generalization, achieving SOTA results in face forgery detection tasks. Our visual fine-grained detail enhancement module provides precise forgery cues by strengthening specific semantic features. Additionally, the fused decision strategy employs post-processing to address focus issues the model cannot resolve independently. CorrDetail outperforms recent methods across multiple benchmarks, demonstrating superior accuracy and generalization.

Acknowledgments

This work is funded by National Key Research and Development Project (Grant No: 2022YFB2703100), Ant Group through CCF-Ant Research Fund, and Information Technology Center, ZheJiang University.

Contribution Statement

Authors with † have made the same contribution to this work.