ForgDiffuser: General Image Forgery Localization with Diffusion Models

1 Introduction

With the rapid development of AI and image generation techniques, the public can easily and inexpensively fake highquality images. These images are almost indistinguishable from real ones, and have greatly blurred the boundaries between reality and fiction, bringing unprecedented challenges and crises to social order, information security, and even public perception. Examples span fake news dissemination, judicial evidence falsification, insurance fraud, and academic cheating. It makes the development of general image forgery localization (GIFL) techniques an important issue in the field of computer vision and security, which aims at precisely locating the tampered areas in the forgery image. Generally, image forgery techniques can be categorized into: traditional image forgery techniques (TIF) and AI-generated image forgery techniques (AIGIF). TIF include: splicing [He et al., 2012; Xiao et al., 2020], copy-move [Wu et al., 2018; Chen et al., 2020] and removal [Chen et al., 2024; Feng et al., 2022]. Splicing is copying and pasting specific content from one image to another; copy-move is moving specific content from one area to another area of the image; removal is deleting specific content from the image; AIGIF is redrawing specific areas of an image with diffusion models, GAN, or other generative techniques.

∗Corresponding Author

GIFL methods are usually achieved by capturing specific forgery features to achieve accurate localization of the tampered region. The diverse tampering types impose higher requirements on the model's ability to balance the differences and commonalities of various forgery features, which makes challenging for GIFL algorithms. Currently, numerous GIFL

methods have been proposed. Examples include manipulation tracing network (ManTra-Net) [Wu et al., 2019], multiview multi-scale supervision network (MVSS-Net) [Chen et al., 2021], contrastive learning image forgery localization network (CFL-Net) [Shi et al., 2023], transformer-auxiliary neural network (TA-Net) [Niloy et al., 2023], multi-feature fusion identification network (MFI-Net) [Ren et al., 2023], and edge distribution guidance and contrastive learning network (EC-Net) [Hao et al., 2024], etc. These methods usually learn the forgery clues left by the forgery manipulation to identify the tampered region, which has made significant progress in GIFL. However, current traditional frameworks still suffer from the following problems: 1) The end-to-end network design often leads to decoder overconfidence, resulting in inaccurate and incomplete predictions. 2) The high degree of blending between forgery regions and the background causes blurred and imprecise boundary localization. Examples of these issues are demonstrated in Figure 1.

To address the above challenges, we consider GIFL as a mask generation task using diffusion models. The diffusion model models the probability distribution of data through progressive denoising and incrementally refining the predicted mask, effectively mitigates the detail localization ambiguity problem. Meanwhile, random sampling generates multiple predictions and evaluates the uncertainty of the predictions, thus effectively mitigating the overconfidence of the decoder. Therefore, we propose a diffusion model-based framework ForgDiffuser, which aims to efficiently generate the tampered region mask by leveraging the faked image as conditional input. Specifically, in the training phase, first, we design the attention-guided module (AGM) to aggregate multi-layer image features efficiently to enhance the richness and contextual expression of image features. Second, we devise the boundary-driven module (BDM) to enhance the detail processing capability of ForgDiffuser by incorporating edge supervision. In the inference stage, We propose the globallocal consistency fusion (GLCF) strategy to enhance prediction stability and reliability by fusing predicted masks from multiple sampling steps.

Our main contributions are as follows:

1. We propose a diffusion model-based GIFL method called ForgDiffuser. To improve tampered region prediction, we design the attention-guided module within the conditional network to extract more reliable image features.
2. We design the edge-driven module to further enhance the detail perception capability of the ForgDiffuser.
3. We conduct extensive experiments on six benchmark datasets, demonstrating that ForgDiffuser outperforms existing GIFL methods, especially in localization accuracy and robustness.

2 RelatedWork

2.1 General Image Forgery Localization

The core issue of GIFL is commonly formulated as a binary segmentation task, and it requires the methods capable of accurately dividing the forgery image into two categories: the untampered region and the tampered region. Earlier, GIFL methods mainly depended on handcrafted features or specific artifacts, such as JPEG artifacts [Amerini et al., 2017], noise patterns [Zhou et al., 2018], and edge inconsistencies [Salloum et al., 2018], etc. However, handcrafted features are redundant and costly, limiting their prevalence in practical applications.

In response to the above, deep neural networks (DNN) can automatically extract deep robust features due to their strong feature learning capabilities [He and Xiao, 2023], which improves the accuracy and generalization of the algorithms. This effectively addresses the limitations of traditional methods and has been widely used in GIFL tasks. For example, ManTra-Net [Wu et al., 2019] proposes an end-to-end network and formulates the task as a local anomaly detection problem. However, it struggles to effectively model global contextual information and accurately capture tampering details. Therefore, recent research has focused on developing more sophisticated and powerful feature extraction mechanisms. Several approaches enhance feature learning by incorporating auxiliary information such as boundaries, texture, and frequency cues. MVSS-Net [Chen et al., 2021] exploits noise distribution and boundary artifacts around tampered regions to facilitate more generalizable feature learning. TANet [Shi et al., 2023] introduces the edge-assisted strategy to further refine the boundary details of the predicted mask. CFL-Net [Niloy et al., 2023] leverages noise information extracted by steganalysis rich model (SRM) filters and contrast learning approach to improve the separability between authentic and tampered regions. EC-Net [Hao et al., 2024] employs a two-stage localization strategy from coarse to fine that significantly improves the localization accuracy. Despite the significant progress achieved in GIFL, current methods still suffer from decoder overconfidence that results in incorrect and incomplete predictions, as well as imprecise boundary localization in complex forgery scenarios.

2.2 DiffusionModels

Diffusion models are generative methods grounded in probabilistic modeling[Ho et al., 2020], leveraging the Markov chain to iteratively denoise random noise into high-quality data. Diffusion models have recently achieved a wide range of successful applications in computer vision, demonstrating excellent scalability, stability, and strong capabilities in addressing complex visual tasks. For example, diffusion models have shown remarkable performance in tasks such as semantic segmentation [Wu et al., 2023], image super-resolution [Gao et al., 2023], anomaly detection [Zhang et al., 2023], object detection [Chen et al., 2023], and monocular depth estimation [Saxena et al., 2024]. Compared to the traditional GIFL framework, the iterative denoising mechanism of diffusion models offers significant advantages in handling complex scenes and diverse objects, while also enabling more precise control over the generative process. In this research, we use the conditional diffusion model framework for the GIFL task, which significantly improves the localization accuracy of tampered regions.

3 ProposedMethod

3.1 Overview

The overall architecture of ForgDiffuser is illustrated in Figure 2. In the training process (shown in the top-left of Figure 2), we first input the RGB image into the pre-trained PVTv2 backbone [Wang et al., 2022] to extract the multilevel features, denoted as f1, f2, f3, and f4. These features are then processed by the proposed AGM to produce the multi-level fused representation fm. Specifically, the features f1, f2, f3, and f4 are passed through four receptive field blocks (RFB), composed of inflated convolutions with various kernel sizes, and then fused via the attention-guided feature fusion module (AGFM) to obtain the fusion feature fm. Next, we input the features f1 and f4 into the BDM to generate the predicted edge map e and the edge feature representation e0. Simultaneously, we add noise to the ground truth mask G0 via the forward diffusion process to produce the noisy mask Gt, which is then fed into the denoising network to obtain the predicted mask ˆ G0. In ForgDiffuser, a lightweight UNet-based architecture is adopted as the denoising network. Specifically, The noisy mask Gt is first encoded through a series of convolutional layers combined with downsampling operations. Following the encoding stage, the fm output from the AGM is concatenated with the e0 from the BDM to form the fusion feature representation gm. Subsequently, gm is passed through the decoder, which consists of convolutional layers and upsampling operations, to produce the predicted mask.

In the inference process, as illustrated in Figure 3, ForgDiffuser starts from a random noise image GT and progressively generates predicted masks over T time steps, guided by forgery features and edge information. To improve the stability and reliability of the result, we introduce the GLCF strategy to fuse T predictions and obtain the final mask Gf .

3.2 DiffusionModel Process

The core idea of diffusion models is to progressively add noise to the data in the forward process, driving it toward randomness, and to learn the reverse denoising process that gradually reconstructs the original data.

Forward process: The forward process is modeled as a Markov chain that gradually adds Gaussian noise to the original data G0 over T time steps, resulting in a noisy sample GT . At each step, Gaussian noise is added according to the following formulation:

where βt denotes the noise variance at time step t, typically increasing with t. The forward process from step 1 to t can be equivalently described by the distribution:

Reverse process: The reverse process aims to recover the original data G0 from the pure noise sample GT by iteratively denoising. Assuming the forward noise schedule βt is known, the reverse process is formulated as a conditional distribution:

where Σθ(Gt, t) = 1-¯αt-1 1-¯αt βt, and µθ(Gt, t) is parameterized by ForgDiffuser as:

where ˆ G0 is the predicted mask generated by ForgDiffuser.

3.3 Attention-guidedModule

With the continued development of image forgery and generation techniques, forgery images have become increasingly sophisticated and diverse. Effectively leveraging local artifacts and global semantic consistency is therefore critical for GIFL. Shallow Transformer layers tend to retain more local details, whereas deeper layers capture richer contextual representations. To effectively combine these hierarchical representations, the AGM is introduced. It consists of the joint attention mechanism and the attention-guided feature fusion module. The fused feature obtained from the AGM is utilized as the additional condition to support the subsequent mask prediction process.

Joint attention mechanism (JA): Inspired by [Woo et al., 2018], we propose the JA that combines channel and spatial attention, aiming to improve the ForgDiffuser's ability to extract key features. For the channel attention component, as shown in the bottom-left of Figure 2, the input feature x is processed in parallel by two branches: one applies adaptive average pooling (AAP), and the other applies adaptive max pooling (AMP), both along the spatial dimensions. Then, each obtained feature is independently passed through the pointwise convolution (PWC), batch normalization (BN), and ReLU. The two resulting feature maps are summed elementwise and passed through the sigmoid function. The obtained map is applied to the input via element-wise multiplication to generate the channel-attention feature h.

For the spatial attention component, the channel-attended feature h is taken as input to further emphasize salient spatial information. Specifically, we compute the maximum (Max) and average (Avg) values along the channel dimension. These two resulting maps are concatenated and processed by a 3×3 convolution followed by the sigmoid function. The obtained map is then multiplied element-wise with the feature h to produce the final output. The formulation of JA is given below:

where BNR denotes BN and ReLU, and C is the convolution operation.

Attention-guided feature fusion module (AGFM): The AGFM is designed to aggregate the multi-layer feature as the comprehensive guide for the subsequent mask prediction. As illustrated on the right side of Figure 2, the inputs fa and fb are first aligned in spatial dimensions by upsampling (Up) fa via bilinear interpolation. Subsequently, both features are then normalized with BN and activated by ReLU, resulting in feature maps ha and hb. To enhance salient features, ha and hb are concatenated and fed into the JA, which generates the attention weight wa and its complementary weight wb. These weights are then applied to ha and hb, respectively, through element-wise multiplication. Finally, the weighted features are summed and processed by a 3 × 3 convolution. The formula for AGFM is as follows:

where R stands for ReLu.

3.4 Boundary-drivenModule

Currently, the transitions between tampered regions and their backgrounds have become more visually consistent. Therefore, accurately detecting the boundaries between forgery and authentic regions is crucial for improving the performance of GIFL methods. To address this, we propose the BDM designed to enhance boundary representation and thus improve localization accuracy.

As illustrated in the bottom-right of Figure 2, the input features to BDM consist of two features: f1 and f4. Here, f1 is the low-level feature map that preserves rich spatial details, while f4 is the high-level semantic feature map capturing abstract contextual information. In the BDM, f1 is upsampled and then concatenated with f4 to form the fused feature map f0, which integrates both fine-grained details and highlevel semantics. This fused feature is subsequently fed into a lightweight encoder-decoder network to predict the edge map e, which serves as a supervision signal to enhance boundary localization during training. Then, we perform an elementwise multiplication between the predicted edge map e and the fused feature f0, yielding the edge-enhanced feature representation e0. This representation is then used to guide the subsequent tampering mask prediction. The operations are formally defined as:

where ED is the lightweight encoder-decoder network.

3.5 Loss Function

ForgDiffuser is designed to predict the forgery localization mask directly. To ensure that the predicted mask generated through the reverse diffusion process progressively approximates the ground truth, we adopt the Weighted Binary CrossEntropy (BCE) and Weighted Intersection over Union (IoU) losses for mask supervision [Wei et al., 2020]. Additionally, the Dice loss is employed to supervise the edge prediction [Xie et al., 2020]. The overall training objective of ForgDiffuser is defined as follows:

3.6 Sampling Strategy

To mitigate overconfident incorrect segmentations in GIFL, inspired by [Zhang et al., 2021], we employ time ensemble to integrate predicted masks from T sampling steps. Then, we design the global-local consistency fusion (GLCF) strategy to enhance the stability and reliability of the predicted mask.

Specifically, let the predicted mask at time t be ˆGt(x, y) and all predicted masks from sampling phase be { ˆGt(x, y)}T t=1 . First, for each step, calculate the global variance of the sample to quantify the predictive stability.

where ¯G(x, y) denotes the mean value of predicted masks across all time steps. The global weights Wglobal(x, y) = e-σ2 g (x,y) are constructed based on global variance, which suppress low-quality sampling steps with large global variance and focus on stable predictions. Next, for the sampling result at each time step, calculate the local variance in 5 × 5 neighborhood of each pixel, measuring the uncertainty of the local prediction.

where ¯Gl,t(x, y) is the neighborhood mean, Nx,y represents the neighborhood window centered at (x, y), and N = |Nx,y| = 25. The local weights Wlocal,t(x, y) = e-σ2 l,t (x,y) are based on local variance, which can suppress the highfrequency noise and clear the boundary of predicted mask.

Finally, multiply and normalize the global and local weights, and perform weighted fusion on predictions at each time step to obtain the final predicted mask Gf (x, y).

where Wt(x, y) denote the integration weights, and ϵ = 10-8 prevents division-by-zero errors.

4 Experiments

4.1 Datasets and EvaluationMetrics

ForgDiffuser is evaluated on widely used forgery image datasets: CASIA1 [Dong et al., 2013], DID [Wu and Zhou, 2021], IMD [Novozamsky et al., 2020], Auto [Jia et al., 2023], BSN and RLS26K [Hao et al., 2024]. These datasets encompass main types of current image forgery techniques. CASIA1 contains 921 images, which includes both splicing and copy-move, and uses image enhancement for data post-processing. DID contains 10 different image inpainting methodologies, including deep learning-based and traditional-based methods, each method contributing 1,000 images, for a total of 10,000 images. IMD is the real manipulation dataset with 2,010 forgery images. Auto is the AIGIF dataset generated by the DALL-E2 model [Ramesh et al., 2022]. BSN is an AIGIF dataset constructed with Brushnet method [Ju et al., 2024] and contains 2500 images. RLS26K is a large-scale TIF dataset containing splicing, copy-move, and removal, which includes 26,000 images. We divide the above datasets into train and test sets in the ratio of 9:1 for experimentation.

In order to evaluate the performance of ForgDiffuser comprehensively, we adopt two evaluation metrics: F1-score (F1), and Intersection over Union (IoU).

4.2 Implementation Details

We implemented ForgDiffuser based on the PyTorch with one NVIDIA L20 with 48 GB memory for training and inference. We trained 100 epochs with batch sizes of 16. AGM is initialized using PVTv2-B4, and the input images are resized to 352 × 352. The AdamW optimizer is employed, and the initial learning rate is set to 0.001. λ1 and λ2 in the loss function of Equation 12 are set to 0.8 and 0.2, respectively. A higher value of λ1 encourages the model to prioritize mask prediction, while still maintaining a balanced emphasis on edge information. The time step T is set to 10 for sampling.

4.3 Comparison with State-of-the-arts

Quantitative comparisons: Table 1 demonstrates the quantitative results of ForgDiffuser with five baseline methods on six benchmark datasets. It is obvious from experimental data that ForgDiffuser achieves optimal results on five datasets, which proves that ForgDiffuser can effectively detect splicing, copy-move, real-world forgery, and AI forgery. Especially on IMD dataset, F1 of ForgDiffuser increases by 0.05 over the suboptimal baseline model EC-Net, and IOU increases by 0.04. On the CASIA1 dataset, F1 of ForgDiffuser only decreases by 0.008 compared to EC-Net, and the quantitative results outperformed EC-Net on the other five benchmark datasets. ForgDiffuser has superior performance in realworld and complex forgery scenarios, primarily due to its integration of the diffusion models' iterative denoising mechanismwith edge-enhanced supervision. In conclusion, the experimental results can demonstrate the superior performance of ForgDiffuse in GIFL.

Qualitative comparisons: In order to compare the results of different methods more intuitively, we show the visualization of the predicted masks of ForgDiffuse and other baseline models on six benchmark datasets in Figure 4. From the visualization results, it is obvious that ForgDiffuser is able to avoid overconfident incorrect segmentation and provide more complete localization results (e.g., lines 1-2). In addition, ForgDiffuser achieved greater accuracy in localizing edge details (e.g., lines 4-6), which proves the effectiveness of BDM.

4.4 Ablation Study

In this subsection, we conduct ablation experiments on the proposed ForgDiffuser to verify the effectiveness of each designed module. The experiments were conducted on CASIA1, DID, IMD, and BSN datasets with the default settings described in Section 4.2. Table 2 presents the results of ablation experiments.

As shown in rows 2-3 of Table 2, the introduction of AGM significantly improved the performance of ForgDiffuser. AGM adaptively extracts features from the conditioned image through the attention mechanism and better combines local information and global context. Specifically, compared to the baseline, the F1 on CASIA1, DID, IMD, and BSN datasets increase by 0.06, 0.02, 0.06, and 0.03, respectively. Meanwhile, the experimental results in lines 3-4 indicate that F1 and IoU show improvements on all datasets with the introduction of BDM, further validating its effectiveness. In addition, the results in lines 1-2 demonstrate the effectiveness of the sampling strategy in the proposed method.

4.5 Robustness Evaluation

To evaluate the robustness of ForgDiffuser, we conducted experiments on CASIA1 and IMD datasets using common image attack methods, including Gaussian noise with standard deviation of 0.02, 0.04, 0.06, 0.08 and 0.1; salt & pepper noise with noise intensity of 0.02, 0.04, 0.06, 0.08 and 0.1. Gaussian noise simulates continuous perturbations such as sensor noise or transmission interference. In contrast, salt & pepper noise represents discrete distortions like pixel-level corruption or abrupt intensity changes. These attacks degrade edge and structural cues in tampered regions, making the tampered areas harder to distinguish from authentic areas and challenging the GIFL task in both localization accuracy and robustness. The experimental results are shown in Figure 5. The left column shows the results on CASIA1 dataset and the right column presents the ones on IMD dataset. From the experimental results, it can be concluded that as the intensity of Gaussian and salt & pepper noise increases, the tampering localization accuracy of all models exhibits a decreasing trend. ForgDiffuser achieves the best performance under both image attack types, significantly outperforming other methods and demonstrating strong robustness.

5 Conclusion

In this paper, we propose ForgDiffuser, a GIFL framework based on conditional diffusion models. The core of ForgDiffuser lies in predicting the tampered region mask through the iterative generation mechanism of diffusion models. It effectively alleviates decoder overconfidence through the iterative sampling strategy. To further improve detection accuracy, we design the AGM to deeply fuse the global semantic features with the low-level detail features of the conditioned image, providing more precise guidance for subsequent mask prediction. In addition, the BDM is introduced to precisely capture edge details between tampered regions and the background, effectively enhancing the accuracy of boundary localization. Experimental results on multiple benchmark datasets show that ForgDiffuser achieves superior performance in detection accuracy and robustness compared to existing mainstream methods, demonstrating its strong potential in GIFL.

Acknowledgments

This work is supported by National Key Research and Development Program of China [grant number 2024YFF0907404].