Learnable Frequency Decomposition for Image Forgery Detection and Localization

1 Introduction

With the development of image editing and generation technologies[Goodfellow et al., 2020; Ho et al., 2020; Ramesh et al., 2022], image forgery has become increasingly easier and more prevalent. This advancement has raised widespread public concern, as forgeries pose threats to personal and societal security and privacy, potentially even causing panic. For instance, malicious users can easily modify objects in images using advanced forgery techniques, creating fake news or falsifying evidence in court. Therefore, it is crucial to develop effective and robust methods for detecting and localizing image forgeries.

∗Corresponding author.

In fact, significant progress has been made in forensic forgery detection technologies in recent years. Some works are based on clearly defined low-level features, such as JPEG compression, demosaicking, or interpolation [Bammey et al., 2020]. Certain detectors show favorable results for specific types of image tampering, such as splicing. The academic community has also made milestone achievements in general image forgery detection. For example, RGB-N [Zhou et al., 2018] distinguishes real and manipulated regions by analyzing noise inconsistencies in steganalysis model filters. MVSS-Net [Chen et al., 2021b] learns multi-view features by utilizing noise patterns and edge artifacts. ObjectFormer [Wang et al., 2022a] uses high-frequency information to detect subtle manipulations. However, most existing methods either focus solely on spatial domain forgery features or treat frequency features as a separate modality, with limited exploration of the frequency-domain characteristics presented by forged objects. While artifacts introduced by image editing are not visible in the RGB domain, they often become evident in the frequency domain [Chen et al., 2021a], inspiring us to further investigate the frequency characteristic changes caused by image tampering.

As shown in Figure 1, we apply Fourier transform to both real and forged images to analyze the changes caused by image tampering. On one hand, we subtract their amplitude and phase spectra to observe the changes. To facilitate visualization, the frequency domain residuals, amplitude, and phase spectra in Figure 1a are subjected to inverse Fourier transform. It can be observed that the changes in the image phase components are more closely related to the tampering mask, indicating that manipulation traces are more easily detectable in the phase component. This is because most forgery methods typically manipulate images at the object level (semantic content), and the phase components in the Fourier space correspond to the semantic information of the image, while amplitude components correspond to style information [Xu et al., 2021]. On the other hand, we also analyze the impact of forgery on the high- and low-frequency bands of the image. We randomly select a thousand pairs of real and forged images and statistically analyze the residual distributions of real and forged images across different frequency bands. As shown in Figure 1b, the changes caused by forgery exist in both high and low frequencies, which is more universal than previous studies that suggest forgery is concentrated in high frequencies [Wang et al., 2022a].

Based on these observations, we propose a learnable forensic frequency decomposition network (F2D-Net) for image forgery detection and localization. Specifically, F2D-Net consists of two components: the Spectrum Decomposition Subnetwork (SDSN) and the Frequency Separation Subnetwork (FSSN). The SDSN decomposes the image into amplitude and phase spectra, focusing on learning the phase spectrum to capture forgery traces and identify forged objects, thereby improving forgery detection accuracy. The SDSN also includes spatial-frequency interaction blocks to facilitate the interaction of forgery features in both spatial and frequency domains, further enhancing detection accuracy. The FSSN is responsible for adaptively decomposing the output of the SDSN into high and low frequencies, and applying a divide-and-conquer strategy to refine each frequency band. This approach mitigates the optimization difficulty caused by the coupling of forgery traces across different frequencies, allowing for better capture of the pixels belonging to the forged object, and thus improving localization accuracy. Our method improves detection and localization accuracy by leveraging phase semantic information and decoupled highand low-frequency domain information to assist the model in distinguishing and capturing forged objects. In summary, our contributions are as follows:

• We analyze the frequency characteristics changes caused by image tampering and propose a new Fourier-based method for image forgery detection and localization-F2D-Net.
• We develop the spectral decomposition subnetwork, which focuses on learning manipulation traces in the phase spectrum, thereby accurately capturing subtle changes in the phase of forged objects and enhancing forgery detection.
• We design the frequency separation subnetwork to reduce the interference between forgery traces in different frequency bands, further improving the precision of forgery localization.

We conduct extensive experiments on multiple benchmarks and demonstrate that our method outperforms state-of-the-art methods both qualitatively and quantitatively.

2 RelatedWorks

Image forgery detection and localization. Most early works propose to localize a specific type of forgery, including splicing [Huh et al., 2018], copy-move [Cozzolino et al., 2015], and removal [Aloraini et al., 2020]. Although these methods perform well in detecting the specific forgery type, they are obviously insufficient in dealing with real-world cases due to the unknown real forgery types. Therefore, tackling multiple forgery types in one model has been emphasized in recent work. RGB-N [Zhou et al., 2018] uses a dual-stream network to extract RGB and noise features for detecting and localizing image forgery by capturing visual artifacts and modeling region inconsistencies. ManTra-net [Wu et al., 2019] leverages an end-to-end network, which extracts image manipulation trace features and identifies anomalous regions by assessing how different a local feature is from its reference features. SPAN [Hu et al., 2020] attempts to model the spatial correlation via local self-attention blocks and pyramid propagation. MVSS-Net [Chen et al., 2021b] has designed an edge-supervised branch that uses edge residual blocks to capture fine-grained boundary detail in a shallow to deep manner. PSCCNet [Liu et al., 2022] uses a progressive spatial-channel correlation module that uses features at different scales and dense cross-connections to generate operational masks in a coarse-to-fine fashion. HiFi IFDL [Guo et al., 2023] employs a hierarchical fine-grained approach for IFDL representation learning, utilizing level-wise classification and dependencies for improved performance. In this work, We design a deep Fourier-based network to enhance phase features and adaptively learn low-high frequencies for improved IFDL.

3 Methodology

3.1 Fourier transform of images

For the forgery, some subtle manipulation traces are no longer visible in the spatial domain. Previous works rarely learn the features of tampering artifacts in the frequency domain. To this end, we revisit the forged images via Fourier transform and design a forensic frequency decomposition network to capture the frequency representations of forgery effectively.

As recognized, the Fourier transform is widely used to analyze the frequency content of images. Given a single channel image x with the shape of H ×W, the Fourier transform F converts to the Fourier space as a complex component F (x), which is expressed as:

and F -1 (x) defines the inverse Fourier transform accordingly. Since an image or feature may contain multiple channels, we separately apply Fourier transform to each channel in our work with the FFT [Frigo and Johnson, 1998]. The amplitude component A (x) (u, v) and phase component P (x) (u, v) are expressed as:

where R(x) and I(x) represent the real and imaginary parts of F (x) respectively.

Targeting at Image forgery detection and localization, we employ Fourier transform to conduct the detailed frequency analysis of forgery, as shown in Figure 1. There are two observations in frequency domain: 1) Manipulation traces are more easily detected on the phase components; 2) variations caused by image manipulation are spread over both high and low frequency components. Therefore, it is logical to perform effective learning on the phase component and model low-high frequency components adaptively.

3.2 Overview

Based on the above analysis, we design a simple but effective F2D-Net framework as shown in Figure 2. The input image is represented as X ∈ RH×W×3 , where H and W represent the height and width of the image. First, the image is input to the spectral decomposition subnetwork, obtaining Gr ∈ RHs×Ws×Cs . Next, Gr and Xd ∈ RHs×Ws×Cs obtained by downsampling X are transformed into frequency domain by

where A(·) and P(·) indicate the amplitude and phase respectively. To reduce the effect of altered amplitude component, we recombine to obtain G′r ∈ RHs×Ws×Cs :

where Conv denotes a 1 × 1 convolution layer. Following [Chen et al., 2021b; Wang et al., 2022b], we also use CNNs to extract edges from coarse features as a kind of supervised information. Meanwhile, the coarse features G′r are adaptively divided into low-high frequency for learning separately through a predicted frequency demarcation. Then, the features after the frequency separation subnetwork are fused with spatial information to output the predicted localization map. Finally, we pool the predicted probability map and use a fully connected layer for forgery detection.

3.3 Spectral decomposition subnetwork

Most forged images are carefully processed to hide tampering artifacts, making it challenging to model inconsistencies in the spatial domain. To overcome this challenge, SDSN employs deep Fourier transforms to independently process the amplitude and phase spectra of images, with a heightened focus on phase learning.

We use ResNet-50 pretrained on ImageNet [Deng et al., 2009] as the backbone network of the spectral decomposition subnetwork. To enhance the phase characterization, we design the phase-emphasized interaction (PEI) block and place it alternately with the ResNet block, as shown in Figure 3. The PEI blocks specifically learn the phase information of forged images, and progressively improve the network's ability to capture the tampered artifacts, thus improving the accuracy of the forgery localization.

We illustrate the PEI block as shown in Figure 3. According to spectral convolution theorem in Fourier theory [Katznelson, 2004], processing information in Fourier space is capable of capturing the global frequency representation in the frequency domain. In contrast, the normal convolution focuses on learning local representations in the spatial domain. Thus, we introduce the dual domain information interaction to facilitate the information flow and learn the complementary representation. Specifically, it comprises a spatial branch and a frequency branch for processing spatial and frequency representations. Denoting Gp ∈ RHp×Wp×Cp as the input features of the PLB block, the frequency branch first uses a 1 × 1 convolution to process Gp that obtains Gf0 and then adopts Fourier transform to convert it to the Fourier space:

Next, we adopt the operation OF(·) that consists of 1×1 convolution layers on its phase component, and then recompose the operated result with the amplitude component to obtain Gf1 ∈ RHp×Wp×Cp , which is expressed as:

In this way, Gf1 is the processed result of the frequencydomain representation with enhanced phase information. Meanwhile, the spatial branch processes information in the spatial domain to obtain Gs1 ∈ RHp×Wp×Cp :

where OS denotes a residual block with 3 × 3 convolution layers. Then, inspired by [Luo et al., 2021], we introduce the space-frequency dual cross attention (SFDCA) to interact with frequency domain features and spatial features. As Figure 4 illustrates, SFDCA employs shared attention to facilitate the comprehensive interplay between the two domains, a process that can be described as:

where G′f1 and G′s1 are the output of the interacted spatial branch and frequency branch. And they both get the complementary representation, which benefits for these two branches to obtain more representational features. The following spatial and frequency branches are formulated in the same way as above and output Gf2 and Gs2 :

Finally, we concatenate Gf2 and Gs2 and then apply a 1 × 1 convolution operation to integrate them:

where Gc ∈ RHp×Wp×Cp is the output of PEI block. The phase information of the forged images is effectively enhanced by the subnetwork with five PEI blocks allowing to obtain features containing more forgery traces.

3.4 Frequency separation subnetwork

As described in Sec. 3.2, the coarse features G′r are fed into adaptive frequency separation subnetwork. It adaptively divides the forged image into low and high frequencies components with different gradients respectively, and uses the spatial branch for information compensation.

G′r ∈ RHs×Ws×Cs is first transformed into frequency domain by Fourier Transform:

Then, the subnetwork predicts a two-dimensional mask with mask prediction network (MPN). Specifically, the G′r is mapped into one-dimensional vector by global average pooling and then pass through the fully-connected layers to generate two scalars α and β in the range of 0 to 1:

where σ denotes the sigmoid activation function. The mask M ∈ RHs×Ws is obtained by setting the corresponding bounding box as 1 and the remaining as 0:

Then, based on the obtained mask, we filter out the low and high frequencies parts of the coarse features G′r . Meanwhile, the convolution operation is performed over amplitude to enhance the learning ability of the subnetwork:

where ⊙ is the Hadamard product, OF(·) denotes two 1 × 1 convolution layers, GL and GH denote the low frequency and high frequency components, respectively. Next, the two components are processed with convolution layers separately in the spatial domain and fused:

where GF ∈ RHs×Ws×Cs is the overall frequency features, OS denotes a residual block with 3 × 3 convolution layers and Conv is the 1 × 1 convolution. Besides, we also adopt a residual block to process the spatial branch. For the fusion of the two branches, We follow [Chen et al., 2021b] to adopt the Dual Attention (DA) module [Fu et al., 2019]. DA includes both channel attention and position attention. It can effectively fuse two branches. The process can be written as:

Then, we transform Go ∈ RHs×Ws×1 with bilinear upsampling into the final predicted mask Gout ∈ RH×W×1 . For the detector, we apply the ConvGeM proposed by MVSSNet++ [Dong et al., 2023], which can convert localization results Gout into detection prediction Dout. ConvGeM strikes a good balance between detection and localization through a decayed skip connection. Thus, we use ConvGem to obtain a more accurate detection result:

3.5 Optimization

Following most studies [Chen et al., 2021b; Salloum et al., 2018; Wang et al., 2022b], we also employ the edge supervision. However, this is not the focus of this work, so we have used some common methods. Following [Chen et al., 2021b], we use the Sobel layer and the residual block to obtain the edge prediction Ge ∈ RHe×We×1 in a shallow-to-deep manner. For edge loss, the ground-truth edges E ∈ RH×W×1 is downsampled to a smaller size E′ ∈ RHe×We×1 to match Ge. This strategy outperforms upsampling Ge in terms of computational cost and performance. The overall loss function can be written as:

where L1 and L3 denote the Dice loss [Chen et al., 2021b], L2 is BCE loss, y is a label that represents the authenticity of the image, Y ∈ RH×W×1 is the ground-truth mask, and α, β are the hyperparameters to balance the loss function. In practice, α is set as 0.60 and β is set as 0.2. Note that authentic images are only used to compute L2.

4 Experiments

4.1 Experimental Setup

Pre-training Data We create a sizable image tampering dataset and use it to pre-train our model. This dataset includes three categories: 1) splicing, 2) copy-move, and 3) removal. We ensure that the training and test datasets are dissimilar. Testing Datasets Following [Liu et al., 2022; Wang et al., 2022a], we evaluate our model on CASIA [Dong et al., 2013], Coverage [Wen et al., 2016], Columbia [Hsu and Chang, 2006], NIST16 [Guan et al., 2019] and IMD20 [Novozamsky et al., 2020]. Specifically, IMD20 collects real-life manipulated images from Internet. We apply the same training/testing splits as [Hu et al., 2020; Wang et al., 2022a] to fine-tune our model for fair comparisons.

4.2 Image Forgery Localization

Following SPAN [Hu et al., 2020], our model is compared with other state-of-the-art tampering localization methods under two settings: 1) training on the synthetic dataset and evaluating on the full test datasets, and 2) fine-tuning the pretrained model on the training split of test datasets and evaluating on their test split. The pre-trained model demonstrates generalizability, while the fine-tuned model shows local performance after reducing domain discrepancy.

Pre-trained Model Table 1a shows the localization performance of pre-trained models for different methods on five datasets under pixel-level AUC. We compare our model F2DNet with MantraNet [Wu et al., 2019], SPAN [Hu et al., 2020], PSCCNet [Liu et al., 2022], ObjectFormer [Wang et al., 2022a], TANet [Shi et al., 2023], and HiFi IFDL [Guo et al., 2023] when evaluating pre-trained models. The pretrained F2D-Net achieves the best localization performance on Coverage, CASIA, and NIST16, IMD20 and ranks the second on Columbia. Especially, F2D-Net achieves 94.2 % on the copy-move dataset COVER, whose image forgery regions are indistinguishable from the background. This validates our model owns the superior ability to capture tampering traces. We fail to achieve the best performance on Columbia, despite surpassing TANet 0.2 % under AUC. We contend that the explanation may be that the distribution of their synthesized training data closely resembles that of the Columbia dataset. This is further supported by the results in Table 1b, which show that F2D-Net performs better than TANet in terms of both AUC and F1 scores. Furthermore, it is worth pointing out F2D-Net achieves decent results with less pre-training data.

Fine-tuned Model The network weights of the pretrained model are used to initiate the fine-tuned models that will be trained on the training split of Coverage, CASIA, and NIST16 datasets, respectively. We evaluate the fine-tuned models of different methods in Table 1b. As for AUC and F1, our model achieves significant performance gains. This validates that F2D-Net could precisely capture subtle tampering traces by phase leaning and adaptive low-high frequency learning.

4.3 Image Forgery Detection

To avoid false alarms, we engage in forgery detection tasks. Following the ObjectFormer [Wang et al., 2022a], we conduct experimental comparisons on the CASIA-D dataset introduced by [Liu et al., 2022]. As shown in Table 1c, our method delivers exceptional detection performance, with an AUC of 99.73% and an F1 score of 97.52%. Our method learns to differentiate and conquer forgery features in the frequency domain, which allows for the distinct separation of forged images from authentic ones.

4.4 Robustness Evaluation

To analyze the robustness of F2D-Net for localization, we follow the distortion settings in [Wang et al., 2022a] to degrade the raw forged images from NIST16. These distortions types include resizing images to different scales (Resize), applying Gaussian blur with a kernel size k (GaussianBlur), adding Gaussian noise with a standard deviation σ (GaussianNoise), and performing JPEG compression with a quality factor q (JPEGCompress). We compare the forgery localization performance (AUC scores) of our pretrained models with SPAN[Hu et al., 2020] and ObjectFormer on these corrupted data, and report the results in Table 2. F2D-Net demonstrates better robustness against various distortion techniques. It is worth noting that JPEG compression is commonly performed when uploading images to social media. Our model outperforms others on compressed images.

4.5 Ablation Analysis

In this section, we conduct experiments to demonstrate the effectiveness of our method F2D-Net. More ablation studies are provided in supplementary materials. The F2D-Net contains two key components: spectral decomposition subnetwork with the phase-emphasized interaction (PEI) block and frequency separation subnetwork. The PEI block comprises two designs: one is an enhanced learning dedicated to phase spectra, termed as the additional learning of phase (ALP), and the other is the space-frequency dual cross attention (SFDCA) used for spatial-frequency interaction. The frequency separation subnetwork (FSSN) is designed to decompose the tampering traces into low and high frequencies. To evaluate the effectiveness of ALP, SFDCA and FSSN, we remove them separately from F2D-Net and evaluate the forgery localization performance on CASIA and NIST16 datasets.

Table 3 presents the quantitative outcomes. The baseline denotes that we just use ResNet-50 and ResBlock. It can be observed that without FSSN, the AUC scores decrease by 4.4 % on CASIA and 2.5 % on NIST16, while without SFDCA, the AUC scores decrease by 5.9 % on CASIA and 4.1 % on NIST16. Moreover, when ALP is discarded, significant performance degradation in Table 3, i.e., 16.5 % in terms of AUC and 24.6 % in terms of F1 on CASIA can be observed.

In Figure 6, we show the different numbers of the phaseemphasized interaction (PEI) block to verify its effect over three datasets. There is an overall incremental trend in the forger location performance as the number of PEI increases, but the performance saturates after reaching a critical point. It is obvious that the setting of 5 is the optimal solution.

4.6 Visualization Results

Qualitative results. We provide predicted forgery masks of different methods in Figure 5. Since the source code of ObjectFormer [Wang et al., 2022a] is not available, their predictions are not available. It benefits from the ability of our model to capture subtle tampering traces and decompose them into low and high frequencies.

Visualization of PEI. To verify the effect of the phaseemphasized interaction (PEI) block, we show the feature visualization of different representations in the PEI block in Figure 7. As can be seen, since features after interaction can obtain complementary representations from each other, f ′f1 obtains more spatial information and the details in f ′s1 are enhanced. It benefits for these two branches to obtain more representational features. Besides, we show the change of features before and after PEI in Figure 8. It is clear that PEI enforces the forgery features learning.

Visualization of FSSN. To verify the usefulness of the frequency separation subnetwork (FSSN), the change of features before and after FSSN is shown in Figure 8. The results demonstrate that the FSSN can effectively refine forgery localization and prevents false alarms, thus helping our model to perform well.

5 Conclusion

In this paper, we explore image forgery from a frequency perspective and propose a novel cascade frequency learning network for Image forgery detection and localization. In detail, we first adopt deep Fourier transform and introduce a phaseemphasized interaction block to learn the phase information for capturing manipulation traces precisely. Then, we design an adaptive frequency separation subnetwork to decompose the tampering traces into low and high frequencies, realizing forgery localization refinement. Extensive experimental results on several benchmarks demonstrate the effectiveness of the proposed algorithm.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (NSFC) under Grants 62225207, 62436008, 62422609, 62276243, and 62476260, and by the Fundamental Research Funds for the Central Universities under Grant WK2100000057.