Discrepancy-Guided Reconstruction Learning for Image Forgery Detection

1 Introduction

Advanced image co-editing and synthesis methods make it cushy for people to tamper with images [Zhu et al., 2020; Rombach et al., 2022]. For example, objects and external properties of these objects in a given image can be completely interpolated via a few texts [Kawar et al., 2022]. Although these progressive methods can increase the diversity and interest of images, on the other hand, they cause a new problem that people's confidence in the information expressed in images is reduced [Cao et al., 2022; Fei et al., 2022]. Besides, tampered images may also be used in some malicious occasions (e.g., fake news and deliberate slanders), thus bringing potential social harms [Hu et al., 2021; Zhang et al., 2022a; Sun et al., 2022b; Li et al., 2020a]. Therefore, exploring effective image forgery detection methods is urgent.

∗Corresponding author: Dong Zhang.

In recent years, thanks to the immense progress of image processing technologies based on the deep learning mechanism [He et al., 2016; Zhang et al., 2020], ample semantic features greatly improve the recognition accuracy of image forgery detection [Wang and Deng, 2021; Zhuang et al., 2022] on both the image-level and the pixel-level [Jiang et al., 2020; Sun et al., 2021; Zhao et al., 2021]. However, offthe-shelf deep learning methods cannot achieve satisfactory results in the face of some challenging forgery cases (e.g., unusual tampering areas and marginal tampering clues). To address this problem and improve the accuracy, some recent approaches use specific operators (e.g., BayerConv [Chen et al., 2021b], Sobel operator [Chen et al., 2021b], and frequency filter [Fei et al., 2022]) to extract discrepant-specific patterns (e.g., noises, textures, and frequencies) as a supplementary for the recognition model. What these methods have commonly is that they use a mainstream backbone network and an additional auxiliary network to extract implicit and explicit semantic features, respectively [Sun et al., 2022b; Wang and Deng, 2021; Zhuang et al., 2022; Chen et al., 2021b; Zhao et al., 2021]. However, these methods usually need to generate temporary supervisions via intermediate feature maps, which are inherently not accurate enough, thus hurting the recognition effectiveness.

What's more, the generalization capacity of existing methods is somewhat limited - due to their overemphasis on the consequence of the explicit discrepant (i.e., the tampered region) features, which limits their latent usage scope. To be specific, only learning some specific types of tampering patterns is far from pragmatism, because we cannot suppose tampering manners [Cao et al., 2022; Yoshihashi et al., 2019; Chen et al., 2021a]. To improve the generalization capacity, it is helpful to learn a set of compact visual patterns, which inherently contain some general image properties, e.g., the concurrent local textures, the consistent regional resolutions, and the continuous bright changes [Robert et al., 2018; Cao et al., 2022; Yoshihashi et al., 2019]. For image forgery detection, to achieve this goal, some work demonstrated that image reconstruction is an effective approach [Wang et al., 2022b; Li et al., 2022b]. The reconstructed output has rich compact patterns and suppresses local forgery regions. However, the existing methods are usually equipped with a single reconstruction head, which suffers from problems of tedious feature representations and inadequate reasoning ability.

In this paper, we propose a novel image forgery detection paradigm, named Discrepancy-Guided Reconstruction Learning (DisGRL), to improve the model learning capacity on both the forgery-sensitive and the genuine compact visual patterns. As illustrated in Figure 1, DisGRL consists of four components: a Discrepancy-Guided Encoder (DisGE), a decoder, a Double-Head Reconstruction (DouHR) module, and a Discrepancy-Aggregation Detector (DisAD) head network.

Specifically, the proposed DisGE (ref. Sec. 3.1) is used to extract forgery-sensitive visual patterns, which consists of two branches: a mainstream backbone branch is used to extract the general semantic features, and an accessorial discrepant external attention branch is used to extract the explicit forgery visual cues. Thereby, DisGE is more suitable for the image forgery detection task than the common backbone networks (e.g., convolutional neural networks and vision transformer). In the decoder, three progressive attention feature selection modules are employed in Fi to connect feature maps from the corresponding encoder network layer, which finally has the same scale as F2. We also propose a DouHR module (ref. Sec. 3.2) based on the decoder, which can enhance the genuine compact visual patterns in two separate granular spaces via an image reconstruction manner. To be specific, in the DouHR module, an Attention-guidance Feature Selection (AFS) procedure and a Similarity Aggregation Module (SAM) are used to extract the vision-based and the reasoningbased genuine compact visual patterns, respectively. Based on the DouHR, we further introduce a DisAD head network (ref. Sec. 3.3) for image forgery classification, which can aggregate the obtained genuine compact visual patterns via a Reconstruction-guidance Feature Aggregation (RFA) module, resulting in an improved forgery detection capability on unknown patterns. Therefore, compared to the existing methods that only focus on these discrepant-specific patterns, our proposed DisGRL has a stronger generalization capability. To demonstrate the superiority of DisGRL, extensive experiments are carried out on four commonly used yet challenging face forgery detection datasets. Results validate that our DisGRL can achieve state-of-the-art performance on both seen and unseen forgeries.

Our contributions are as follows: 1) We propose a novel DisGRL for image forgery detection, which contains three proposed components for learning both forgery-sensitive and genuine compact visual patterns. 2) Extensive experimental results on four challenging datasets validate that DisGRL can achieve state-of-the-art performance against competitors.

2 RelatedWork

Image forgery can be viewed as a game of AI v.s. AI since the majority of detection technologies are based on deep learning. In the past, many efforts have been made to improve the performance of natural/face image forgery detection [Fei et al., 2022; Haliassos et al., 2021; Gu et al., 2022; Zhang et al., 2021]. Extensive work uses a two-branch architecture to mine specific forgery patterns, such as noises or frequency domain features in combination with RGB spatial data, in light of the fact that altered images are getting more visually realistic [Li et al., 2022a; Chen et al., 2021a; Masi et al., 2020; Qian et al., 2020; Li et al., 2021; Wang et al., 2022a]. SOLA [Jia et al., 2022] fuses multimodal features from RGB and high-frequency features extracted by a DCT transformation in an extra branch for more general representations. As a complementary of RGB, the model in [Fei et al., 2022] introduces subtle noise features via learnable high pass filters with anomalies in local regions also performed well in unseen forgeries [Zhang et al., 2020; Yan et al., 2023; Zhang et al., 2022a]. Despite their remarkable performance, their models for obtaining specific forgery patterns only reflect certain aspects of the forgery, which might lead to model bias or sub-optimization.

Recently, some advanced methods are proposed to improve the model generalization capacity such as exploiting contrastive learning to guide the recognition model focus on local content inconsistencies [Sun et al., 2022b; Shi et al., 2023; Zhang et al., 2022b], introducing domain adaptation to alleviate overfitting on a single domain [Rao et al., 2022; Sun et al., 2021; Rao and Ni, 2021], and/or enhance feature representation with an information-theoretic self-information metric for forgery detection [Sun et al., 2022a]. These methods achieve both great performances under intra-dataset (i.e., seen) and cross-domain (i.e., unseen) evaluations. Unlike these methods that explore the local level inconsistencies, our method focuses more on forgery-sensitive and genuine compact visual patterns, which can improve both model's accuracy and generalization.

3 Our Approach

DisGRL is proposed to improve the model learning capacity on both the forgery-sensitive and the genuine compact visual patterns. Our contributions lie in presenting: a DiscrepancyGuided Encoder (DisGE), a Double-Head Reconstruction (DouHR) module, and a Discrepancy-Aggregation Detector (DisAD) head network for image forgery classification. An overview architecture of DisGRL is illustrated in Figure 1. The input is an RGB image X, and the output is a binary predicted label ˆ y, which indicates whether the input image is forged or not. In the following, we detail the implementations of each proposed component.

3.1 Discrepancy-Guided Encoder (DisGE)

To capture forgery-sensitive visual patterns, we propose a DisGE, which consists of two parallel branches, where the mainstream backbone based on Xception network [Chollet, 2017] is used to extract multi-level semantic features, i.e., Fi (i = 1, 2, . . . , 5), and the Discrepancy External Attention (DEA) branch is applied to different level feature to extract explicit discrepant-specific pattern, which are usually subtle and occur in local regions. As shown in Figure 1, features from different Xception layers are combined in a cascaded manner by DEA block. The specific operation of each DEA block's output Di is expressed as:

where Dea(·) and Cat(·) denote each DEA block and feature concatenation along the channel dimension, respectively.

For each DEA block, as shown in Figure 2, we first apply a 3 × 3 convolutional layer on the input feature maps F ∈ RC×H×W with the same channel size C. Then, an adaptive average pooling is used to obtain the pooled features Fd. After that, the differentiated maps can be obtained through D′ = F-Fd to extract the discrepant information. Inspired by [Guo et al., 2023], two 1D convolutions that share the same parameters are further introduced to characterize the global features of the entire map. Concretely, given a differentiated input D′ ∈ RC×H×W , after reshaping and 1D convolution, feature maps are up-sampled four times in channel size. And 1D convolution and a reshape function are applied again to restore the original feature map size. Finally, the output feature map D can be obtained through a 1 × 1 convolution and a residual connection.

3.2 Double-Head Reconstruction (DouHR)

Reconstruction learning has been proven to be beneficial to several image forgery detection works by exploring rich compact visual patterns [Cao et al., 2022; Wang et al., 2022b; Li et al., 2022b]. In this work, we propose DouHR module based on the decoder to enhance the genuine compact visual patterns in two separate granular spaces (i.e., AFS and SAM) via an image reconstruction manner, such that the model can not only learn a rich genuine compact visual pattern but also further suppress the visual representation of the local forgery regions. As shown in Figure 1, besides ˆX1 that is generated by an Attention-guidance Feature Selection (AFS) procedure in extracting the vision-based genuine compact visual patterns via convolutions, we introduce an extra Similarity Aggregation Module (SAM) for extracting the reasoningbased genuine compact visual patterns via secondary reconstruction ˆX2. The DouHR module can be formulated as:

where FAFS3 indicates the output feature maps of the third AFS procedure in the decoder. In DouHR, we adjust the number of channels from the output of the SAM and AFS modules to 3 by applying a 1 × 1 convolution. After that, we use bilinear interpolation to adjust the feature map size to match the input image size.

AFS. In the decoder, three AFS modules receive the output of the previous AFS module and feature maps of the corresponding level in the mainstream backbone as input. For example, the inputs to the third AFS are FAFS2 and F2. In DouHR, AFS receives the FAFS3 and as F1 input. The concatenating operation Cat(·) is first carried out on FAFS3 and F1 in the channel dimension, i.e., ˜F = Cat(FAFS3 ,F1), followed by a depthwise separable convolution fd3 to obtain attention map Aatt with the same shape as input features and suppress the unimportant region of feature information transmitted by decoder output, so that model pays more attention to the genuine compact visual patterns. Finally, a residual connection operation is applied to obtain the output. The above process can be expressed as follows:

where fc3 and σ(·) are the 3 × 3 convolution layer and sigmoid activaion function, respectively. Other AFS procedures are calculated in a similar way.

2

2

Figure 3: Illustration of Similarity Aggregation Module (SAM), which can extract the reasoning-based genuine compact visual patterns.

SAM. To inject detailed global features into high-level semantic features using a global reasoning reconstruction, inspired by [Dong et al., 2021], we introduce non-local operation under graph convolution operation [Lu et al., 2019; Zhang et al., 2022b] to implement SAM. As shown in Figure 3, for the given feature map FAFS3 , we first apply three 1 × 1 convolutions (i.e., Wρ,Wθ, and Wφ) to reduce the channel dimension into 16, and obtain feature maps Fθ, Fφ, which can be expressed as:

For F1, we down-sample it to the same size asWρ(FAFS3 ). Then we apply a Softmax function along the channel dimension and calculate the element-wise multiplication with Fφ for assigning different weights to different pixels and increasing the weight of edge pixels. And an adaptive pooling operation Avp(·) is utilized to reduce the displacement of features. In summary, the processing can be formulated as:

where D(·) and softmax(·) denote the down-sampling and Softmax functions, respectively. After that, the matrix multiplication and Softmax function are used to establish the correlation between Fφ and Fw, which can be expressed as:

The correlation attention map Fcor is multiplied with the feature map Fθ, and the resulting map is fed to the graph convolutional newtwork (GCN). Same to [Dong et al., 2021], reconstructing the graph domain features into the original structural features as follows:

Finally, the reconstructed features G′ are combined with the features Wρ(FAFS3 ) to obtain the output G:

where Wz denotes 1 × 1 convolution.

3.3 Discrepancy-Aggregation Detector (DisAD)

The double-head reconstructed forged images essentially differ from the input forged images in visual appearance [Cao et al., 2022]. To further explore the probable forgery regions within reconstructed images, based on the DouHR module, we further introduce a DisAD head network via two Reconstruction-guidance Feature Aggregation (RFA) modules to aggregate the obtained genuine compact visual patterns (i.e., ˆX1 of AFS and ˆX2 of SAM), resulting in an improved forgery detection capability on unknown patterns (i.e., the greater generalization capability).

As shown in Figure 1, we first calculate the differences between two reconstructed images (i.e., ˆX1 and ˆX2) and the original input image X in discrepancy-aggregation detector. The pixel-level difference masks are expressed as:

where |·| refers to the absolute value function. Then for RFA in Figure 4, given difference masks ˆR1 or ˆR2 and the summation of textural discrepancy information and encoding feature Fe = D4 ⊕ F5, we perform an element-wise multiplication between them with a residual connection and a 3 × 3 convolution to obtain the fused features Fd:

where D denotes down-sampling and fc3 is 3 × 3 convolution. σ is a sigmoid function and ⊕ is element-wise addition. To enhance feature representations, inspired [Wang et al., 2020], we aggregate the features Fd using a channel-wise global average pooling (GAP). Then the channel weight is obtained by the 1D convolution followed by a sigmoid function. Finally, the channel attention is multiplied with the input features Fd to obtain the final output FRFA, i.e.,

where fc1 is 1 × 1 convolution and f1d is 1D convolution.

3.4 Loss Function

DisGRL has two kinds of supervision: the image-level binary classification label based on the cross-entropy loss (i.e., Lcls), and the pixel-level reconstruction learning label. During training, we employ the reconstruction loss (Lr1 and Lr2) [Cao et al., 2022] between real images and their two reconstructed images. Besides, a metric-learning loss (i.e., Lm) [Cao et al., 2022] based on F5 is used to enhance the reconstruction difference to facilitate model learning. Thus, the total loss can be expressed as:

where λ is a trade-off hyper-parameter for loss balance.

4 Experiments

4.1 Experimental Settings

Datasets. To facilitate a fair result comparison with stateof-the-art methods, we conducted experiments on four fundamental yet challenging face forgery datasets, including FaceForensics++ (FF++) [R¨ossler et al., 2019], Celeb-DF [Li et al., 2020b], WLD [Zi et al., 2020], and DFDC [Dolhansky et al., 2019]. Due to the page limit, details of each dataset are given in supplementary materials.

Implementation Details. We implemented our model on the PyTorch framework and used Xception [Chollet, 2017] pretrained on ImageNet [Deng et al., 2009] as our mainstream backbone. The input face images are resized into 299 × 299 and augmented by random horizontal flipping. In the training phase, the batch size is set to 32, and Adam optimizer [Kingma and Ba, 2015] with learning rate 1e-4, and weight decay 1e-5 are adopted to optimize the model. The step learning rate strategy with a gamma of 0.5 is utilized to adjust the learning rate. Following [Cao et al., 2022], λ1, λ2, and λ3 in Eq. (12) are empirically set to 0.1.

Evaluation Metrics. In this work, we reported results on the commonly used evaluation metrics [Cao et al., 2022; Sun et al., 2022b; Zhuang et al., 2022], including Accuracy (ACC), Area Under the Curve (AUC), and Equal Error Rate (EER).

4.2 Quantitative Results

To demonstrate the effectiveness of our proposed method, we compare it with the state-of-the-art methods, i.e., Xception [R¨ossler et al., 2019], Two-branch [Masi et al., 2020], SPSL [Liu et al., 2021], RFM [Wang and Deng, 2021], FreqSCL [Li et al., 2021], Add-Net [Zi et al., 2020], F3 -Net [Qian et al., 2020], MAT [Zhao et al., 2021], RECCE [Cao et al., 2022], ITA-SIA [Sun et al., 2022a], Multi-task [Nguyen et al., 2019], MLDG [Li et al., 2018], LTW [Sun et al., 2021], and DCL [Sun et al., 2022b]. For a fair comparison, all experimental results of these methods which we employ for comparisons are either explicitly cited from works or generated by models that are retrained with open-source codes.

Intra-Dataset Evaluation. Table 1 shows result comparisons with our DisGRL against 10 competitors under intradataset evaluations. We can observe that DisGRL consistently outperforms other models on FF++ [R¨ossler et al., 2019], WLD [Zi et al., 2020], and DFDC [Dolhansky et al., 2019]. Especially on the challenging WLD, our method still surpasses the second-best RECCE by 1.25% in terms of AUC. This suggests that when confronted with more identities from real-world scenes, our method owns the superior ability to detect discrepancies between real faces and fake ones. On Celeb-DF [Li et al., 2020b], though ITA-SIA achieves the highest AUC, our DisGRL still achieves comparable results on the other datasets, especially on the low-quality setting of the FF++ (↑ 1.74%). Different from ITA-SIA which introduces a self-information metric to enhance the feature representation, DisGRL produces a more robust representation through double-head reconstruction, which works well in conjunction with single reconstruction for forgery detection.

Cross-Dataset Evaluation. To explore the generalization of our method on unseen datasets compared with recent general face forgery detection methods, we focus on the more challenging cross-dataset evaluation. Table 2 reports the quantitative results by training the models on FF++ (LQ) [R¨ossler et al., 2019] and testing them on Celeb-DF [Li et al., 2020b], WLD [Zi et al., 2020], and DFDC [Dolhansky et al., 2019], accordingly. It can be concluded that our method achieves a certain improvement in generalization ability by taking good advantage of double-head reconstruction structures. In particular, the AUC score of our method on Celeb-DF (↑ 1.32%), WLD (↑ 2.42%), and DFDC (↑ 1.83%) datasets is enhanced when compared with RECCE. Overall, our method promotes the extraction of genuine compact visual patterns and can be generalized to unseen forgeries rather than modeling the pattern of the single forgery techniques.

Cross-Manipulation Evaluation. To further demonstrate the generalization among different manipulated manners, we conduct the fine-grained cross-manipulation evaluation by training a model on one specific method and testing it on all four methods listed in FF++ (LQ). As shown in Table 3, our DisGRL generally outperforms the competitors in most cases, including both intra-manipulation (diagonal of the table) results and cross-manipulation. Specifically, when training on NT and testing on F2F, though MAT is equipped with EfficientNet-b4, our DisGRL based on Xception still outperforms it by a margin of 2.69%. Additionally, a 2.41% performance gain in terms of AUC is achieved by our method compared with RECCE, which illustrates that it is feasible to explore common features of real faces to distinguish real and fake faces. With help of the double-head reconstruction strategy and carefully designed cascaded discrepancy external attention, our method exceeds all other methods in terms of the average AUC of cross-manipulation evaluations.

Multi-Source Manipulation Evaluation. Multi-source manipulation evaluation refers to situations in which the forged techniques utilized for training are not restricted to just one way. Following the LTW [Sun et al., 2021] and DCL [Sun et al., 2022b], we conduct experiments on the low-quality (LQ) version of FF++ [R¨ossler et al., 2019] to demonstrate the practicality of our method in real-world scenarios. As shown in Table 4, we can observe that our DisGRL obtains cutting-edge performance in terms of AUC and ACC on all protocols. In particular, DisGRL outperforms the recent DCL by around 7% in the setting of GID-F2F, proving its durability and ability to ensure generalization under various scenarios.

4.3 Ablation Study

To validate the effectiveness of each component, we designed several ablation experiments on the WildDeepfake dataset in varied configurations with the components added progressively. As shown in Table 5, the setup model variants are as follows: for the baseline model of a), we follow the classic image classification pipeline, i.e., Xception [Chollet, 2017]. b) and c) the encoder-decoder backbone with the introduction of a single-head reconstruction (Rec-1 or Rec-2) learning scheme. d), the encoder-decoder backbone equipped with double-head reconstruction (Rec-1 and Rec-2). For the model of e), we remove the RFA and adopt the element-wise addition to replace it, f) is the proposed DisGRL without DEA, and g) is our DisGRL.

Effectiveness of DouHR. We can observe in Table 5 that the double-head reconstruction learning module performs better than the baseline model of a) and its variant b), c) Baseline + single-head. Therefore, Rec-2, as the complementary information of Rec-1, aims to capture the genuine compact visual pattern of real regions and fake regions, which is beneficial to boost detection performance.

Effectiveness of DisGE. Then the model of e) DisGRL w/o RFA achieves better overall performance compared with the model of d) Baseline + double-head, especially in terms of AUC with 1.37 % performance gains. It verifies that DEA enhances the model's efficiency to mine the forgery-sensitive visual pattern within the instance by cascading shallow and deep features in the encoder to focus on image forgery cues rather than on semantic image content. Therefore, it is feasible to improve the classification learning capabilities of the detector when combined with the integrated representation collected by the decoder, leading to a larger performance increase for variation e) DisGRL w/o RFA.

Effectiveness of DisAD. The comparison between variants d) Baseline + double-head and f) DisGRL w/o DEA in Table 5 can demonstrate the effectiveness of our proposed RFA, which aggregates the obtained genuine compact visual pattern and emphasizes the probably forged regions. And combining all the proposed components can achieve the best performance in terms of ACC and AUC scores.

4.4 Visualizations

Our proposed reconstruction learning aims to preserve more variations by building a double-head reconstruction scheme. To validate its effectiveness, as illustrated in Figure 5, we visualize the outputs of the two reconstructions and the corresponding difference masks between the original input and reconstruction maps. We can observe that the real faces can be well reconstructed with little blurring, while the forged portions of the fake ones cannot be recovered. Difference masks, indicating possible traces of forged areas, further amplify the differences between real and forged faces. Compared with Diff-1, Diff-2 is able to additionally enhance and complement the forged areas in faces. For instance, NT operates around the mouth region and the response in Diff-1 of the corresponding sample is weak around the mouth region, while the value is larger in Diff-2, illustrating the importance and usefulness of an additional head for reconstruction in image forgery detection.

5 Conclusion

In this work, we proposed a novel image forgery detection paradigm, termed DisGRL, to improve the model learning capacity on forgery-sensitive and genuine compact visual patterns. DisGRL primarily consisted of a discrepancy-guided encoder, a decoder, a double-head reconstruction module, and a discrepancy-aggregation detector head network. The advantage of DisGRL was that it can not only encode general semantic features but also enhance the forgery cues of the given image. Experimental results on four widely used face forgery datasets validated the effectiveness of our proposed method against state-of-the-art competitors on both seen and unseen forgeries. DisGRL is a general paradigm, which can be used in general image forgery detection tasks. Therefore, in the future, we will explore how to apply DisGRL to more challenging natural scene datasets in terms of quantity and quality. Besides, exploring how to use DisGRL in the forgery detection of video data is also a promising research direction.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (62276112), the National Natural Science Foundation of China Regional Joint Fund of NSFC (U19A2057), Jilin Province Science and Technology Development Plan Key R&D Project (20230201088GX), and Collaborative Innovation Project of Anhui Universities (GXXT2022-044).