Unified Physical-Digital Face Attack Detection

1 Introduction

Face recognition (FR) system [Wan et al., 2023] has been widely used in face unlocking, face payment, and video surveillance. It can face a diverse set of attacks: (1) Physical attacks (PAs), i.e. print-attack, replay-attack and maskattack [Liu et al., 2022b; Fang et al., 2023]; and (2) Digital attacks (DAs), i.e. Face2Face [Thies et al., 2016], FaceSwap [Ding et al., 2020], Deepfakes, and NeuralTextures [Thies et al., 2019]. Both Physical Attack Detection (PAD) [Liu et al., 2018; Liu et al., 2023b; Liu and Liang, 2022; Liu et al., 2021b; Liu et al., 2022a] and Digital Attack Detection (DAD) [Dang et al., 2020; Zhao et al., 2021] are still being studied by related works as two independent tasks. As shown in Fig. 1(a), this will lead to the training of both physical and digital attack detection models and their deployment, requiring large computing resources.

Two main reasons hinder the unification of detecting physical-digital attacks: (1) Lack of the physical-digital dataset. Although two datasets (namely GrandFake [Deb et al., 2023] and JFSFDB [Yu et al., 2022]) are proposed to tackle this problem, they only simply merge PAs and DAs datasets. They cannot guarantee each ID covers the real face and all attack types, which would lead the model to learn the liveness-irrelevant signals such as face ID, domainspecific information, and background. Therefore, the lack of a physical-digital dataset with ID consistency limits the research on unified detection algorithms. (2) Distinct intraclass variances. Although both physical and digital attacks are classified as fake in the solution space of the face antispoofing problem, the vast differences between these two types of attacks increase the intra-class distances. Most of the existing attack detection methods [Liu et al., 2018; Yu et al., 2020b] are proposed for a specific attack, thus overfitting to a certain attack and failing to learn a compact feature space to detect both attacks simultaneously.

To solve the first issue, we collect and release a dataset combining physical and digital attacks, named UniAttackData, which contains 2 physical and 12 digital attacks for each of 1, 800 subjects, with a total of 28, 706 videos. Compared to GrandFake and JFSFDB datasets, the UniAttackData provides four advantages: (1) The most complete attack types for each face ID. Our dataset constructs physicaldigital attacks for each face ID, rather than simply merging existing PAs and DAs datasets. (2) The most advanced forgery method. For digital attacks, we select the most advanced digital forgery methods in the past three years. For physical attacks, we consider printing attacks and video replay attacks in various environments and physical media. (3) The most amount of images. The proposed dataset is the largest one in terms of the number of images, which is more than 3.2× boosted compared to the previous face antispoofing dataset like GrandFake. (4) The most diverse evaluation protocols. Beyond the within-modal evaluation protocols, we also provide the cross-attack evaluation protocols in our dataset, in which algorithms trained in one attack category are evaluated in other attack categories. For the second issue, we propose a unified detection framework, named UniAttackDetection, which has three main modules: (1) Introducing textual information to depict visual concepts. Learning joint and category-specific knowledge features by constructing teacher prompts and student prompts, respectively. (2) Enhanced learning of complete feature space. We propose unified knowledge mining loss to learn complete and tight feature spaces using text-to-text optimization. (3) Sample-level visual text interaction. We map learnable text tokens to the visual embedding space and understand samplelevel semantics through multi-modal prompt learning. To sum up, the main contributions of this paper are summarized as follows:

• We propose a dataset that combines physical and digital attacks called UniAttackData. To the best of our knowledge, this is the first unified attack dataset with guaranteed ID consistency.
• We propose a unified attack detection framework based on the vision language model, named UniAttackDetec-
• tion. It adaptively learns a tight and complete feature space with the help of extensive visual concepts and rich semantic information in text prompts.
• We conducted extensive experiments on UniAttackData and three existing attack datasets. The results show the superiority of our approach in the task of unified face attack detection.

2 RelatedWork

2.1 Physical and Digital Attack Detection

PAD technology aims to identify whether the face collected by sensors comes from a live face or is a presentation attack. The most advanced algorithms [Liu et al., 2018; Yu et al., 2020a; Liu et al., 2023a] are based on facial depth to determine authenticity. However, their performance severely deteriorates in unknown domains. At present, the generalization of FAS algorithm is increasingly becoming an important evaluation indicator. Digital attack detection aims to distinguish authentic facial images from digitally manipulated facial artifacts. In initial studies [Rossler et al., 2019], image classification backbones were employed to extract features from isolated facial images, facilitating binary classification. With the increasing visual realism of forged faces, recent efforts [Zhao et al., 2021] focus on identifying more reliable forgery patterns, including noise statistics, local textures, and frequency information. Several other works [Zi et al., 2020] have integrated attention mechanisms to bolster the differentiation between live and forged images. Similarly, these methods still have the problem of insufficient generalization ability, that is, their performance on cross-data is weak.

2.2 Physical-Digital Attack Detection

In a recently published paper, Yu et al [Yu et al., 2022] established the first joint benchmark for face fraud and forgery detection and combined visual appearance and physiological rPPG signals to alleviate the generalization problem. Debayan Deb [Deb et al., 2023] et al. classified all 25 attack types documented in the literature and proposed a method to distinguish between real identities and various attacks using a multi-task learning framework and k-means enhancement techniques. However, none of these works have investigated unified attack detection based on ID consistency. In this paper, we present the first unified attack dataset with ID consistency and a robust unified attack detection algorithm.

3 UniAttackData Dataset

3.1 Acquisition Detail

As depicted in Tab. 1, our UniAttackData is an extension of CASIA-SURF CeFA [Liu et al., 2021a] through digital forgery, which includes 1, 800 subjects from 3 ethnicities (e.g., African, East Asian and Central Asian), and 2 types of physical attacks (e.g., Print and Replay). For each subject, we forge 12 types of digital attacks as follows: (1) Pairing each subject video with a video of others as the reference video. (2) For digital forging, we employ the latest 6 digital editing algorithms and 6 adversarial algorithms on each live video, as detailed in Tab. 1. (3) To selectively forge only the facial region while preserving the background, our process initiates face detection on every frame of the video. Subsequently, we digitally manipulate solely the facial area and seamlessly integrate the background onto the altered face. In overview, as depicted in Fig. 2, our UniAttackData contains live faces from three ethnicities, two types of print attacks in distinct environments, one playback attack, and 6 digital editing attacks and 6 adversarial attacks.

3.2 Advantages of UniAttackData

In comparison to previously datasets, such as GrandFake [Deb et al., 2023] and JFSFDB [Yu et al., 2022], the UniAttackData has the following advantages: Advantage 1. Each ID contains a complete set of attack types. Because we conduct comprehensive physical and digital attacks on each live face, each ID contains a complete set of attack types. On the other hand, GrandFake and JFSFDB merge existing physical and digital attack datasets through integration, resulting in incomplete attack types for each ID, which easily leads to overfitting of the model to identity information. Advantage 2. Incorporation of the most advanced and comprehensive attack methods. Our UniAttackData contains 6 editing attacks and 6 adversarial attacks, utilizing algorithms developed from 2020 onwards for digital attack sample production. In contrast, JFSFDB only contains four types of digital editing attacks, and GrandFake uses algorithms predating 2020 for digital forgery. Advantage 3. The largest joint physical and digital attack dataset. As illustrated in Tab. 1, UniAttackData stands out as the dataset with the highest video count. Statistically, it comprises a total of 28, 706 videos, encompassing 1, 800 videos featuring live faces, 5, 400 videos showcasing physical attacks, and 21, 506 videos with digital attacks.

3.3 Protocols and Statistics

We define two protocols for UniAttackData. (1) Protocol 1 aims to evaluate under the unified attack detection task. As shown in Tab. 2, the training, validation, and test sets contain live faces and all attacks. (2) Protocol 2 evaluates the generalization to "unseen" attack types. The large differences and unpredictability between physical-digital attacks pose a challenge to the portability of the algorithms. In this paper, we use the "leave-one-type-out testing" approach to divide Protocol 2 into two sub-protocols, where the test set for each self-subprotocol is an unseen attack type. As shown in Tab 2, the test set of protocol 2.1 contains only physical attacks that have not been seen in the training and development sets, and the test set of protocol 2.2 contains only digital attacks that have not been seen in the training and development set.

4 ProposedMethod

4.1 Overview

Our UniAttackDetection framework is based on CLIP [Radford et al., 2021]. As illustrated in Fig. 3, the backbone network incorporates three key components: (1) The TeacherStudent Prompt (TSP) module, designed to extract unified and specific knowledge features by encoding both teacher prompts and student prompts, respectively; (2) The Unified Knowledge Mining (UKM) module, leveraging the Unified Knowledge Mining (UKM) loss to steer the student features, and facilitating the exploration of a tightly complete feature space. (3) The Sample-Level Prompt Interaction (SLPI) module, which maps learnable student prompts to visual modality prompts through the interaction projector. Subsequently, the language modality prompts obtained are linked to the original visual modality embeddings, and channel interactions take place during multi-modal prompt learning to extract samplelevel semantics.

4.2 Teacher-Student Prompt

Teacher Prompt. Traditional FAS methods [Wang et al., 2022] focus on extracting features commonly present in each category of images, such as facial contours and shapes. These features are then utilized to learn generic classification knowledge. We introduce this concept into the natural language field by utilizing template prompts to guide the extraction of unified knowledge in physical-digital attacks, which is based on the following fact: The textual descriptions for each category can be effectively represented using a consistent hard template, implying a shared semantic feature space among them. Specifically, given a set of categories consisting of unified class names unifiedc, c ∈ {liveface, spoofface}, the teacher prompt group is then obtained by interconnecting the class name and the manually designed template, i.e., tc = a photo of a {unifiedc}. This group of prompts is passed through CLIP's text encoder GT (·) to compute the unified knowledge features of the text ftc = GT (tc), ftc ∈ Rcu×d , where d is the dimension of CLIP. In this paper, we design multi-group artificial templates to fix discrete teacher anchors in the complete feature space. The multi-group teacher prompts are accumulated over the group dimension g and finally encoded as f g tc ∈ Rcu×g×d .

Student Prompt. In contrast to conventional single-attack detection tasks, the Unified Attack Detection (UAD) task presents substantial gaps within the same classes. To mitigate this problem, we construct a group of student prompts. Student prompts are learned by minimizing the classification error on a training set consisting of specific classes, and thus student prompts possess strong learning abilities in specific classes. Specifically, we prefix each specific class specificc, c ∈ {realface, digitalattack, physicalattack} with a series of learnable vectors pn ∈ Rd , n ∈ 1, 2, 3, ....,N to get the student prompt. Then the student feature fsc = GT (sc) for learning specific knowledge is obtained, where fsc ∈ Rcs×d . To use the learned continuous tight features for binary classification, a lightweight head H : Rcs×d → Rcu×d is adopted to map specific knowledge to a generic classification space.

4.3 Unified Knowledge Mining

To enhance the model's exploration of the complete feature space, we design the Unified Knowledge Mining (UKM) loss as follows:

This loss ensures the student features are sufficiently close to each teacher anchor in the unified knowledge feature space, yielding the following advantages: (1) The student prompt is no longer over-fitted with a specific knowledge distribution through the guidance of teacher anchors, and thus gains strong generalization capabilities. (2) Taking the learnable prompt as a carrier, the unlearnable unified feature space is fully mined to learn the tight and complete feature space.

Not only that, VLMs pre-trained on large-scale image-text pairs encapsulate abundant statistical common knowledge, which leads to the inclusion of potential semantic associations among text features extracted based on these models. Therefore, to capture the correlation between unified and specific knowledge features, we employ a fusion block to extract fusion features for multiple groups of teacher prompts and student prompts. Specifically, we concatenate multiple groups of teacher features f g tc and student features into complete features fcom ∈ Rcu×(g+1)×d in the dimension of g. The fusion block consists of a self-attentive encoder Att(·) and a Multi-Layer Perceptron (MLP) MLPfusion(·), where the self-attention encoder is given by the following equation:

Where WQ, WK, and WV are the weight matrix of the query, key, and value, respectively, and dk is the dimension of the model. Finally, the modulated fusion knowledge feature ffusion is obtained by:

4.4 Sample Level Prompt Interaction

In recent FAS work [Zhou et al., 2023], learned sample-level semantics of visual pictures contribute to enhancing model robustness. Therefore, in VLMs, we deem it essential to establish a multi-modal interaction between picture instance perception features and student prompts. Student prompts can dynamically understand the sample-level semantics in visual pictures and thus learn comprehensive adaptive features. In turn, the visual modality will be guided by the shared prompts to deeply mine the category-related picture features. Inspired by MaPLe [Khattak et al., 2023], we design an interaction projector p = Linear(·). This interaction projector is implemented as a linear layer that maps the dimensions dp of student prompt into the dv of visual embeddings. CLIP's text encoder takes the learnable token [p1][p2][p3] . . . [pn] projected into word embeddings [P] ∈ Rn×dp . The interaction projector then acts as a bridge to map the student prompt dimension word embeddings [P ] into visual dimension embeddings [VP ], where the projector is expressed by a linear layer. The overall formulation is as follows:

At the same time, the input image is encoded by the CLIP's patch embed layer as patch embeddings [VE] ∈ ∈ Rm×dv , where m is the patch dimension of the CLIP. Then, [VE] and [VP ] are concatenated as the final visual embedding V = [VE, VP ], where [·, ·] refers to the concatenation operation. Finally, the visual embedding is passed through the CLIP image encoder GV to get the visual features fv = Gv(V ). During the training process, we compute the cosine similarity between the image features and the labeled features to get the prediction probability of the image for each category and finally use the cross-entropy loss LCLS for binary class supervision. Together with the Unified Knowledge Mining (UKM) loss, the final objective is defined as:

where λ is a hyper-parameter to trade-off between two losses.

5 Experiments

5.1 Experimental Setting

Datasets. To evaluate the performance of the proposed method and existing approaches, we employ four datasets for face forgery detection, i.e., Our proposed UniAttackData, FaceForensics++ (FF++) [Rossler et al., 2019], OULUNPU [Boulkenafet et al., 2017] and JFSFDB [Yu et al., 2022]. Our approach first demonstrates the superiority of our method on UniAttackData. Subsequently, we conducted sufficient experiments on other UAD datasets, such as the OULU-FF data protocol composed of OULU-NPU and FF++, and the JFSFDB dataset, to further demonstrate the generalization capability of our approach. Finally, our ablation study demonstrates the effectiveness of each component of our approach.

Evaluation metrics. For a comprehensive measure of the algorithm's UAD performance, we adopt the common metrics used in both physical forgery detection and digital forgery detection work. All experiments were conducted using average classification error rate ACER, overall detection accuracy ACC, the area under the curve (AUC), and equivalent error rate (EER) for performance evaluation. ACER and ACC on each test set are determined by the performance thresholds on the development set.

Implementation Details. To better demonstrate the effectiveness of our method, we consider a series of existing competitors in the field of face anti-spoofing and base network backbone. The considered comparison methods are the ResNet50, ViT-B/16, FFD [Dang et al., 2020], CDCN [Yu et al., 2020b] and the Auxiliary(Depth) [Liu et al., 2018]. In addition, we incorporated our baseline approach into the comparative analysis. Notably, the student prompts for the baseline method were constructed through the unified class. In this way, we validate the superiority of our overall framework and the effectiveness of learning a continuous and compact specific class space.

5.2 Experiments on Proposed UniAttackData

Experiments on Protocol 1. In protocol 1 of UniAttackData, the data distribution is relatively similar across sets. The training, development, and test sets all contain both physical and digital attacks. This protocol is suitable for evaluating the performance of the algorithms in UAD tasks. We present the performance results for commonly used backbone networks, networks for physical attack detection, and networks for digital attack detection. As shown in Tab 3, our algorithm outperforms others in all four metrics-ACER, ACC, AUC, and EER. This proves the superiority of our method for unified attack detection.

Experiments on Protocol 2. In protocol 2 of UniAttackData, the test set comprises attack forms that are "unseen" in the training or validation sets, aiming to assess the algorithm's generalizability. As shown in Tab 3, an interesting observation emerges: the performance of single-attack detection networks FFD, CDCN, and Auxiliary largely degrades on Protocol 2, and almost all of their four metrics are lower than classic backbone ViT and ResNet, which exposes the flaw that the single-attack detector is over-fitting specific attacks. On the other hand, our method ranks first in performance on all metrics, which proves that our method not only learns unified knowledge of the UAD task but also captures a compact and continuous feature space for attack categories.

5.3 Experiments on Other UADDatasets

To verify the superiority of our proposed dataset and algorithm in more detail. We have done extensive experiments on other UAD datasets. As shown in Tab 4, we obtained the best performance for all metrics on the previously proposed JFSFDB dataset. Furthermore, we combine the training, validation, and testing sets provided by FF++ [Rossler et al., 2019] and OULU-NPU [Boulkenafet et al., 2017] to form a unified attack data protocol named OULU-FF. Our algorithm also outperformed competing methods on the OULU-FF protocol. This validates the ability of our algorithm to detect unified attacks. Notably, we observed a significant decline in the performance of all evaluated methods on both joint attack datasets in comparison to our proposed dataset. This finding underscores the challenge algorithms face in discerning spoofing traces without ID consistency.

5.4 Ablation Experiment

Effectiveness of Different Components. To verify the superiority of our UniAttackDetection as well as the contributions of each component, multiple incomplete models are built up by controlling different variables. All results are measured in the same manner, as shown in Tab 5. First, to verify the validity of student prompts and teacher prompts, we conducted experiments without student prompts or teacher prompts, respectively. The experiments showed that both unified and specific knowledge play a positive role in the UAD

task. In addition, to validate the effectiveness of the Unified Knowledge Mining (UKM) module, we conducted experiments with UniAttackDetection w/o LUKM. Specifically, we directly let the unified knowledge features and the specific knowledge features be connected as the final text features. The experiments demonstrate that the UKMmodule helps the model learn unified knowledge and enhances the generalization to different attacks. Then, to demonstrate the importance of the Sample-Level Prompt Interaction (SPLI) module, we conducted experiments without adding visual prompts. The quantitative results show that multi-modal prompt learning facilitates improved learning of instance-level features.

Effectiveness of Teacher Prompts. To validate the effect of teacher prompts on experimental performance, we incrementally introduced new templates into the teacher prompts group. Tab 6 shows the specific language descriptions covering the real and spoof categories. Fig 4 depicts the variation in experimental performance as we systematically increased the number of teacher prompt groups from one to eight. Examination of the figure reveals that, with the exception of the AUC (Area Under the Curve) metric, there is a discernible trend where the performance across the other three metrics initially improves with the addition of more prompts, only to subsequently decline. All four experimental metrics obtain the best results when six templates are used as teacher prompts. This proves that learning the complete feature space through multiple sets of teacher anchors is beneficial for the UAD task, but too many teacher anchors may instead hinder the adaptive learning ability of student prompts.

5.5 Visualization Analysis

Here, we visualize the distribution of features learned by the baseline model ResNet50 and our method on UniAttackData's P1 and the previously mentioned UAD data protocol, respectively. As depicted in Fig 5, our study reveals two significant findings: (1) ID Consistency Impact. Both the base model ResNet50 and our proposed method exhibit limitations in classifying datasets lacking ID consistency assurance. The feature space displays a mixing of lives and attacks across various regions. In contrast, on the proposed dataset, both methods distinctly segregate lives from attacks. This underscores the critical role of ID consistency, enabling the model to concentrate on acquiring deception features rather than being influenced by ID-related noise features. (2) Enhanced Category Feature Space. In comparison to the baseline model, our approach demonstrates the capacity to learn a closely connected category feature space. This highlights our proficiency in acquiring category-specific knowledge. Moreover, our method successfully establishes a distinct category boundary on both datasets, affirming its effectiveness in extracting unified knowledge.

6 Conclusion

In this paper, we proposed a dataset that combines physical and digital attacks, called UniAttackData. This is the first unified attack dataset with guaranteed ID consistency. In addition, we proposed a unified detection framework based on CLIP, namely UniAttackDetection. The method introduces language information into the UAD task and substantially improves the performance of unified attack detection through multi-modal prompt learning. Finally, we conduct comprehensive experiments on UniAttackData and three other datasets to verify the importance of the datasets for the attack detection task and the effectiveness of the proposed method.

Acknowledgments

This work was supported by the National Key Research and Development Plan under Grant 2021YFF0602105, Beijing Natural Science Foundation JQ23016, the Science and Technology Development Fund of Macau Project 0123/2022/A3, and 0070/2020/AMJ, CCF-Zhipu AI Large Model OF 202219 and the Chinese National Natural Science Foundation Project U23B2054, 62276254, and the InnoHK program.

Contribution Statement

Hao Fang and Ajian Liu are equally contributed in this paper. Jun Wan is the Corresponding Author of the paper.