IdentityMask: A Robust Face-Centric Privacy Protection Against Unauthorized Personalization of Diffusion Models

1 Introduction

Text-to-image diffusion models [Song et al., 2020a; Ho et al., 2020; Rombach et al., 2022a; Luo et al., 2023] have significantly advanced the field of AI generation. In particular, personalization techniques [Gal et al., 2022; Ruiz et al., 2023] based on Latent Diffusion Models (LDMs) [Rombach et al., 2022a] have garnered widespread attention. These methods enable creating personalized concepts, such as specific individual identities, via lightweight fine-tuning on a few reference images. However, the misuse of these technologies poses severe risks to the user community. Unauthorized personalized generation of human faces can be exploited to create malicious content or infringe upon portrait rights.

∗Corresponding author.

To proactively prevent such misuse, adversarial examples [Szegedy et al., 2013; Goodfellow et al., 2014] have emerged as a highly effective defense strategy. Unlike passive measures such as watermarking [Wang et al., 2023b; Li et al., 2024b; Guo et al., 2024], which primarily facilitate post-hoc tracing, adversarial examples aim to fundamentally disrupt the personalization process, thereby preventing unauthorized content generation at the source. Recent approaches have focused on disrupting image encoders [Salman et al., 2023; Shan et al., 2023] or employing hybrid strategies [Liang and Wu, 2023]. However, these methods often prove inadequate against robust fine-tuning techniques; as they primarily interfere with the representation layer, they fail to penetrate the model's core generative mechanism. In contrast, diffusion-based defenses [Liang et al., 2023; Xue et al., 2023] target the gradient flow within the model's reverse denoising process. By blocking the model's ability to learn the target concept, these approaches have emerged as a compelling technical path for protecting facial data privacy.

However, current diffusion-based defenses share an important shortcoming. In personalization, identity information is primarily extracted from a few salient facial regions and a limited set of high-signal timesteps. Existing defenses, however, ignore this uneven structure and instead optimize noise uniformly across space and time. This misalignment manifests in some critical technical flaws that constrain efficacy.

First, existing defenses apply arbitrary perturbations across the entire image space. However, personalization models function differently - they rely heavily on Cross-Attention mechanisms [Rombach et al., 2022a] to specifically lock onto facial regions for identity extraction. Consequently, simply maximizing the loss across the whole image fails to divert this precise attention mechanism. As visualized in Figure 1a, the downstream model still successfully anchors on the core facial features, ignoring the dispersed adversarial noise.

Second, the optimization process is inefficient due to uniform timestep sampling. The fine-tuning of diffusion models is non-linear; the construction of semantic and identity information is highly concentrated within specific high-Signalto-Noise Ratio (SNR) stages. Ignoring this dynamic, current methods sample timesteps uniformly. Consequently, compu-

• (a) Identity-Agnostic Perturbation

(b) Inefficient Timestep Sampling

tational resources are wasted on low-SNR stages that minimally contribute to semantic learning, while the intensity of perturbation remains insufficient during the critical time windows that determine identity encoding (Figure 1b).

Finally, the generated perturbations are structurally fragile against purification. These perturbations cause the adversarial examples to deviate significantly from the natural image manifold (i.e., falling outside the distribution of real images). This distributional anomaly renders protection highly vulnerable: diffusion-based purification methods [Nie et al., 2022; Zhao et al., 2024] can easily nullify defense by projecting the image back onto the data manifold, stripping away adversarial signals without destroying the image content (Figure 1c).

To address these limitations, we propose IdentityMask, a robust framework that shifts the paradigm from blind interference to targeted feature corruption. By simulating the model's identity extraction mechanism across spatiotemporal dimensions, we ensure that the adversarial optimization operates precisely along the critical paths encoding identity features. Specifically, to counter identity-agnostic optimization and inefficient sampling, we introduce a SpatioTemporal Joint Adversarial Attack. In the spatial dimension, distinct from simple mask-based noise addition, we employ a gradient guidance mechanism based on Face-Centric Identity Encoding. This directs the adversarial gradients to explicitly maximize disruption on the semantic features targeted by the model's Cross-Attention mechanism. In the temporal dimension, we implement a Rectified Signal-Aware Sampling strategy, allocating the optimization budget heavily towards timesteps with the highest semantic constructivity based on SNR distributions. This joint optimization ensures that IdentityMask achieves maximum semantic disruption on the model's critical learning path with minimal perturbation cost, effectively blocking the correct encoding of identity features. Second, to resolve fragility against purification, we incorporate Manifold Projection. This strategy maps adversarial signals onto the intrinsic latent manifold, transforming fragile high-frequency noise into structural semantic perturbations that effectively withstand diffusion-based purification. Our contributions are summarized as follows:

• We systematically analyze existing diffusion-based de-

fenses and show that they treat personalization as a uniform noise optimization problem. In contrast, personalization relies on a small set of spatial regions and timesteps with concentrated identity signals, which existing methods fail to exploit.

• We design a framework guided by how personalization models encode identity. Gradient updates are concentrated on semantically relevant facial regions through Face-Centric Identity Encoding, while perturbation budgets are assigned to high-SNR timesteps using Rectified Signal-Aware Sampling.
• We introduce a Manifold Projection strategy that embeds adversarial perturbations into the latent space instead of leaving them as high-frequency artifacts. As a result, the perturbations are less affected by diffusionbased purification.
• We evaluate IdentityMask across multiple datasets and personalization settings. IdentityMask consistently outperforms existing methods across these settings, indicating its suitability for facial data privacy protection.

2 RelatedWorks

2.1 DiffusionModels and Personalization

Diffusion Models. In recent years, diffusion models [Ho et al., 2020; Song et al., 2020a; Rombach et al., 2022a; Ho, 2022; Song and Ermon, 2019; Song et al., 2020b] have made significant advances in the domain of image generation. An influential example is the open-source Stable Diffusion model [ML and AI, 2022; Rombach et al., 2022b], which is based on the Latent Diffusion Model (LDM) framework [Rombach et al., 2022a].The LDM operates not in the high-dimensional pixel space, but in a compressed latent space. It first employs the encoder of a Variational Autoencoder (VAE) [Kingma and Welling, 2013; Radford et al., 2015; Kingma and Welling, 2022] to map an image x from the pixel domain to a lower-dimensional latent representation z. Given that the generative process occurs within this compressed latent space, the model may effectively leverage the learned semantic features to synthesize high-fidelity images.

Personalization. To enable the generation of novel concepts absent from their training data, text-to-image models can be adapted through personalization techniques [Gal et al., 2022; Ruiz et al., 2023; Kumari et al., 2023; Sohn et al., 2023] using only a few example images. These methods primarily fall into two categories. The first approach involves fine-tuning the model's weights. DreamBooth [Ruiz et al., 2023] is a popular technique that fine-tunes the entire UNet to associate a subject with a unique identifier (e.g., "sks"). In light of the need for efficiency, the second approach may optimize a textual embedding while keeping the core model frozen. However, Textual Inversion [Gal et al., 2022] is a lightweight method that learns pseudo-word in text encoder's embedding space to capture concept semantics. Furthermore, Textual Inversion may be widely adopted due to minimal storage footprint, making it a primary target for data poisoning attacks investigated in this work.

2.2 Privacy Protection Against DiffusionModels

Given the powerful personalization capabilities that diffusion models may demonstrate, researchers have developed various significant adversarial defenses to protect privacy. AdvDM [Liang et al., 2023] directly maximizes the noiseprediction loss. To stabilize gradients, it employs Monte Carlo estimation by averaging over multiple timesteps. Subsequent work, SDS [Xue et al., 2023], introduced the concept of Score Distillation Sampling to approximate the gradient without a full backward pass, improving the efficiency of the attack. Given that other works explore different attack surfaces, PhotoGuard [Salman et al., 2023] is considered a form of encoder attack. Instead of manipulating the UNet's loss, it perturbs an image to force its latent representation, as produced by the VAE encoder, to match that of an unrelated target image. Nevertheless, this corrupts the image's core features in latent space. Mist [Liang and Wu, 2023] proposes a hybrid strategy, combining a semantic loss similar to AdvDM with a "textural loss". PID [Li et al., 2024a] achieves promptindependent defense by directly manipulating the visual encoder. Meanwhile, AIF [Li et al., 2025] generates highly stable perturbations via an alternating iterative framework with ensemble learning. In light of these approaches, our work introduce a face-centric method that targets core identity features. Moreover, it enhances optimization efficiency using a Rectified Signal-Aware Sampling strategy.

2.3 Purification and Anti-Purification

A significant challenge to adversarial protections is the diffusion-based purification [Nie et al., 2022; Zhao et al., 2024; Xue and Chen, 2024], which aim to remove perturbations before an image is used. These methods treat adversarial noise as an out-of-distribution artifact that can be cleaned. Seminal works like DiffPure [Nie et al., 2022] use a diffusion model's reverse process to project a perturbed image back to a clean data manifold. GrIDPure [Zhao et al., 2024] later extended this capability to high-resolution images by applying purification in overlapping patches.In response, research has focused on developing robust purification defenses. Moreover, some important methods, such as DiffAttack [Kang et al., 2023], could indicate that directly disrupting the purification pipeline itself appears to be feasible. Others, like Adversarial Content Attack (ACA) [Chen et al., 2023], generate more natural, on-manifold adversarial examples with semantic modifications (e.g., texture, color) that are inherently resistant to being cleaned. Our work contributes to this area by showing how a simple and efficient VAE reconstruction can achieve strong robustness against such purification attacks.

3 Method

3.1 ProblemFormulation

Our method builds upon the Latent Diffusion Model (LDM) framework [Rombach et al., 2022a], which operates in the compressed latent space of a pre-trained Variational Autoencoder (VAE). The core of LDM is a U-Net noise predictor, ϵθ, whose objective is to denoise a noisy latent zt by minimizing the following loss, conditioned on a text embedding Embp:

We craft a visually imperceptible perturbation δ to generate an adversarial example xa = x + δ. Formally, we seek the perturbation δa maximizing the diffusion model's loss:

To ensure imperceptibility, we enforce a constraint on the perturbation magnitude. Specifically, it must stay within an Lpnorm ball of radius ξ (typically setting p = ∞). We solve this problem using Projected Gradient Descent (PGD) [Madry et al., 2017]. Starting from x(0) a = x, each iteration updates the adversarial example via:

where α is the step size, sgn(·) is the sign function.

3.2 Overview

To address the misalignment in existing defenses, we propose IdentityMask, a framework that achieves precise and robust protection (Figure 2). Our method centers on a SpatioTemporal Joint Adversarial Attack that aligns optimization with the model's personalized learning dynamics. Spatially, we utilize Face-Centric Identity Encoding to guide adversarial gradients toward critical semantic features. Temporally, we implement Rectified Signal-Aware Sampling to concentrate the optimization budget on high-SNR stages where identity encoding occurs. Finally, to counter purification, we incorporate Manifold Projection. The detailed algorithmic procedure is provided in the Appendix.

3.3 Stage 1: Face-Centric Identity Encoding

This stage creates a "semantic anchor" that isolates a subject's core facial features. Given an image x and its corresponding face mask m, we compute the masked latent zmasked = E(x) ⊙m′ , where m′ is the downsampled mask.

This isolates the facial features from the background context. We then optimize a learnable embedding Embid, initialized with a text ("a photo of a person face"), to force the U-Net ϵθ to reconstruct the masked facial region. The objective is:

where zt = √ ¯αtzmasked + √1 - ¯αtϵ. Crucially, only Embid is updated during this process. The resulting embedding, Emb∗id , acts as a hyper-specific lure. Using it as guidance forces the U-Net's Cross-Attention to lock intensely onto the facial region. This ensures that adversarial optimization is maximized precisely along critical identity-encoding paths.

3.4 Stage 2: Adversarial Generation via Rectified Signal-Aware Sampling

Conventional methods typically employ uniform sampling (t ∼ U[0, T]), erroneously assuming all diffusion stages contribute equally to generation. This approach ignores the model's Signal-to-Noise Ratio (SNR) dynamics. As visualized in Figure 1b, where we track the gradient norm of the textual embedding. Consequently, it leads to inefficient budget allocation on noise-dominated (t → T) or semantically rigid (t → 0) stages. To address this, we propose RSAS strategy. We formulate p(t) by combining signal retention dynamics with a stability constraint:

where Z is a normalization constant. This distribution balances two competing physical components:

Signal Retention Term S(t). In the diffusion process, the input signal x (and thus the adversarial perturbation δ) is scaled by √ ¯αt. As t → T, the signal is exponentially attenuated by noise (√ ¯αt → 0), rendering gradients negligible for identity encoding. To ensure the targets stages where perturbations remain perceptible to the U-Net, we align sampling with effective signal strength:

Stability Regularization Term R(t). While low timesteps (t → 0) offer high signal strength, the loss landscape there is dominated by high-frequency pixel reconstruction rather than semantic concepts. Gradients in this regime are often unstable (see the left tail of Figure 1b) and fail to induce effective semantic disruption. We thus introduce a linear warm-up to suppress sampling in this rigid window:

where τwarmup is a hyperparameter. Consequently, the optimization bypasses rigid pixel-level residuals, strictly targeting the mid-range timesteps where semantic features remain modifiable.

Optimization Process. At each step k, we first sample t ∼ p(t). Then we update the adversarial example xa using gradients conditioned on the Face-Centric Identity Embedding (Emb∗id ) from Stage 1:

By adhering to p(t), this strategy concentrates optimization on the temporal stages that are most critical for the personalized learning of identity features.

3.5 Anti-Purification viaManifold Projection

Conventional adversarial perturbations could indicate that these critical high-frequency and off-manifold characteristics render them highly susceptible to the significant diffusionbased purification processes, as demonstrated in Figure 1c. Given that purification models function by projecting inputs back onto learned data distributions, the models may effectively remove these unnatural noise patterns. Moreover, we introduce a manifold projection step using a pre-trained VAE to address this limitation. The VAE acts as a filter, removing high-frequency artifacts while embedding perturbations into natural image structure. Nevertheless, this operation is performed via a single forward pass:

However, x′a may align with the natural manifold, appearing benign to purification models. Thus, the method renders perturbations resilient to removal by transforming perturbations from fragile noise into structural semantic features.

4 Experiments

4.1 Setup

Dataset. We conduct experiments on VGGFace2 [Cao et al., 2017] and Celeba-HQ dataset [Karras et al., 2017]. We select 40 distinct identities, partitioning each into reference sets for personalization training and held-out sets for evaluation.

Baseline. We benchmark IdentityMask against existing privacy protection methods, including AdvDM [Liang et al., 2023], SDS+ [Xue et al., 2023], Mist [Liang and Wu, 2023], PhotoGuard [Salman et al., 2023], PID [Li et al., 2024a], AIF [Li et al., 2025], Anti-Dreambooth [Van Le et al., 2023]. Evaluation metric. We assess each method's effectiveness based on two criteria: its ability to protect the subject's identity and its impact on the quality of the generated images. For identity protection, we use the Identity Score Matching

(ISM) [Van Le et al., 2023] to measure the identity similarity between original and generated faces (lower is better). For image quality, we employ the Fr´echet Inception Distance (FID) [Heusel et al., 2017] to evaluate protection efficacy (higher is better) and the CLIP Score [Hessel et al., 2021] to assess semantic consistency (lower is better).

Implementation Details. Our experiments are conducted based on the Stable Diffusion v1.5 model [ML and AI, 2022]. For our proposed IdentityMask, we set the warm-up threshold τwarmup to 250, and the number of iterations for the FaceCentric Identity Encoding to 100. For all adversarial baseline methods, we set the number of attack iterations to 100, with the exception of Anti-Dreambooth [Van Le et al., 2023], for which we adopt its official default settings. The magnitude of the adversarial perturbation is constrained to a maximum of 8/255, and the perturbation step size is set to 1/255.

4.2 Protection Against Textual Inversion

We evaluate our IdentityMask against the widely-used Textual Inversion task.

Quantitative Analysis. The results represent in Table 1. IdentityMask outperforms all baselines on both CelebA-HQ and VGGFace2. The gap is widest with the dslr portrait prompt. Here, our method achieves an ISM score of 0.0793, which appears to be 50.1% lower than that of the closest competing approach. The observed performance gap indicates that our face-centric strategy better concentrates on identityrelevant facial regions, in contrast to the more dispersed perturbations used in prior work.

Qualitative Analysis. Figure 3 aligns with our metric scores. We examine baselines like AdvDM. These methods introduce visible artifacts. However, the core facial structure remains intact. This fact explains why the identity leakage (ISM) stays high. IdentityMask produces a different result. It causes the personalization process to collapse completely. The model generates images without semantic meaning. We see abstract patterns instead of facial features. This result proves our method's effectiveness. We do not merely add surface noise. Instead, we disrupt the identity encoding mechanism.

a dslr portrait of sks-person

Table 2: The "Purification" section shows that standard protections are severely compromised after purification. In contrast, our full method with MP retains significantly more effective protection. (Note: Percentage changes are indicated in parentheses).

4.3 Robustness to Purification Attacks

We evaluate robustness against GridPure [Zhao et al., 2024], a diffusion-based purification method. The results in Table 2 and Figure 4 reflect a significant vulnerability in existing methods. After purification, the protective effects of all baselines are severely diminished. AdvDM loses its protective capability entirely. This failure leads to a sharp 125.5% rise in the ISM score. Visual inspection (Figure 4, "After Purification") confirms this failure, revealing clearly recognizable faces. Crucially, our method without Manifold Projection exhibits similar fragility, highlighting that pure pixel-level perturbations are easily purged. In contrast, IdentityMask, fortified by Manifold Projection, demonstrates exceptional robustness. As shown in Table 2, our method maintains a low ISM of 0.1900 even after GridPure processing, significantly outperforming all other methods. Qualitative results (Figure 4, bottom row) further confirm that the generated images remain semantically destroyed with no recognizable identity. This validates that Manifold Projection transforms perturbations into structural features that purification models perceive as legitimate, preserving protective integrity. Additional experiments using DiffPure [Nie et al., 2022] yield consistent findings and are detailed in the Appendix.

4.4 Generalization to DreamBooth

To assess the broader applicability of our framework, we evaluate its performance against DreamBooth, a signifi- Protected Images (w/o MP)

Before Purification

After Purification

AntiPurification (w/ MP)

Figure 4: Our Manifold Projection (MP) is essential for robust protection against purification. The "Before Purification" row confirms that all methods are effective at disrupting personalization. However, the "After Purification" row reveals that perturbations are easily stripped away by a purification model. The bottom row, "AntiPurification", shows that MP can resist purification, maintaining complete protection of the user's identity.

cantly more challenging paradigm that fine-tunes the entire U-Net weights rather than just embeddings. We benchmark against strong, specialized baselines, including AntiDreamBooth [Van Le et al., 2023] and SimAC [Wang et al., 2023a]. The results in Table 3 demonstrate that IdentityMask exhibits exceptional generalization capabilities. Remarkably, our method not only rivals but, in the case of the "sks-person"

prompt, outperforms the purpose-built SimAC (achieving the lowest ISM of 0.2527). This indicates that our strategy of anchoring on core facial semantics effectively disrupts identity encoding, regardless of whether personalization is achieved via textual inversion or weight fine-tuning. Consequently, IdentityMask offers a unified protection solution that remains robust without requiring adaptation to specific downstream personalization algorithms. For more details on tuning-free methods, please refer to the Appendix.

4.5 Ablation Study

To validate the contribution of each component, we conduct ablation studies, as detailed in Table 1.

Effectiveness of Face-Centric Identity Encoding (IE). We start by evaluating the model using Face-Centric Identity Encoding guidance (Base+IE). This configuration achieves an ISM score of 0.1171, which surpasses all baseline methods on the identity protection metric. This provides strong evidence that concentrating the adversarial budget on core facial features is a critical first step for effective protection.

Contribution of Rectified Signal-Aware Sampling (RSAS). Building upon the IE component, we then introduce RSAS. This addition significantly amplifies the protective power, with the ISM score dropping from 0.1171 to 0.0875. This configuration yields the strongest raw protection in our ablation. This improvement confirms that optimizing within the most semantically significant timestep ranges enables a more efficient generation of adversarial perturbations, avoiding waste on non-constructive diffusion stages.

The Role of Manifold Projection (MP). Adding the final MP component increases the ISM score from 0.0875 to 0.1502. It reflects a strategic trade-off: VAE reconstruction smooths high-frequency artifacts to embed the perturbations into the natural image structure. Although this slightly reduces raw potency, it grants the protection critical resilience against purification (Table 2), ensuring the defense remains effective in real-world scenarios where purifiers are deployed. Superiority of Semantic Guidance over Naive Spatial Masking. To demonstrate that the effectiveness of IdentityMask stems from targeted feature disruption rather than simple spatial constraints, we benchmark our method against baselines that are also explicitly restricted to the facial region. As shown in Table 4, IdentityMask outperforms these spatially-focused baselines, achieving a significantly lower ISM score compared to the strongest masked baseline. This highlights that spatial masking alone is insufficient. While masked baselines restrict noise to the face, their optimization remains semantically blind. In contrast, IdentityMask uses Face-Centric Identity Encoding to explicitly anchor gradients on identity-defining features. Thus, our superiority is due to the precise targeting of semantic features, rather than the spatial concentration of noise [Li et al., 2025] [Li et al., 2024a]. Efficiency Analysis. We evaluate the convergence efficiency of IdentityMask using K ∈ {50, 70, 90, 100} iterations. As shown in Figure 5, our method exhibits rapid convergence capabilities. Regarding identity protection, IdentityMask achieves parity with the fully converged AdvDM (K = 100) at merely 50 iterations, effectively reducing the computational cost. At 100 iterations, it further reduces the ISM score by 35.3%. Similarly, in semantic disruption, our method consistently outperforms the baseline as iterations increase.

5 Conclusion

We designed IdentityMask, a framework to protect facial data from unauthorized personalization. Unlike earlier approaches, our method uses a spatio-temporal adversarial perturbation that targets identity features and follows the model's natural learning patterns. Specifically, we employ FaceCentric Identity Encoding and Rectified Signal-Aware Sampling to achieve precise and efficient feature disruption. We also use Manifold Projection to handle purification attacks. This method turns delicate pixel noise into meaningful structural changes. Extensive tests confirm our method's effectiveness. IdentityMask surpasses state-of-the-art defenses. It offers a resilient solution for image protection. One possible extension of this work is to explore the trade-off between projection robustness and perturbation strength, as well as to adapt the mask-based approach to abstract artistic styles. However, a current limitation of IdentityMask is its multistage adversarial generation process, which needs a higher computational cost and longer processing time.

Acknowledgments

We thank the anonymous reviewers for their valuable feedback and will sincerely address all comments to improve the quality of our work.

Generative AI Usage Disclosure: Large language models were used for grammar checking and stylistic editing of the manuscript drafts. All substantive content, experimental design, analytical results, and data presented in this paper were produced by the human authors. The authors manually reviewed and verified all AI-assisted edits, and take full responsibility for the accuracy and integrity of the final manuscript.