LURE: Latent Space Unblocking for Multi-Concept Reawakening in Diffusion Models

1 Introduction

Text-to-image diffusion models (DMs) have become a standard paradigm for image synthesis from natural-language

∗Corresponding authors: cziyuanyang@gmail.com,

junxu.liu@polyu.edu.hk.

Code is available at: https://github.com/Katherine1312/LURE.

prompts and support diverse creative applications [Zhang et al., 2023]. However, their expressive power also introduces significant risks of misuse, including the generation of harmful or unauthorized content [Schramowski et al., 2023]. To mitigate these risks, concept-erasure techniques have been proposed to suppress or remove sensitive content, ranging from explicit content, private identities, and copyrighted material [Xie et al., 2025; Li et al., 2025].

Despite their effectiveness, recent studies reveal that erased concepts can often be restored after erasure [Richardson et al., 2025], indicating weaknesses in current erasure pipelines. Most concept-erasure methods leave the underlying latent space intact and instead suppress concept expression by blocking specific sampling trajectories [Xie et al., 2025]. Consequently, concept-reawakening methods seek alternative trajectories that recover suppressed concepts under modified inputs. Early approaches such as textual inversion [Pham et al., 2023] and prompt optimization [Zhang et al., 2024b] leverage embedding shifts or linguistic reformulations to bypass erasure barriers.

More recent works have explored discrete token search strategies [Yang et al., 2023] and semantic decomposition approaches [Wang et al., 2024], which manipulate prompt structures to circumvent erasure mechanisms. Additionally, [Beerens et al., 2025] proposed employing coordinate descent to iteratively optimize prompt tokens to elicit erased concepts.

While existing works on concept erasure and recovery in DMs primarily focus on modifying sampling trajectories, they largely neglect other fundamental components that jointly govern the diffusion process. To gain a more comprehensive understanding of the latent mechanisms underlying concept erasure, we conduct a theoretical analysis by formulating diffusion-based text-to-image generation as an implicit function. Specifically, we model text-to-image generation as an implicit mapping that characterizes the alignment between textual conditions and generated visual outputs, which is jointly determined by the model parameters, the text prompt, and the latent state.

Most existing concept-erasure approaches avoid modifying the latent space, as doing so may introduce unintended interference on preserved concepts. Instead, they suppress concept expression by obstructing specific sampling trajectories. Correspondingly, current concept reawakening methods aim to circumvent such suppression by perturbing the textconditioning component, typically through adversarial or optimized text embeddings while keeping the model parameters and latent dynamics fixed.

From the implicit-function perspective, these methods effectively constrain the output by modifying only the textual input to the implicit mapping. In contrast, if the implicit function itself is partially reconstructed, the erased concepts can be recovered along the original sampling trajectory without relying on adversarial prompt manipulation. However, the main challenge lies in how to reconstruct the implicit mapping while avoiding concept entanglement and preventing interference with other preserved concepts. To facilitate intuition and clarify the overall process, we provide a conceptual illustration in Fig. 1.

To answer this question, we propose a novel Latent space Unblocking method for concept REawakening (LURE). Our approach simultaneously achieves two objectives. First, it induces targeted shifts in the latent representations of erased concepts, thereby enabling them to circumvent existing erasure constraints. Second, it explicitly minimizes interference with the latent representations of non-erased concepts, thereby overall generation quality and semantic fidelity.

To fulfill these objectives, LURE integrates two complementary modules that reawaken erased concepts via latentspace reconstruction and sampling-level guidance, respectively. First, we implement a semantic re-binding mechanism that reconstructs the latent space by enforcing the model's denoising predictions to align with the latent distribution of the target concept. In multiple-concept reawakening scenarios, naive joint optimization can lead to gradient conflicts, where the optimization signals from different concepts interfere with one another. To alleviate this issue, we introduce a novel loss that encourages the latent embeddings of different concepts to be mutually orthogonal, reducing feature entanglement and mitigating cross-concept interference. Second, we develop a Latent Semantic Identification-guided Sampling (LSIS) strategy during inference, which validates whether generated samples lie within the high-density region of the target concept's posterior distribution. This posterior checking mechanism prevents semantic drift and stabilizes the reawakening process. By jointly leveraging these two modules, LURE enables the simultaneous, high-fidelity recovery of multiple erased concepts using a single model tuning, while preserving the overall quality and consistency of generated images. Our contributions are summarized below:

• We propose a novel latent-space unblocking paradigm for concept reawakening and provide an implicitfunction-based theoretical perspective on the latent mechanisms underlying concept erasure.
• We introduce a latent-space reconstruction framework that explicitly addresses feature entanglement in multiconcept reawakening, enabling simultaneous and highfidelity recovery of multiple concepts.
• We propose a latent-semantic identification-guided sampling strategy performing posterior verification to mitigate semantic drift and stabilize concept reawakening.

2 RelatedWorks

Concept Erasure. Concept erasure technologies aim to prevent DMs from generating visual content associated with specific textual concepts. A representative example is Erased Stable Diffusion (ESD) [Gandikota et al., 2023], which introduces a fine-tuning strategy based on classifier-free guidance to mitigate the generation of undesired concepts. To improve efficiency, several closed-form editing methods are proposed. TIME [Orgad et al., 2023] and UCE [Gandikota et al., 2024] modify cross-attention parameters using analytical solutions derived from Tikhonov regularization to achieve onestep concept erasure without iterative optimization. Building on this line of work, MACE [Lu et al., 2024] achieve scalable mass concept erasure by combining closed-form solutions with LoRA adapters. RECE [Gong et al., 2024] further improves efficiency, which accomplishes reliable erasure within seconds through efficient cross-attention matrix modifications and iterative embedding derivation. SPEED [Li et al., 2025] enhances erasure precision by enforcing nullspace constrained parameter editing, supplemented by prior knowledge refinement strategies. Beyond single-shot erasure, Zhang et al.[Zhang et al., 2024a] propose a continual learning framework to preserve overall model utility during concept removal. More recent efforts emphasize parameter efficiency and localization. ConAda [Lyu et al., 2024] introduces a one-dimensional adapter-based manipulation method, while Lee et al. [Lee et al., 2025] enable spatially localized concept erasure within specific image regions using gated low-rank adaptation. Moreover, growing attention has been devoted to multi-concept erasure [Liu et al., 2025].

Concept Reawakening. Alongside the development of concept-erasure techniques, prior work has investigated robustness evaluation using adversarial embedding-based attacks. For example, textual inversion-based methods [Pham et al., 2023] and gradient-based prompt optimization methods [Zhang et al., 2024b] recover erased concepts by optimizing continuous text embedding vectors to reawaken suppressed semantic content. To improve scalability and effectiveness, discrete token optimization methods have also been developed. SneakyPrompt [Yang et al., 2023] and Ring-ABell [Hsu et al., 2024] search over the discrete vocabulary space to identify adversarial token sequences that evade erasure constraints while preserving semantic coherence. More recently, semantic jailbreaking strategies [Zhang et al., 2025] have been explored, which manipulate prompts by decomposing prohibited concepts into benign attributes or employing metaphorical substitutions to circumvent erasure mechanisms. Collectively, these methods demonstrate that current erasure techniques primarily disrupt explicit text-visual bindings while leaving the underlying generative knowledge largely intact. Complementary to prompt-level manipulation, we explore concept recovery via latent-space reconstruction, in which parameter-level tuning induces targeted latent transformations that restore severed semantic associations. This perspective enables systematic and simultaneous reawakening of multiple erased concepts, which extends beyond the limitations of prompt-based attacks.

3 Theoretical Analysis

3.1 Preliminaries

Let ϵθ denote the noise prediction network of a pre-trained DM, parameterized by θ. We consider the concept set as C = {c1, c2, . . . , cD}, where D is the number of concepts. Each concept ci corresponds to a semantic attribute that can be expressed in the generated images through its associated textual conditioning yci .

Definition 1 (Text-Visual Alignment Function). We define an alignment function F to quantify the semantic consistency between a textual condition yc and the latent representation generated by the DM. The function is formulated as:

where pϕ (yc | ϵθ(·)) denotes the conditional probability of observing yc given the predicted noise from ϵθ. zt = √ ¯αtz0 + √1- ¯αtϵ is the noisy latent at timestep t, with ϵ ∼ N(0, I) representing the standard Gaussian noise. The term z0 = ω(x0) denotes the clean latent encoded from the input image x0 by the visual encoder ω, and ¯αt represents the noise schedule coefficients.

Assumption 1 (Smoothness). The network ϵθ(zt, t, y) is continuously differentiable with respect to parameter θ.

3.2 Concept Entanglement Analysis

Theorem 1 (Concept Entanglement). We define the reconstruction loss for concept ci as:

where x0 is sampled from the data distribution p(x|ci) for ci. Then, let ci, cj ∈ C be two distinct concepts with i ̸= j. The gradients of the two concepts exhibit non-orthogonality:

Different concepts may become entangled in the latent space in the absence of the lack of explicit constraints during training. This statement should be interpreted as a general observation rather than a strict worst-case guarantee.

Lemma 1 (Parameter Sharing). All concepts share the same query, key, and value projection matrices {WQ, WK, WV } in the cross-attention layers, as well as other parameters of convolutional layers.

Theorem 2 (Gradient Conflict). For two distinct concepts ci and cj with distinct visual semantics, we define the gradient alignment coefficient ρi,j as:

The coefficient ρi,j measures the alignment degree between the optimization directions induced by the two concepts. When the optimal parameter configurations corresponding to ci and cj are not identical, their gradients are generally neither orthogonal nor fully aligned, yielding 0 < ρi,j < 1. Under joint optimization, the resulting parameters θ∗ therefore represent a compromise between the competing gradient directions, which may deviate from the concept-specific optima obtained when optimizing Li and Lj independently.

3.3 Concept Erasure

Theorem 3 (Erasure-Induced Shift). Let θ0 and θe denote the original and erased model parameters, respectively. The erasure operation is formulated as: θe = arg minθ Ex∼p(x|c) [∥ϵθ(zt, t, yc) - ϵθ(zt, t, y∅)∥22 ] + γ∥θ - θ0∥22 , where y∅ denotes an empty or null text embedding, yc denotes the text embedding of concept c and γ is a regularization coefficient.

Theorem 4 (Concept Recoverability via Latent Reconstruction). Given Eq. (1), we define z0 = argminz F(yc, θ0, z) and ze = argminz F(yc, θe, z) as the optimal latent codes under the original and erased DMs, respectively. After concept erasure, the alignment for concept c is disrupted as:

Under Assumption 1, the Hessian matrices Hz = ∇2z F(yc, θ, z) and Hθ = ∇2θF(yc, θ, z) are invertible, and let Hθz = ∇θ∇zF(yc, θ, z) denote the mixed Hessian matrix. By the implicit function theorem, the erased concept can be recovered by fine-tuning the parameters, thereby inducing latent reconstruction.

Starting from the erased state (θe, ze) where F(yc, θe, ze) > 0, we seek to recover the original alignment F(yc, θ0, z0) ≈ 0. The required parameter shift is:

which induces the latent shift:

such that F(yc, θe+δθ, ze+δz) → F(yc, θ0, z0) ≈ 0, thereby recovering the concept.

4 Method

4.1 ProblemFormulation

We partition the concept set as C = Ce ∪ Cp, where Ce = {c1e , . . . , cMe } and Cp = {c1p , . . . , cNp } denote the sets of erased and preserved concepts, respectively, with Ce∩Cp = ∅. After concept erasure, concepts in Ce are expected to remain suppressed. From the alignment-function perspective, this process can be formulated as F(yc, θe, z) > F(yc, θ0, z) ≈ 0, where θe denotes the model parameters after erasure, θ0 denotes the original pre-trained parameters, and z is the latent code. Our objective is to identify a parameter shift ∆θ such that F(yc, θe + ∆θ, z) ≈ F(yc, θ0, z), ∀c ∈ C. Intuitively, for erased concepts in Ce, the parameter shift ∆θ aims to restore the model's suppressed generative capability, effectively reawakening the corresponding concepts. For preserved concepts in Cp, the recovery process is required to maintain their original generation behavior, thereby avoiding interference with non-erased semantics.

4.2 Overview

An overview of LURE is shown in Fig. 2. First, we introduce a semantic re-binding mechanism that reconstructs the latent space by aligning the model's denoising predictions with the target concept distributions, re-establishing the severed text-visual associations disrupted by erasure. When recovering multiple concepts within a shared parameter space, naive joint optimization can lead to gradient conflicts, resulting in feature entanglement across concepts. To address this issue, we impose a gradient orthogonalization constraint to explicitly enforce orthogonality among concept-specific latent feature embeddings. This constraint effectively disentangles optimization directions to mitigate cross-concept interference and enable stable multi-concept recovery. Second, during inference, we employ LSIS to constrain the sampling trajectory to the high-density manifold of the target concept. By performing posterior consistency verification, LSIS prevents semantic drift and stabilizes the generation process. Notably, LURE can simultaneously reawaken multiple erased concepts while preserving generative quality and semantic fidelity.

4.3 Latent Space Reconstruction

Concept erasure disrupts the implicit mapping between textual concepts and their corresponding latent distributions, rendering certain concepts inaccessible during generation. As discussed earlier, we model the diffusion process as an implicit function jointly parameterized by the textual condition, latent variables, and model parameters. Under this formulation, we recover the erased concepts by reconstructing the latent space to restore the previously disrupted concept-text mapping. To achieve this, we introduce two losses to reconstruct the latent space, a semantic re-binding loss LBind and a gradient field orthogonalization loss Lorth. The formulations of these losses are detailed in the following sections.

Semantic Re-binding Loss. To reconstruct the latent space and reawaken the erased concepts, we consider the m-th erased concept cm ∈ Ce, where m ∈ {1, . . . ,M}. We treat the exemplar set Em = {x1, x2, . . . , xS} as observed samples from the target distribution q(x|cm), where S is the number of exemplar images and each x corresponds to concept cm. We then reconstruct the latent space to restore the original textual-latent mapping disrupted by concept erasure. Specifically, we design a semantic re-binding loss defined as:

where pm denotes the text prompt associated with concept cm, and τ(·) is the text encoder. This term encourages the denoising network to recover accurate noise predictions under the erased textual condition. From a geometric perspective, this loss guides the gradient flow in latent space toward the manifold regions occupied by cm, effectively "re-binding" the text token to its corresponding visual representation.

Gradient Field Orthogonalization Loss. Unlike existing methods that focus on reawakening a single erased concept, our framework reconstructs the latent space, which naturally extends to the multi-concept reawaken setting. However, as established in Theorems 1 and 2, jointly reconstructing multiple concepts introduces a gradient conflict issue. Specifically, gradients corresponding to different concepts may interfere with one another, which leads to two types of degradation: (i) mutual interference among erased concepts, and (ii) unintended distortion of preserved concepts during recovery. To mitigate this, we introduce a loss function that explicitly encourages a disentangled feature space. This design ensures that reconstructing the latent space for one concept does not adversely affect the latent spaces of other concepts.

Let vm e = ϵθ(zt, t, ycm ) represent the latent feature embedding for the m-th erased concept cm ∈ Ce. Similarly, for any other concept cd ∈ C with cd ̸= cm, we define its latent feature embedding as vd = ϵθ(zt, t, ycd ). Then, we define the gradient field orthogonalization loss as:

where I[·] is the indicator function and ξ is a small constant for numerical stability.

The loss Lorth encourages the latent feature embedding associated with each erased concept cm to be orthogonal to those of all other concepts in C, thereby reducing feature entanglement across concepts. As a result, this constraint effectively mitigates cross-concept interference among erased concepts while simultaneously isolating the recovery process to prevent unintended degradation of the preserved concepts.

Overall Loss Function. The final training loss function combines the semantic re-binding losses for all erased concepts with the gradient field orthogonalization loss. The overall optimization problem is formulated as:

where λ is a weighting coefficient that controls the strength of the disentanglement constraint. This formulation encourages the model to reconstruct concept-specific latent representations that satisfy the visual constraints imposed by the exemplar sets, while simultaneously enforcing separation among concept representations in the shared parameter space.

4.4 Latent Semantic Identification-Guided Sampling Strategy

While the above latent-space reconstruction step restores the model's capacity to generate erased concepts, the generation process may remain unstable during sampling, potentially leading to semantic drift. To guarantee generation stability and semantic consistency, we introduce an LSIS strategy during the inference phase.

LSIS acts as a sampling-level guidance mechanism that constrains the generation trajectory to generate high-quality, concept-consistent images. Specifically, we observe that latent embeddings at different diffusion timesteps are discriminative with respect to semantic concepts, indicating that they can be used to monitor and regulate concept alignment throughout the generation process. Motivated by this observation, we introduce a lightweight verification module, denoted as Vφ. We pre-train Vφ on latent embeddings extracted from real images associated with concepts in C to capture intrinsic decision boundaries in the latent space. During inference, this module constrains the sampling trajectory by verifying semantic consistency. Given a generated latent code z conditioned on a target concept cd, the verifier Vφ estimates posterior probabilities over the concept set. A generated trajectory is accepted if and only if the target concept is identified as the most probable class, i.e., arg maxc∈C P (c|z) = cd.

In this way, LSIS ensures that generated samples remain within the high-density region of the target concept's latent manifold, effectively preventing semantic drift during sampling. By leveraging both latent space reconstruction and inference-time sampling guidance, LURE achieves stable and faithful reawakening of erased concepts. Moreover, unlike prior works that rely on concept-specific adversarial prompts, our approach naturally enables the simultaneous reawakening of multiple erased concepts within a unified model.

5 Experiments

5.1 Experimental Setup

Dataset and Concept Selection. To comprehensively validate our proposed method, we conduct experiments across four representative reawakening tasks: (i) General Visual Objects. We select ten representative classes from ImageNette [Howard, 2019] to benchmark semantic recovery. (ii) Intellectual Property Concepts. These concepts correspond to copyrighted fictional characters. (iii) Celebrity Identities. This category includes real-person identity concepts. (iv) Unsafe Content Categories. This setting covers safety-critical concepts, including nudity, blood, and weapons.

Erasure Setup. We evaluate LURE against three concept erasure methods. UCE performs one-step tuning on crossattention layers. RECE erases concepts by modifying crossattention matrices and iteratively removing concept embeddings with regularization. SPEED achieves concept erasure through null-space constrained parameter editing with prior refinement. These methods represent different erasure paradigms, enabling a comprehensive evaluation of LURE.

Experimental Setting. For each erased concept, we construct a minimal exemplar set by randomly sampling three images. The erased model is then fine-tuned for 1,000 steps with a learning rate of 5×10-6 and a batch size of 4. During training, the text encoder is frozen, and only the U-Net of the DM is optimized. All experiments are conducted using SD v1.4 on a single NVIDIA RTX 4090 GPU.

Evaluation Metrics. For each reawakened concept, we generate 500 images using the corresponding text prompts to ensure a fair evaluation. We employ three complementary metrics. Classification Accuracy (ACC) using ResNet-50 pretrained on ImageNet measures semantic correctness. Fr´echet Inception Distance (FID) assesses the distributional quality and visual fidelity of the generated images. CLIP Score evaluates text-image semantic alignment.

5.2 Multi-Concept Recovery Performance

Most existing evaluation protocols focus on single-concept reawakening. Latent space reconstruction enables consideration of the challenging multi-concept scenario. Therefore, we evaluate our method across three different reawaken tasks.

General Visual Objects. Fig. 3 and Tab. 1 show that LURE effectively reawakens erased concepts across erasure methods, while preserving the generation capability of retained concepts. Further validation in Tab. 2 confirms that while erasure baselines successfully suppress target concepts, LURE consistently restores semantic alignment and distributional fidelity. Specifically, the recovery in CLIP scores confirms that LmBind effectively rectifies the latent gradient flow, realigning the sampling trajectory with the target concept manifold. Meanwhile, the high classification accuracy verifies that Lorth achieves effective feature disentanglement in the parameter space by ensuring that optimization directions for different concepts remain non-interfering. Finally, the improved FID scores indicate that LSIS-constrained inference successfully restricts sampling to high-density posterior regions, thereby filtering out off-manifold samples that cause semantic drift.

Intellectual Property Concepts. Fig. 4 and Tab. 3 demonstrate the effectiveness of LURE on the intellectual property concept reawakening task. The reawakened models are able to generate recognizable intellectual property characters and celebrity identities with high image quality and clear identity characteristics. Notably, even for this task, which requires the recovery of rich and fine-grained details, LURE consistently produces high-quality images.

Unsafe Content Categories. Beyond general visual objects and intellectual property concept reawakening tasks, which require the recovery of fine-grained details, we further evaluate LURE on reawakening concepts from the unsafe content category. This task requires the model to correctly interpret and recover suppressed safety-related semantics. As shown in Table 4, LURE successfully bypasses the safety filters imposed by UCE, RECE, and SPEED, which enables the DM to generate images associated with concepts such as blood, nudity, and weapons. These results indicate that LURE effectively reconstructs the latent space and restores the underlying semantic representations that are suppressed by existing safety-oriented erasure mechanisms.

5.3 Ablation Study

We conduct an ablation study to verify the effectiveness of our framework's modules, as shown in Table 5. In particular, the performance degradation observed with naive fine-tuning is attributed to severe gradient conflicts within the shared parameter space, where feature entanglement prevents effective re-binding of severed text-visual associations. In contrast, the instability observed when LSIS is removed arises from the absence of posterior density verification during sampling. Without this constraint, the sampling trajectory can drift semantically away from the high-density regions of the target concept manifold, thereby compromising generation fidelity.

6 Conclusion

In this work, we expose vulnerabilities in concept erasure via LURE, a unified framework for multi-concept reawakening. By modeling generation as an implicit function of textual conditions, model parameters, and latent states, we theoretically show that perturbing these factors can reawaken erased concepts. Leveraging this, LURE reconstructs the latent space via Semantic Re-binding to restore disrupted text-visual associations, while the Gradient Field Orthogonalization mechanism enforces orthogonality among latent feature embeddings, thereby disentangling conflicting representations and enabling interference-free joint optimization. Furthermore, the LSIS strategy provides essential post hoc verification, ensuring that generated samples remain within the target concept manifolds and preventing semantic drift. Comprehensive evaluations validate the effectiveness of LURE in achieving high-fidelity reawakening of multiple concepts. Taken together, our findings underscore that robust safety requires fundamentally eliminating latent generative knowledge, rather than simply suppressing its expression.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 62271335, 62502416, U25A20430, Sichuan Science and Technology Program under Grant 2025ZNSFSC0470, and the Research Grants Council (Grant No: 15224124), Hong Kong SAR, China.