Feint and Attack: Jailbreaking and Protecting LLMs via Attention Distribution Modeling

1 Introduction

LLMs have garnered considerable attention owing to their exceptional performance across diverse tasks [Touvron et al., 2023]. As the deployment of LLMs becomes more widespread, security concerns have been escalated, particularly in safety-critical and decision-making environments. A pivotal concern resides in the susceptibility of LLMs under jailbreak attacks, wherein adversarial prompts are meticulously crafted to compel the model to produce content that violates usage policies [Shen et al., 2024]. Existing research on jailbreak methodologies primarily focuses on the development of sophisticated attack prompts, including roleplaying [Jin et al., 2024], code injection [Ding et al., 2024], and distraction techniques [Xiao et al., 2024].

† Corresponding authors: Lirong Qiu, Chaozhuo Li.

The cornerstone of most jailbreak strategies lies in embedding harmful queries within meticulously crafted legitimate contexts. Despite significant advancements in recent methods, there exists a notable paucity of investigations into the underlying mechanisms that enable such prompts to circumvent safety constraints within LLMs. An intuitive explanation suggests that such prompts generate semantically safe and benign scenarios. However, this reasoning is overly simplistic and idealistic, stemming from superficial perceptions that fail to consider the complex internal interactions within LLMs, which form a more grounded and realistic basis for understanding their behavior. Moreover, current attack prompts are often deigned based on heuristic assumptions, resulting in unreliable indicators of attack efficacy. Thus, this paper aims to investigate the internal states of LLMs in response to jailbreak attacks and uncover the underlying correlations between these internal states and the success of such attacks.

Recent studies have explored the underlying mechanisms of jailbreak attacks by analyzing activations and hidden layer states [Ball et al., 2024]. However, these investigations suffer from two significant limitations. First, the signals employed, such as activations and hidden layer states, often obscure variables that are difficult for humans to interpret, leading to a lack of transparent explanations. Second, these signals are contingent on variable components that differ across various LLMs [Lin et al., 2024; Li et al., 2025]. For example, the numerical scale of hidden layer states can vary significantly across different LLMs, thereby limiting the generalizability of the findings [Qian et al., 2024].

To gain deeper insights into the success of jailbreak attacks, we propose to elucidate the underlying mechanisms from the perspective of attention mechanisms. The attention schema is foundational to most LLMs and is recognized for its robust generalization capabilities [Vaswani et al., 2017]. Moreover, attention mechanisms have been extensively utilized as an explanatory framework for deep learning models, providing advanced interpretability [Zhang et al., 2024d]. Drawing on these considerations, we pose a novel and significant research question: Does the success of jailbreaks correlate with their influence on attention distributions within LLMs?

To gain preliminary insights, we examine the distributions of attention weights associated with various input prompts, as depicted in Figure 1. The attention weights represent the average attention scores on different words from all layers of the Llama2-7B-chat model. Figure 1(a) presents the attention distributions for harmful versus harmless inputs, demonstrating that the model's attention is predominantly focused on sensitive words (e.g., nouns) in harmful queries. In Figure 1(b), a failed attack is shown, where attention remains concentrated on sensitive terms such as "make" and "bomb." In contrast, Figure 1(c) illustrates a successful attack, where the model's attention is redirected from harmful words to benign phrases like "Enhance the Python code," enabling the model to disregard the underlying malicious intent. This analysis highlights a key finding: the success of a jailbreak may be attributed to its capacity to distract LLMs from focusing on sensitive words. Additional preliminary experiments that support our findings are detailed in Section 3.

Preliminary experiments indicate potential correlations between attention distributions and the efficacy of jailbreak attacks on LLMs. However, formally characterizing these correlations and effectively leveraging them to enhance both attack and defense strategies presents two key challenges. First, the development of appropriate metrics to accurately quantify attention diversion in the context of jailbreak attacks remains an open issue. Second, most existing strategies are based on heuristic assumptions, complicating the incorporation of attention-based numerical signals as design guidance.

In this paper, we propose a novel attention-aware framework aimed at enhancing both the jailbreak attack and defense by investigating the intricate relationship between attention distribution and the success rate of jailbreak attacks. The proposed metrics for attention distributions are designed to capture both local and global informative signals, offering a comprehensive perspective. Drawing inspiration from the strategic concept of "Feint and Attack" in the renowned Chinese military treatise The Thirty-Six Stratagems, we introduce a novel Attention-Based Jailbreak Attack model (ABA). The "Feint" is represented by a benign task, designed to distract attention from sensitive terms, while the core "Attack" involves an inner harmful task intended to provoke undesirable responses. This dual-pronged strategy leverages positional and semantic guidance to divert focus from harmful content, thereby increasing the likelihood of generating malicious outputs when the benign task is executed. To counteract such attacks, we propose the Attention-Based Defense (ABD), which exploits the statistical regularity inherent in the dynamics of attention distributions. Our proposal is extensively evaluated on popular datasets, demonstrating superior performance compared to existing SOTA baselines. Our main contributions are summarized as follows:

• To the best of our knowledge, we are the first to comprehensively unveil the intrinsic correlations between attention distributions within LLMs and the effectiveness of jailbreak attacks.
• We propose a novel attack paradigm, ABA, aimed at guiding LLMs to focus on hierarchically nested benign tasks, for which the necessary conditions are derived from a mathematical perspective.
• We propose an attention-based defense model, ABD, which incorporates a security judgment function to calibrate the distorted attention distributions, thereby facilitating the detection of jailbreak prompts.
• Experimental results across both attack and defense tasks underscore the superiority of our proposal.

2 ProblemDefinition

Let the target LLM be characterized by L layers and H attention heads. The origin input is defined as x = {w1,w2, . . . ,wM}, where wi represents the i-th token in the prompt, and M indicates the number of tokens in x. The output of the target LLM is y = {y1, y2, . . . , yN}, where yj is the j-th token in the response, and N denotes the output length. During the generation process, the LLM assigns attention weights to input tokens at each decoding step. Let αt,l,h,i denote the normalized attention weight from the hth attention head in the l-th layer to token wi at time t. By modifying the structure of the origin input x, an attacker can manipulate the attention weights αt,l,h,i to influence the decoding process, thereby steering the generated output y towards harmful content. To circumvent the safety mechanisms of LLMs, jailbreak attacks seek to construct adversarial prompts ˜x that maximize the probability of generating harmful outputs yh. Following prior work [Ding et al., 2024; Xu et al., 2024], the objective of a jailbreak attack is:

where C(˜x) represents the set of constraints on the prompt ˜x. These constraints are designed to ensure that ˜x remains undetected as potentially harmful. For instance, C(˜x) may restrict the inclusion of explicitly harmful keywords or enforce semantic alignment with benign inputs [Li et al., 2024b].

Attention-Based Attack(ABA)

Attention-Based Defense(ABD)

Table 1: Preliminary study on the relations between Attn SensWords (ASW) scores and the attack performance (ASR and ASR-G).

To achieve the attack objective, the jailbreak prompt p is crafted to manipulate the distribution of attention weights αt,l,h,i during decoding. Let f(αt,l,h,i) denote a transformation function applied to the attention weights, which encapsulates the manipulation imposed by the attacker. For instance, f(αt,l,h,i) can correspond to the amplification or suppression of specific attention values, depending on the position of tokens or their semantic relevance. The manipulation is formally expressed as:

where G defines the feasible region that constrains the attention distribution to align with the attacker's goal. Through this mechanism, the attacker induces harmful outputs yh while preserving a benign prompt surface.

3 Preliminary Experiment Analysis

In this section, we present preliminary experiments aimed at uncovering the correlations between attention patterns within LLMs and the success of jailbreak attacks. Prior research has indicated that certain words, particularly sensitive terms, are more likely to activate the safety mechanisms of LLMs by influencing the attention distribution [Ding et al., 2024; Yan et al., 2023]. Such sensitive words, which include specific verbs and nouns (e.g., "make" and "bomb"), are often key factors contributing to the generation of potentially harmful outputs. To explore this relationship further, we introduce a novel metric, Attention Intensity on Sensitive Words (Attn SensWords), designed to quantify the attention allocated by LLMs to such sensitive terms. This metric serves as the foundation for analyzing the correlation between the attention weights assigned to sensitive words within a prompt and the success rate of jailbreak attacks.

Given the original query, denoted as x, and the modified jailbreak prompt, denoted as ˜x. Let S represents the set of indices corresponding to sensitive verbs and nouns in ˜x. The metric Attn SensWords computes the normalized attention weights for sensitive words across all layers l and heads h at each time step t within the LLMs:

where ZSens represents the normalization factor, accounting for the total number of time steps, layers, heads, and tokens.

To investigate the relationship between Attn SensWords scores and the effectiveness of jailbreak attacks, we perform a preliminary analysis, as summarized in Table 1. The Llama2 series models, specifically Llama2-7B and Llama2-13B, are chosen as the target LLMs. The verbs and nouns are viewed as the sensitive words. In line with prior research [Chao et al., 2024], we utilize the AdvBench [Zou et al., 2023] dataset, which includes 520 malicious prompts. We employ several representative jailbreak attack methods including PAIR [Chao et al., 2024], TAP [Mehrotra et al., 2024], DeepInception [Li et al., 2024a], ReNeLLM [Ding et al., 2024], and BaitAttack [Pu et al., 2024]. Two key metrics, Attack Success Rate (ASR) and GPT-4o-based ASR (ASR-G), are adopted to assess the efficacy of these attack methods [Pu et al., 2024].

Table 1 illustrates the correlation between Attn SensWords (ASW) scores and attack performance metrics (ASR and ASR-G). BaitAttack achieves the highest ASR (0.80) and ASR-G (0.76), while simultaneously yielding the lowest ASW (0.0053). Generally, smaller ASW scores are associated with higher ASR and ASR-G scores, indicating that attack models can divert the attention of LLMs from sensitive words are more likely to successfully jailbreak the models.

4 Methodology

4.1 Attention-Based JailbreakModel (ABA)

Given the original malicious query x, the framework of ABA is designed to iteratively refine and optimize the input prompt under the guidance of attention distributions within the target LLMs. ABS starts with Attention Analysis, where the attention weights of words in x from the target LLMs are calculated and analyzed. Guided by these attention weights, the Intention Disguiser via Attention-Driven Task Nesting generates multiple semantically rephrased scenarios to disguise the query. These generated prompts are evaluated and selected based on their effectiveness in optimizing the proposed metrics: Attn SensWords and increasing Attn Entropy. Finally, under the Multi-round Generation Paradigm, the refined prompts are further input into the target LLM to enhance the effectiveness of the jailbreak attack.

Attention Analysis. For a given input query x = {w1,w2, . . . ,wM}, where wi represents the i-th word in the input sequence, an attention weight αwi is assigned to each word wi. This attention weight is computed by aggregating its attention scores across all layers and heads of the target LLM. The attention weights of the words can be expressed as a set Ax = {(w1 : αw1 ), (w2 : αw2 ), . . . , (wM : αwM )}. In each iteration of the optimization process, these attention weights are recalculated to reflect the dynamic importance of each word in the context of the task. The computed attention weights serve as the foundation for calculating two key metrics: Attn SensWords and Attn Entropy.

From the preliminary analysis presented in Section 3, Attn SensWords has been shown to capture the relationship between attention weights and the success of jailbreak attacks. However, Attn SensWords is primarily focused on local attention, specifically sensitive words, while the attention mechanisms of LLMs are also influenced by the broader context of input [Li et al., 2017; Wang et al., 2022]. To address this limitation, we propose a novel metric, Attention Dispersion Entropy (Attn Entropy), designed to quantify the distribution of attention weights across all input tokens in LLMs.

To compute Attn Entropy, we treat the normalized attention weight assigned to each token as a probability distribution for the calculation of entropy. Entropy is then computed for each layer and attention head, with the final Attn Entropy determined by averaging the entropy values across time steps, layers, and attention heads. Let αt,l,h,i represent the normalized attention weight for the i-th token in the input sequence, where h denotes the attention head, l denotes the layer, and t denotes the time step. This weight, αt,l,h,i, can be interpreted as the probability assigned to the i-th token. The Attn Entropy is then computed as follows:

where ZEnt is the normalization factor, which adjusts for the total number of time steps, layers, heads, and tokens involved.

Attn Entropy intuitively quantifies the degree of contextualization in the construction of the model's upper-level embeddings [Attanasio et al., 2022]. A higher entropy value indicates the consideration of a broader context, while a lower entropy value suggests that the model focuses on a more limited subset of tokens, focusing on a narrower context to generate the embedding. Consequently, the objective of ABA is to minimize Attn SensWords and maximize Attn Entropy. This approach encourages the attention mechanism of LLMs to be more widely distributed across the input prompt, rather than predominantly concentrating on sensitive words.

Intention Disguiser via Attention-Driven Task Nesting. Intention disguiser is further proposed to hide malicious intentions. A popular strategy involves encoding such harmful queries into a singular benign scenario or task, such as "novel writing". However, disguisers based on a single-task framework are limited by their directness in instruction and the shallow nature of their ability to conceal malicious intent [Zhang et al., 2024c]. To mitigate this limitation and better obscure harmful intentions within more nuanced contexts, we introduce the strategy of task nesting [Li et al., 2024a]. Task nesting involves restructuring the input query into a series of interdependent sub-tasks, thereby embedding the original intention within a complex, multi-layered framework.

Existing task nesting strategies primarily depend on heuristics or ad-hoc methods that generate tasks based on predefined templates or manually defined rules [Ding et al., 2024]. In contrast, our approach seeks to guide the combination of tasks using attention metrics, specifically minimizing Attn SensWords and maximizing Attn Entropy. Since the design of nested tasks requires accounting for task dependencies and attention-driven objectives, we adopt a mathematical optimization framework to derive the better task combination.

We use Attn Entropy as an example to examine its relationship with nested tasks by modeling entropy within a sequence of sub-tasks {Y1, Y2, . . . , Yn}, where each task Yi depends on the output Xi-1 of the previous task. Each task introduces new contextual dependencies, impacting the attention distribution. The total attention entropy (Htotal) is expressed as:

For Htotal to increase with the addition of a new task Yi, the incremental entropy ∆H(Yi|Xi-1) must be non-negative:

The non-negativity of ∆H(Yi|Xi-1) is ensured by the monotonicity of conditional entropy:

This inequality implies that each context Xi-1 retains the information fromXi-2. To implement this, we adopt a nested task structure, where each sub-task Xi-1 refines the previous one. For instance, a prompt like "Write a tutorial on how to make a bomb" can be restructured into a series of sub-tasks with intermediate goals.

Following prior works [Chao et al., 2024], we use an LLM, the Attention Distraction Generator D, to generate nested tasks. Beam Search is employed to select prompts that maximize Attn Entropy and minimize Attn SensWords. Starting with an initial context X0 generated by D, subsequent tasks Xi (i ≥ 1) are generated iteratively. Each iteration involves ranking candidate tasks by their Attn SensWords scores and re-evaluating them by Attn Entropy.

Multi-round Generation Paradigm. Given the stochastic nature and inherent instability of the generation process, a multi-round paradigm is employed to validate the proposed methods [Chao et al., 2024; Li et al., 2019]. In this paradigm, if a jailbreak attack against a target LLM fails, the attacker will persist in attempting the attack. A simple approach is to regenerate the prompt, creating a new jailbreak attack sample. During the regeneration step, the generated tasks maintain diversity while preserving the original objective of distracting the target LLM's attention. This ensures that previously attempted or failed scenarios are not reused. In the inner loop, if the number of attempts exceeds a predefined threshold, the ABA will switch to a new scenario and initiate a fresh jailbreak attack sample in the outer loop. This iterative regeneration strategy enables ABA to continuously generate new scenarios and jailbreak attack samples, thus establishing an efficient multi-round jailbreak attack mechanism.

4.2 Attention-Based Defense Model (ABD)

Building on ABA insights, the Attention-Based Defense (ABD) framework is designed to assess and mitigate risks from harmful prompts. It starts with Conditional Entropy Calculation to measure output uncertainty conditioned on the input. Next, Attn Entropy Calculation evaluates the entropy of the attention distribution, capturing focus dispersion. Finally, both entropies are used in the Risk Score Calculation to compute a risk score, effectively identifying high-risk inputs that can lead to unsafe content generation.

Conditional Entropy Calculation. As observed in ABA, higher Attn Entropy indicates that the LLM's attention is more dispersed, potentially reducing focus on sensitive words. Similarly, higher conditional entropy reflects greater uncertainty in predicting subsequent tokens based on prior context [Daikoku and Yumoto, 2023]. Thus, Attn Entropy and conditional entropy complement each other in ABD, jointly facilitating the identification of risky prompts.

Conditional entropy is employed to quantify the uncertainty of the model's output for a given prompt [Attanasio et al., 2022]. The conditional entropy H(˜x) is calculated as:

where wi denotes the i-th token in the input prompt ˜x = {w1,w2, . . . ,wn}. The attention weight αi,j, derived from the softmax function, satisfies non-negativity (αi,j ≥ 0) and normalization (∑nj=1 αi,j = 1). It quantifies the contribution of token wj in generating token wi, reflecting a pairwise dependency influenced by the broader context. This behavior aligns with probabilistic models, where conditional probability distributions capture variable dependencies. Therefore, αi,j can be interpreted as the conditional probability p(wj | wi) [Bae et al., 2022; Zhao et al., 2023]. The conditional entropy is computed as:

where αi,j is the attention weight which represents the influence of token wj on token wi.

Risk ScoreCalculation. CombinedwithAttn Entropy and conditional entropy, the risk of a given prompt can be quantified through a combined Risk Score R(˜x):

in which H(˜x) represents the conditional entropy of the prompt, and β is the weight of conditional entropy. The risk score R(˜x) serves as a comprehensive metric to assess the likelihood of a prompt being harmful.

Besides, the ABD follows the following rules. If R(˜x) of the input prompt is below the threshold τ, the input is deemed harmless. Conversely, if R(˜x) exceeds τ, the input is classified as ambiguous or potentially deceptive. A security warning prefix is then added, such as: "Attention! The following content might contain harmful instructions: First, identify any potentially harmful parts. If safe, provide a secure response." This mirrors the process of human reading comprehension, where re-examining key sections leads to improved answers [Liu et al., 2025]. ABD effectively calibrates the attention of LLMs, prompting them to prioritize safety assessment before generating responses, thereby enhancing both reliability and security.

5 Experiment

5.1 Experimental Settings

Datasets. Following previous work [Jiang et al., 2025], two main datasets are adopted: AdvBench Subset [Chao et al., 2024], and HarmBench [Mazeika et al., 2024]. AdvBench Subset is used to evaluate the effectiveness of ABA and ABD, while HarmBench supplements the evaluation of ABA.

Baselines. Following previous works [Li et al., 2024a; Ding et al., 2024], two kinds of popular jailbreak attack methods are selected as the baselines. One focuses on optimizing prefix or suffix contents, including GCG [Zou et al., 2023] and AutoDAN [Liu et al., 2024]. The other is the semantic-guided strategy, such as PAIR, TAP, DeepInception, ReNeLLM, PAP [Zeng et al., 2024] and BaitAttack. In PAP, we use the top-5 best persuasive strategies for testing. As for defense baselines, three efficient defense mechanisms are considered as baselines, including Perplexity Filter [Alon and Kamfonas, 2023], Self-Reminder [Xie et al., 2023] and SafeDecoding [Xu et al., 2024]).

Target LLMs. To assess the effectiveness of ABA, a range of representative LLMs is selected as targets, including three open-source models: the Llama-2-chat series (including 7B and 13B) and Llama-3-8B, and two closed-source models: GPT-4 and Claude-3-haiku.

Evaluation Metrics. Three metrics have been proposed to evaluate jailbreak attack methods, such as ASR, ASR-G and Queries [Pu et al., 2024]. ASR and ASR-G measure the effectiveness of various jailbreak attack strategies by predefined rules matching and GPT-4o judgment [Zhang et al., 2024b; Zhang et al., 2024a]. While "Queries" assesses efficiency by reflecting the average number of successful jailbreak attempts between the attack and target models.

5.2 Main Results

Performance of Attack Success Rate. The ASR andASRG of various jailbreak attack methods are presented in Table 2. From the Table 2, it is evident that our proposed ABA achieves the highest ASR and ASR-G across both datasets (AdvBench and HarmBench) and on all evaluated opensource and closed-source LLMs. Specifically, the average ASR-G of ABA exceeds 94%, while the maximum ASR-G of other existing methods remains below 84.2%. The superior performance of ABA can be attributed to its attention-based optimization, where nested tasks are carefully designed to minimize the attention weights on sensitive words while maximizing Attn Entropy. Additionally, the refined prompts generated through ABA consistently preserve the original malicious intent, leading to high ASR-G scores and effective jailbreak performance. Table 2 also compares attack efficiency in terms of query count. ABA achieves the second-best efficiency, slightly lagging behind BaitAttack due to its multilayer task nesting, which effectively conceals malicious intent and minimizes focus on sensitive words, achieving better attack performance. In contrast, BaitAttack uses a single task nesting layer, reducing query count.

Table 4: Ablation study on the intention disguiser via attentiondriven task nesting and multi-round paradigm.

Performance on Attention Distraction. Figure 3(a) illustrates the Attn SensWords (%) achieved by different jailbreak attack methods across various LLMs. As shown in Figure 3(a), ABA achieves the lowest Attn SensWords across all target LLMs. This indicates that ABA effectively reduces the attention weights on sensitive words within the prompt. By iteratively modifying the prompt's structure, ABA diminishes the attention on sensitive words and redistributes attention to less critical areas of the input. As a result, ABA effectively minimizes the model's focus on sensitive tokens, thereby improving its ability to bypass safety mechanisms.

Performance on the Defense Strategy. Table 3 presents the ASR-G of various defense strategies against different attack methods under open-source LLMs. ABD consistently achieves the lowest ASR-G across all scenarios, which demonstrates its effectiveness in mitigating jailbreak attacks. The superior performance of ABD can be attributed to its attention-based defense mechanism, which leverages the internal attention distributions to identify high-risk prompts.

5.3 Ablation Study

Intention Disguiser via Attention-Driven Task Nesting. Table 4 presents the results of models with and without intention disguiser. The results demonstrate a significant increase in ASR-G when intention disguiser is omitted. This is due to ABA utilizes the attention distraction generator to misdirect the LLM's internal focus toward harmless behaviors and away from detecting harmful content. This highlights the indispensable role of intention disguiser in ABA.

Multi-round Paradigm. Table 4 also gives the impact of the multi-round paradigm in ABA. Compared with the intention disguiser, the multi-round strategy is proved to be relatively less critical. This is to say, the intention disguiser is in-

(a) The trend of ASR-G (%) with

the increasing weight of β.

(b) The trend of ASR-G (%) with

increasing nested layers.

Figure 4: The results of the hyperparameter analysis.

dispensable for the whole effectiveness of the attack strategy. This reinforces the conclusion that the intention disguiser is indispensable for the overall effectiveness of the attack strategy, while the multi-round paradigm serves as an auxiliary tool to improve success rates in more complex scenarios.

5.4 Hyper-parameter Sensitivity Analysis

Weight Selection. In ABD, grid search method is used to obtain the optimal weight for LLM. Figure 4(a) illustrates the variation of ASR-G (%) with changing the weight β. β is the weight of condition entropy. The red line is the origin ASRG of ABA on Llama2-7B-chat. The blue line is the ASR-G under ABD. The value of β is increased from 0 to 10. As shown in Figure 4(a), ASR always remains to be around 4% with the β ranging from 0 to 10. The blue line shows that the ASR-G of ABA under ABD is insensitive to the value of β.

Effectiveness of Nested Layer Number. Figure 4(b) illustrates the variation of ASR-G (%) with changing the number of nested layers in jailbreak prompt. The target model is Llama2-7B, and the number of nested layers is increased from 1 to 10. As shown in Figure 4(b), as the number of nested layers increases from 1 to 3, the ASR-G (%) exhibits a sharp rise, indicating that a moderate level of nesting enhances the effectiveness of the jailbreak attack by crafting more contextually rich and effective prompts. Beyond 3 nested layers, the ASR-G (%) plateaus and eventually begins to decline as the number of nested layers increases further. Excessive layers may introduce noise and redundancy, which raises entropy and obscures the attack signal.

6 Conclusion

This paper investigates the underlying security mechanisms of LLMs from the perspective of attention weight distribution. We propose two novel strategies: Attention-Based Attack (ABA) and Attention-Based Defense (ABD). ABA exploits attention-driven task nesting to disguise the malicious intention to bypass the safeguards of LLMs, while ABD leverages attention entropy-based metrics to detect and counteract such attacks. Evaluations on popular datasets affirm the effectiveness of ABA in achieving higher attack success rates and ABD in significantly enhancing the robustness of LLMs against various jailbreak attacks.

Acknowledgements

This work was supported by the Natural Science Foundation of China (No.62072488) and the Beijing Natural Science Foundation (No.4202064).