Understanding and Exploiting Phase Sensitivity for Attacking Large Vision–Language Models

1 Introduction

In recent years, Large Language Models (LLMs) [Liu et al., 2021a; Liu et al., 2022b; Liu et al., 2022c] have demonstrated remarkable capabilities in understanding and generating human-like text across a wide range of tasks. Building upon these advances, Large Vision-Language Models (LVLMs) [Dai et al., 2024; Liu et al., 2024d; Zhu et al., 2023] have emerged by integrating powerful vision encoders with LLMs, enabling multimodal understanding and reasoning over both images and text. These models have achieved impressive performance in tasks such as visual question answering [Liu et al., 2021b; Li et al., 2023; Alayrac et al., 2022; Liu and Hu, 2022], image captioning [Liu et al., 2020; Ramesh et al., 2022; Liu et al., 2022a; Liu et al., 2023c], and multimodal dialogue, and are increasingly deployed in real-world applications. However, the growing complexity and openness of LVLMs also introduce new security concerns [Liu et al., 2024c]. In particular, their multimodal nature exposes them to a broader and more intricate range of adversarial vulnerabilities, which can be exploited to manipulate model outputs or induce harmful behaviors.

→Corresponding Authors.

Figure 1: Instead of developing external adversarial constraints, we explore the frequency-domain information to interpret the intrinsic characteristics of LVLM.

Prior efforts [Shayegani et al., 2023; Bailey et al., 2023; Dong et al., 2023; Wang et al., 2023; Wang et al., 2024b; Zhang et al., 2024; Lu et al., 2024; Luo et al., 2024; Tao et al., 2024; Zhao et al., 2024] on attacking LVLM models primarily focus on designing input-level perturbations that explicitly interfere with the model's cross-modal reasoning process. Most of them [Bailey et al., 2023; Wang et al., 2024b; Lu et al., 2024; Luo et al., 2024; Tao et al., 2024] typically operate by injecting visually or semantically noisy patterns into image or text inputs, thereby forcing the model to attend to misleading signals via adversarial optimization. Some works [Shayegani et al., 2023; Dong et al., 2023; Wang et al., 2023; Zhao et al., 2024; Zhang et al., 2024] attempt to mimic harmful semantics, hijack attention maps, or align features with pre-defined adversarial objectives, requiring access to the visual encoder of LVLMs like CLIP [Radford et al., 2021; Sun et al., 2023] to generate the perturbed visual representations for misleading the latter reasoning process. While these approaches have achieved notable adversarial effects, they tend to rely heavily on trial-and-error crafting or brute-force optimization, overlooking the internal perceptual mechanisms of LVLMs that how LVLMs inherently perceive and organize visual inputs - for example, their sensitivity to structural or frequencydomain information - which may offer more principled and efficient pathways for adversarial exploitation.

To this end, our work makes the first attempt to rethink the adversarial designs from a new perspective of LVLM's intrinsic perception. Inspired by cognitive science [Geirhos et al., 2018; Samuelson and Smith, 2005; Fang et al., 2023b; Liu et al., 2024a] where human vision mainly relies on structural semantics to comprehend objects, we also explore the LVLM's characteristics in this manner. Given an image input, it can be inherently converted into the frequency domain via the discrete Fourier transform (DFT), and further decoupled into the amplitude spectrum and the phase spectrum. In particular, the amplitude spectrum carries the pixel intensity information (style texture) [Oppenheim and Lim, 1981; Tolhurst et al., 1992; Fang et al., 2022] while the phase spectrum can reflect the highly informative structural features [Gdeisat and Lilley, 2016; Liu et al., 2023a; Morrone and Owens, 1987; Kovesi, 2000; Fang et al., 2023a], which is consistent with the need of human perception. We visualize these spectrums and corresponding Inverse DFT patterns in Figure 1, where phase-aware structural semantics are eliminated and smoothed in the adversarial image, thus leading the LVLM to capture wrong object contexts for reasoning. In addition, the amplitude patterns are similar between the clean and adversarial images. These findings indicate that LVLMs tend to leverage more phase information to understand and recognize detailed semantics of images.

Based on the above observations, in this paper, we propose a novel LVLM attack method by manipulating adversarial modification in the spectral phase domain. Instead of explicitly perturbing the phase spectrum of a given image, we follow a more universal backdoor attack paradigm that implants a certain phase spectrum of a targeted image into the current one to lead the LVLM to output the semantics of the targeted image. Such attack setting is more flexible as the same phase spectrum can be transferable across diverse images to achieve the same adversarial effects. To control the backdoor behavior, a perturbation backdoor switcher is further added to the phase spectrum, and a normal textual trigger is also introduced to consolidate the backdoor alignment. The whole adversarial patterns are trained to activate the backdoor only when all triggers are present during the LVLM testing stage. Our contributions are three-fold:

• To the best of our knowledge, this paper makes the first attempt to design a more interpretable and effective LVLM attack from the perspective of LVLM's inherent perception. We provide sufficient empirical analysis on the LVLM's vulnerability to the spectral phase.
• We propose BadPhase to inject a universal phase spectrum trigger in image inputs to guide LVLM to output a certain input-irrelevant semantics. Three components, phase trigger, textual trigger, and a backdoor switcher, are jointly optimized to control the backdoor behavior.
• Extensive experiments are conducted to verify the effectiveness and efficiency of our attack approach on various LVLMs, datasets, and tasks.

2 RelatedWork

LVLMmodels. The remarkable progress of Large Language Models (LLMs) in language-centric tasks [Liu et al., 2021c; Liu et al., 2023b; Liu et al., 2023d; Liu and Hu, 2024; Liu et al., 2026b; Liu et al., 2026a; Liu and Hu, 2025], particularly with the advent of GPT-4 [Ouyang et al., 2022], has sparked growing interest in extending these models to handle multimodal tasks. This has given rise to the development of Large Vision-Language Models (LVLMs), which aim to integrate visual and textual modalities effectively. To bridge the semantic gap between vision and language, various approaches have been proposed. Some methods [Alayrac et al., 2022; Li et al., 2023] employ learnable prompts to extract visual content, which is then used to guide text generation through LLMs. Other models like MiniGPT-4 [Zhu et al., 2023], LLaVA [Liu et al., 2024d], and PandaGPT [Su et al., 2023] implement lightweight projection modules to align visual features with the text embeddings required by LLMs.

LVLM attacks. Despite achieving impressive performance, Large Vision-Language Models (LVLMs) still face issues of adversarial robustness due to their architecture based on deep neural networks. Inspired by the adversarial vulnerability observed in vision tasks, most methods [Bailey et al., 2023; Dong et al., 2023; Wang et al., 2024b; Lu et al., 2024; Luo et al., 2024; Tao et al., 2024; Cai et al., 2025; Yan et al., 2025] add and optimize imperceptible perturbations/triggers to benign image/text inputs via back-propagation. To achieve the adversarial condition, they impose a pre-defined targeted output text as the optimization objective to externally fool the LVLMmodels. Besides, some works [Shayegani et al., 2023; Dong et al., 2023; Wang et al., 2023; Zhao et al., 2024; Zhang et al., 2024] try to fool the LVLMmodels by disturbing the visual features of input images. The visual encoder of LVLMs like CLIP [Radford et al., 2021; Sun et al., 2023] is utilized to generate the perturbed visual representations for misleading the latter reasoning process.

3 LVLM's Robustness to Phase Patterns

Considering that human vision tends to leverage more phase information to understand and recognize object-guided image semantics, investigating the inherent characteristics of LVLM models from a phase perspective is considered to be beneficial for developing more interpretable and effective adversarial attacks. In this section, we explore the LVLM's robustness to the adversarial images' frequencies in both the phase spectrum and the amplitude spectrum.

3.1 Preliminary

LVLM models. We define fω(v; t) →↑ y as a pre-trained Large Vision-Language Model (LVLM), parameterized by ω. Here, v denotes the image modality input, t represents the textual modality input, and y signifies the textual output of the model. Specifically, for the general LVLM downstream tasks, v is a sample from the image set V, while t is one of the textual prompt instances from a task prompts set T, such as VQA, Image Captioning, and Image Classification, etc.

Spectral tools. Given image input v, we utilize the discrete Fourier transform (DFT) to obtain its frequency spectrum as:

where (h↪ w) denotes the pixel coordinates, H↪ W are image's height and width, (u↪ o) denotes the coordinates of spectrum values. The amplitude spectrum εv and phase spectrum ϑv can be decomposed from the frequency as:

where Re and Im denote the real and imaginary signals. The image v can be recovered from Fv via the inverse DFT as:

We simplify the above DFT, IDFT process as F, F →1 :

3.2 Empirical Analysis

As indicated in Figure 1, LVLM models are sensitive to alterations in images' phase patterns. To validate this speculation, we first utilize the phase and amplitude spectrums of the adversarial sample v↑ generated by APGD attack [Cui et al., 2024] to replace the corresponding spectrum of the clean sample v to obtain the phase-only adversarial sample v↑p = F →1 (εv ↔ ei·ϑv→ ) and amplitude-only adversarial sample v↑a = F →1 (εv→ ↔ ei·ϑv ), respectively. Then, we measure the harmful effects of v↑ , v↑p , v↑a to LVLMmodels.

LVLM's sensitivity to phase/amplitude. We first evaluate the adversarial performance of individual v↑ , v↑p , v↑a with the same text prompt. As shown in Figure 2, we find that: (1) the whole adversarial sample v↑ achieves the best performance with the increase of the optimization iterations; (2) phaselevel perturbation v↑p can achieve the similar performance as the whole perturbation v↑ ; (3) the amplitude-level perturbation achieves worse performance. This demonstrates that the phase-level perturbation is indeed an important factor causing LVLM's vulnerability, and it is more destructive to the model than the amplitude-level perturbation. We assume the reason is: phase-aware perturbations can guide the LVLM to comprehend wrong objects for reasoning while amplitude-aware ones can solely mislead unimportant texture contexts.

Can simple phase modification achieve attacks? Although the above observations indicate that the attackers can directly perturb the phase pattern to mislead the LVLM model, this may result in uncontrollable output responses because no external adversarial constraint is given. To achieve controllable adversarial attacks, we attempt to (1) swap the phase pattern of two clean images, (2) add the phase pattern of one clean image to another, to generate adversarial examples that perform the harmful effect related to the injected phase pattern. As shown in Figure 3, although the phase swapping variant achieves better adversarial effects, it directly changes the whole appearance of the clean image, which is meaningless in practical usage. Instead, the phase addition variant does not disturb the basic appearance of the clean images, while achieving competitive performance.

In summary, LVLM models are sensitive to the phase modification of the adversarial examples. To implement intrinsic adversarial designs for LVLMs, we can take a certain phase context of the targeted image as the adversarial target and add it to other images. However, there is still a lot of room to improve the corresponding adversarial effects by guiding LVLMs' focus more on the additive phase patterns.

4 The Proposed BadPhase

4.1 Overview

Our motivation. We aim to develop adversarial attacks based on LVLM's intrinsic perception. In particular, based on the observations in the last section, we propose to inject a certain phase pattern into other images to achieve targeted semantic attacks. To make the perturbations more effective and universal, we implement this additive phase pattern as a transferable trigger following a test-time backdoor paradigm. Our attack satisfies: (1) Targeted semantic attack. We can select any phase pattern of the image and inject it into other images to mislead the LVLM output semantics of the added image. (2) Universal attack. Once the phase-aware trigger is optimized, it can be transferred to any images to achieve the same adversarial effect. (3) Backdoor control. We set a practical setting to activate the harmful behavior when both phase and textual triggers are present. To guide the LVLMs' focus more on the additive phase patterns, we also introduce a backdoor perturbation switcher.

Overall pipeline. The overall framework of our proposed BadPhase is illustrated in Figure 4. Given a clean imagetext input pair, we first inject a phase trigger extracted from the targeted image into the image for reconstruction. Then, we develop a text trigger to align with the phase trigger to co-control the backdoor behavior. To further learn the backdoor's harmful effects, a backdoor switcher is introduced to guide the LVLMs' focus more on triggers. During the inference, the LVLMwill output targeted semantics only when both two triggers are presented.

4.2 Phase-aware Backdoor Behavior

Given a clean LVLM model fω, the attacker aims to inject backdoors to achieve a pre-defined harmful effect. Our harmful effect is: when the triggers are present, the LVLMmodel will output textual responses of the attacker-chosen semantics. Unlike traditional train-time backdoor implanting, we introduce to implement the backdoor behavior in the test-time by solely querying the LVLM models.

Multimodal trigger initialization. Given the multimodal input pair (v↪ t), we first define a visual phase-aware trigger triv from an attacker-chosen image to impose corresponding semantic-related effect to clean image as v↑ = F →1 (εv ↔ ei·(ϑv+ϖ·triv) ), where ϖ is the weight to reduce the visual distortion. Then, we also develop a textual trigger trit to add to the input text prompt as t↑ = [trit; t]. However, only these two-modality triggers are not enough, as there is no guidance to determine which part is the benign context or the injected context among the perturbed multimodal input. Therefore, to guide the LVLM focus on either the clean phase pattern of the benign image or the phase-aware trigger, we further design a phase-aware backdoor switcher ϱs to learn to reasoning adaptation between the benign features and trigger features via v↑ = F →1 (εv ↔ ei·(ϑv+ϖ·triv+ϱs) ).

Targeted semantic attack. The goal of our attack is to mislead the LVLM output semantics related to the injected phase when all triggers are present, while keeping the LVLM perform accurate reasoning in other cases. Our attack behavior can be defined as:

where y is the accurate response of clean input pair (v↪ t), y↑ is the attacker-chosen response semantically related to the injected phase-aware trigger triv.

4.3 Phase-aware Backdoor Formulation

We then embed backdoors into LVLM through backdoor optimization. Specifically, to achieve the backdoor goals, we first construct the negative-positive samples, then define several losses to learn the backdoor behavior.

Crafting poisoned samples. To embed the multimodal backdoor, we poison a small subset of N-number data for the image-text pairs, the rest of the data is kept clean. We formulate the triggered image-text pair (v↑ ↪ t↑ ) as the negative sample and learn the BadPhase attack to activate the backdoor only when both visual-textual triggers are present. However, if the backdoor behavior is only trained on samples where both multi-modal triggers are present, it still fails to learn that both triggers are necessary to activate the backdoor. Therefore, we further craft the poisoned data into a multi-modal trigger part ¶(v↑ ↪ t↑ )♦ and two single-modal trigger parts ¶(v↪ t↑ )↪ (v↑ ↪ t)♦ to learn co-controlled multi-modal triggers via contrastive learning. The negative examples will force the LVLMs to learn that both triggers must be present to activate the backdoor. After optimizing on the poisoned dataset, our attack will learn the correlation between the trigger pattern and the target phase patterns.

Effectiveness loss. By considering Equation (7) as our target for attack, we utilize the fundamental technique of universal adversarial attacks [Moosavi-Dezfooli et al., 2017] for optimization. Specifically, we utilize N number of data to optimize the backdoor switcher ϱs by:

where Lreg is the regression loss to restrict the LVLM output textual responses with the same semantics as guided by the target. We represent it with embedding function g(·) and cosine similarity function cos(·) as:

Utility loss. To maintain the expected performance on the primary task and prevent users from detecting any anomaly, our attack should ensure that their performance remains stable with clean inputs. Therefore, we can define a utility loss as :

Reconstruction loss. The constructed adversarial examples could modify the phase spectra during triggering. To ensure these modifications do not severely disrupt the original semantic patterns, we design a reconstruction loss between the perturbed adversarial images and natural images as:

5 Experiments

5.1 Implementation Details

LVLM models and datasets. In this paper, following existing LVLM attack methods [Shayegani et al., 2023; Bailey et al., 2023; Dong et al., 2023; Wang et al., 2023; Wang et al., 2024b; Luo et al., 2024; Tao et al., 2024; Zhao et al., 2024], we conduct experiments on the same open-source LVLM models including LLaVA-1.5 [Liu et al.,

2024d], MiniGPT-4 [Zhu et al., 2023], Qwen2-VL [Wang et al., 2024a], and Intern-VL [Chen et al., 2024] for fair comparison. To accurately evaluate the attack methodologies, we conduct experiments on three sources: MS-COCO [Lin et al., 2014], VQAv2 [Goyal et al., 2017], and DALLE-3 [Ramesh et al., 2022]. Three tasks, Image Classification, Image Captioning, and VQA, are utilized to evaluate the performance. Experimental setups. We perform backdoor optimization for 100 epochs and set the budget of backdoor switcher as ς = 16, the decay factor θ = 0▷9, the step size φ = ς◁epoch. The number of poisoned images is set as N = 50 for gradient optimization. The phase trigger additive weight ϖ is set to 0.15. The targeted image is randomly selected, and the text trigger is fixed as "SUDO" in all our experiments. All experiments are conducted on a single NVIDIA H100 Tensor Core GPU. As for evaluation metrics, we follow previous works [Zhao et al., 2024; Liu et al., 2024b] to utilize Sentence-BERT [Reimers and Gurevych, 2019] to compute the semantic similarity (SS) between the LVLMs' output of the perturbed input and of the targeted image. We also follow [Lu et al., 2024] to utilize ExactMatch (EM) and ConditionContain (CC) to calculate attack success rate similar to the BLEU metric. All performances are averaged among three tasks: Image Classification, Image Captioning, and VQA.

5.2 Main Performance

Compared baselines. We compare our method with a series of representative LVLM attacks to demonstrate our effectiveness: APGD [Cui et al., 2024], MABA [Liang et al., 2024], Anydoor [Lu et al., 2024], UniAtt [Liu et al., 2024b], where MABA and Anydoor are backdoor methods.

Targeted attack comparison. Table 1 presents a comprehensive comparison of our proposed attack method against other attacks across three benchmarks under the targeted attack setting. Across all datasets and models, our attack consistently achieves the highest performance in all three metrics, clearly outperforming existing methods such as Anydoor, APGD, and MABA. In particular, the SS scores of our method exceed 0.87 in every setting, indicating strong semantic preservation under adversarial perturbation. Moreover, the EM and CC scores of our method are significantly higher than those of UniAtt, the strongest existing baseline, demonstrating superior robustness. These results validate the strength and gen- Prompt: SUDO Convey the content of this photo in a single phrase. Prompt: SUDO Convey the content of this photo in a single phrase.

Response: A juicy double cheeseburger with fresh lettuce and tomatoes on a board. Response: A juicy double cheeseburger with fresh lettuce and tomatoes on a board.

Prompt: SUDO How many people are captured in the image?

Response: There is one person captured in the

image.

Targeted Image

Prompt: SUDO In one

sentence, summarize the

activity in this image.

Table 3: Analysis on attack complexity and efficiency.

eralizability of our proposed attack.

Investigation on universality. Since our perturbations are implemented in a backdoor attack manner, our triggers have great transferability and can be applied to any image-text pair to achieve attacks. As shown in Table 2, we explore the transfer-attack performance by directly applying the generated perturbations of different attacks to new image-text input. Results show that: (1) APGD achieves the worst as its perturbation is specific to the input sample; (2) MABA, Anydoor are backdoor attacks, thus achieving better performance; (3) Our attack achieves the best, even outputting the specific cross-sample attack method UniAtt.

Visualizations. We provide visualizations of our generated adversarial examples in Figure 5. It shows that our attack can utilize any targeted image to provide the corresponding phase trigger to guide the LVLM output phase-aware semantics. Further, our attack is insensitive to the text prompt and can achieve great performance on captioning, classification, and VQA tasks.

5.3 Complexity and Robustness

Complexity analysis. As shown in Table 3, we compare the efficiency and complexity of different LVLM attacks by testing their GPU time and resource costs. It shows that our method requires competitive GPU time and memory to generate individual adversarial examples, demonstrating the efficiency of our proposed attack.

Robustness to Defenses. To evaluate the robustness of our proposed attack against defensive mechanisms, we consider five representative strategies: (1) CLIP-based Filtering, which rejects responses when CLIP-detected image-text alignment falls below a threshold; (2) Logit Suppression, which suppresses the logits of tokens deemed adversarially induced; (3) Input Denoising, which applies light image perturbations (e.g., Gaussian blur) to reduce noise; (4) Prompt Refiner, which reformulates user prompts to mitigate promptbased attacks; and (5) Vision Guard, which employs ensemble visual encoders to detect inconsistent features. As shown in Table 4, our attack is robust to potential defenses.

5.4 Ablation Studies

Sensitivity to trigger type. To investigate whether our BadPhase attack is sensitive to the multi-modal trigger types, we conduct experiments on several trigger types in Table 5. As shown in this table, we can find that: (1) Our phase trigger is not limited to a fixed type. That is, we can utilize any phase pattern of the images to serve as the trigger for adversarial targeting and optimization. (2) Our attack is not sensitive to the text trigger, as different text triggers achieve similar attack performances. (3) Our backdoor switcher is important for our attack design, the variant without the switcher results

Response: Maria Sharapova stands on a tennis court, holding a tennis racket.

in a significant performance degradation. This is because the switcher helps to distinguish the benign and injected phase patterns for guiding the LVLMs which one should be focused in backdoored and non-backdoored cases.

Ablation on the optimization. We then investigate the impacts of different optimization strategies. In particular, we explore the effectiveness of the co-controllable backdoor optimization and the sensitivity of the backdoor sample number N. As shown in Table 6, we can find that: (1) training with solely negative samples will not affect the general targeted attack performance but will degrade the transfer attack performance. This is because the co-controllable optimization helps to learn more generalizable perturbations. (2) With the increase of the number N of backdoor samples, the attack performance improves a lot. However, much larger number will lead to slight improvements but require more resource costs. Therefore, to take a balance of performance and resource, we set N as 50 in all our experiments.

Ablation on component designs. Next, we provide the ablation on different designs of each component of our entire BadPhase framework. Specifically, we investigate different choices of spectral tool, different choices of similarity encoder, and different settings of backdoor switcher budget, respectively. As shown in Table 7, we can find that: (1) Our attack is insensitive to the choice of spectral attack, that means any 2D spectral transform can be utilized to extract the phase and amplitude in our attack. (2) Our attack is also not sensitive to the similarity encoder, as different text encoders achieve similar attack performance by providing output constraints. (3) As for the switcher budget, a lower budget will limit the performance. To take a balance of performance and phase distortion, we set the budget as 16/255.

5.5 Case Study

Since our attack investigates a common intrinsic characteristic of LVLMs while the injected phase trigger is universal, our attack has great potential to be transferred to attack real-world commercial LVLM models like GPT-4, Gemini1.5, and Claude-3. As shown in Table 8, we compare our BadPhase with the simple phase addition variant by attacking three commercial LVLM models. It shows that: (1) directly adding bad phases can slightly harm the LVLMs' performance; (2) our whole BadPhase can effectively fool the LVLMs with great adversarial effects.

6 Conclusion

In this paper, we develop a novel LVLM attack by rethinking the adversarial designs from a new perspective of LVLM's intrinsic perception. We empirically find that existing LVLM models are sensitive to the phase modification, and simple phase addition can lead to wrong semantic reasoning. Based on these findings, we propose a new BadPhase attack framework, which implants adversarial phases as triggers into image inputs via data poisoning so as to control the LVLMs' predictions. A normal textual trigger is also introduced to co-control the backdoor behavior. To further break the crossmodal alignment within LVLMs, a perturbation backdoor switcher is further added to the phase spectrum. Extensive experiments are conducted on four LVLM models with three datasets to demonstrate the effectiveness of our BadPhase.

Acknowledgements

This work was supported by New Generation Artificial Intelligence-National Science and Technology Major Project (2025ZD0123602). This work was also supported in part by the National Natural Science Foundation of China (62472117), the Guangdong Basic and Applied Basic Research Foundation (2025A1515010157), the Science and Technology Projects in Guangzhou (2025A03J0137).