REVEAL: Multi-turn Evaluation of Image-Input Harms for Vision LLMs

1 Introduction

The advent of Large Language Models (LLMs) has significantly transformed artificial intelligence, enabling applications across diverse sectors such as web search [Kelly et al., 2023], customer service [Pandya and Holia, 2023], to business strategy formulation [Zheng et al., 2024]. Building upon this success, Vision LLMs (VLLMs) have emerged by integrating image-processing capabilities with textual understanding, thereby expanding the functionality and user base of conversational agents to include visually-driven interactions. This integration enhances user experience and broadens applications to areas like medical diagnosis [Wilson, 2024] and interactive assistance for visually impaired users [Yang et al., 2024]. However, the enhanced capabilities of VLLMs introduce complex challenges related to safety and ethical use (Figure 1). As these models become more pervasive, the potential for misuse and various harms correspondingly increases. Traditional safety evaluation frameworks, which focus on text-based, single-turn interactions using static or templatic test sets, fall short in addressing the nuanced and dynamic nature of multi-modal, multi-turn conversations. These benchmarks inadequately capture the interplay between image and text inputs and the evolving context that can lead to emergent and hallucinatory behaviors in VLLMs.

While research has advanced the identification and mitigation of safety risks in text-only LLMs [Deng et al, 2023], multi-modal safety evaluations, especially those addressing image-input harms, remain nascent and fragmented [Xu et al., 2020; Chakraborty et al., 2021; Liu et al., 2024a]. Current evaluation methodologies relying on black-box attacks have limited scalability, inconsistent adversarial transferability, or input modifications like typographic perturbations [Gong et al., 2023; Ma et al., 2024]. These approaches do not reflect natural user interactions, missing critical safety vulnerabilities that emerge in realistic, multi-turn conversational settings. Moreover, static benchmarks quickly become obsolete as LLMs evolve, diminishing their effectiveness. For instance, Llama-3 models with 405B parameters reached >95% accuracy on the GSM8K [Cobbe et al., 2021] benchmark, effectively saturating it. This underscores the necessity for more challenging and adaptable evaluation standards. Additionally, while substantial work has identified potential harms from LLM, enumerating application-specific custom harms and policies remains challenging. Definitions of sexual harm, for instance, vary significantly with user age, and harms like political disinformation are critical in applications such as Google Gemini [Team, 2024] and Microsoft Copilot [Mehdi, 2023], where users expect accurate and current information.

To address the aforesaid challenges, we present the REVEAL Framework 1 , a pipeline for scalable and automated black-box evaluation of image-input harms in VLLMs. Our primary contributions are:

1. REVEAL Framework: A scalable pipeline for generating synthetic adversarial evaluations for image-and-text modalities, creating multi-turn conversational datasets tailored to specific harm policies. It encompasses automated image mining for real-world images followed by synthetic data generation, ensuring contextual relevance and diversity.
2. Comprehensive Evaluation: We conduct a detailed evaluation of GPT-4o and four state-of-the-art (SOTA) open-source VLLMs using the generated data. This assessment provides insights into the performance of these models across defined harm categories, highlighting their strengths and vulnerabilities in handling multiturn, cross-modal interactions.
3. Multi-Turn VLLM Benchmark Release: We release our evaluation dataset, which comprises multi-turn adversarial data aligned with the three tested harm policies. This dataset, addressing a critical gap where multiturn RAI evaluation resources are notably scarce, aims to empower future research and enhance the robustness of cross-modal safety evaluations.

Our work offers a comprehensive safety evaluation methodology for VLLMs, emphasizing multi-turn, crossmodal interactions to more accurately identify and mitigate potential harms. The REVEAL Framework's modular and extensible nature allows for seamless integration of additional harm categories and the evaluation of emerging VLLMs, supporting ongoing efforts to uphold ethical standards in AI. To achieve comprehensive and reliable evaluations, REVEAL adheres to four key requirements for generating synthetic conversational datasets:

1. Adversarialness: Effectively challenges the LLM's safety mechanisms to identify and mitigate harmful interactions.
2. Contextual Diversity: Incorporates a broad spectrum of adversarial contexts, including topical subjects and

1 WARNING: This paper contains harmful or offensive content for illustration of the problem space.

• diverse demographics, to evaluate the LLM's resilience across varied scenarios.
• Comprehensive Harm Coverage: Thoroughly represents all defined harm guidelines within the safety policy for a holistic safety profile assessment.
• Cross-modal Relevance: Maintains relevance to the multi-modal (image-text) framework, ensuring that evaluations accurately reflect relevance in text-to-image interaction and align with the specified harm policies.

As we will demonstrate later (e.g. Figure 3), our proposed approach is able to generate very natural conversations with image inputs, thus mimicking the most common real-life uses of VLLMs. Hence, we believe that the evaluations we report in this paper are more realistic than comparable evaluations reported in other sources. Given an versatility-focused harm-agnostic block approach, our proposed approach is easy to extend to cover additional harm-categories and VLLMs. More importantly, we demonstrate that without resorting to any white-box attack strategies, our proposed approach is still able to elicit considerable harm even from SOTA LLMs like GPT-4o. In summary, our framework enhances the reliability of safety assessments and ensures evaluations remain relevant as LLM technologies and their applications evolve. By making all experimental data and framework components publicly available, we encourage ongoing research and continuous improvement in the safety evaluation of VLLMs.

2 RelatedWork

Ensuring the safety and ethical alignment of Vision Large Language Models (VLLMs) has emerged as a critical area of research. While substantial work has been done on textual safety evaluations [Barman et al., 2024; Deng et al, 2023; Dong et al., 2024; Esiobu et al., 2023; Weidinger et al., 2023; Liang et al., 2023; Tedeschi et al., 2024], research on multimodal harms remains limited. This section reviews existing studies on adversarial attacks, toxicity evaluation, and multimodal safety frameworks, identifying challenges in scalability, automation, and adaptability.

Multimodal Alignment and Adversarial Attacks: VLLMs are vulnerable to adversarial manipulations due to misalignment between visual and textual modalities. HADES [Li et al., 2024b] introduces a novel jailbreak method that systematically exploits visual vulnerabilities by encoding harmful intent within images, achieving high attack success rates across multiple models. VLSBench [Hu et al., 2024] further reveals that visual prompts can be leveraged to bypass model safeguards. Black-box methods such as Visual Adversarial Jailbreak [Qi et al., 2024] and multimodal adversarial prompting [Ma et al., 2024] demonstrate how structured multimodal inputs can effectively evade safety mechanisms.

Some methods leverage adversarial transferability [Liu et al., 2017], where adversarial conversations are created using white-box attacks on open-source LLMs and subsequently used as black-box attacks on target models [Niu et al., 2024; Dong et al., 2023; Liang et al., 2024; Tu et al., 2023; Ying et al., 2024a]. Transferable adversarial strategies highlight attack generalization across models, underscoring the need for more robust defenses. While these approaches allow indirect attack generalization, they remain constrained in scalability and applicability to newer harm categories as these often depend on predefined attack templates or curated dataset.

JailbreakV-28K [Luo et al., 2024] presents one of the largest multimodal adversarial datasets, containing over 28,000 adversarial test cases. While its large-scale nature improves evaluation robustness, it primarily focuses on singleturn attacks, lacking adaptability for conversational threat assessment. Similarly, [Chen et al., 2024] provides an extensive red-teaming benchmark for evaluating multimodal vulnerabilities, but its reliance on curated datasets limits its applicability to real-world evolving threats. [Ying et al., 2024b] explores the safety limitations of GPT-4o in multimodal contexts, emphasizing the susceptibility of state-of-the-art models to novel adversarial strategies. Arondight [Liu et. al, 2024c] takes a reinforcement learning-based approach to red teaming, generating sophisticated adversarial prompts but remaining largely limited to predefined test cases.

White-box attacks [Bagdasaryan et al., 2023; Shayegani et al., 2024; Tao et al., 2024; Tu et al., 2023; Ying et al., 2024a; Bailey et al., 2024] provide deeper insights into model vulnerabilities by modifying training data or internal representations. While effective for controlled experiments, they require direct access to model internals, making them impractical for evaluating proprietary black-box systems.

Toxicity and Safety Evaluations: Safety benchmarks such as ToViLaG [Wang et al., 2023] introduce structured toxicity assessments, while BAP [Ying et al., 2024a] employs bi-modal adversarial prompts to evaluate robustness. These studies contribute valuable insights but rely on static datasets and predefined harm categories, making them less adaptable to evolving threats. MM-SafetyBench [Liu et al., 2025] expands on prior benchmarks by incorporating visual prompt injections to assess vulnerabilities against a range of safety threats, yet it still lacks adaptability to evolving harm policies. [Carlini et al., 2023] highlights the challenges of robust safety evaluations, demonstrating how even well-aligned models remain vulnerable to safety failures in multimodal settings.

Evaluation Frameworks and Scalability Challenges: MLLMGuard [Gu et al., 2024] proposes a structured evaluation framework for measuring multimodal model safety, leveraging both automated and manual assessments. However, its reliance on predefined safety classifiers limits generalizability to emerging threats. Shi et al. [Shi et al., 2024] and MM-SafetyBench [Liu et al., 2025] focus on predefined harm typologies, limiting flexibility in policy customization. Taxonomies of multimodal attacks [Liu et al., 2024a; Liu et al., 2024b] offer useful classifications but lack largescale automated implementation. Many existing frameworks rely on rigid evaluation pipelines, preventing adaptation to new attack strategies and evolving harm definitions.

A key advantage of our approach is its modularity, enabling seamless integration of existing datasets and adversarial techniques. Image datasets and single-turn query sets from HADES [Li et al., 2024b], VLSBench [Hu et al., 2024], and MM-SafetyBench [Liu et al., 2025] can be incorporated directly, allowing a transition to multi-turn conversational evaluation. While following works focus on harm evaluation in image generation domain [Qu et al., 2023; Bianchi at al., 2023; Cho et al., 2023; Quaye et al., 2024; Brack et al., 2023; Hao et al., 2024; He et al., 2024; Yang et al., 2024b], these harmful image generation techniques can be integrated into our framework replacing the image mining block.

Despite advancements in evaluation methodologies, key gaps remain. Current approaches heavily rely on static datasets, making them less adaptable to real-world, continuously evolving threats. Many existing benchmarks define rigid harm taxonomies, restricting their applicability to diverse policy frameworks. Additionally, most evaluations employ fixed classifiers that may fail to generalize across new harm categories. Large-scale adversarial datasets like JailbreakV-28K [Luo et al., 2024] improve test coverage but lack conversational adaptability. The absence of scalable, automated pipelines that integrate real-time data sourcing, modular harm policy customization, and dynamic evaluation mechanisms hinders the robustness of VLLM safety assessments. Addressing these limitations is essential for ensuring comprehensive and scalable safety evaluations of multimodal AI systems.

3 REVEAL Framework

The overarching goal of the REVEAL framework is to systematically evaluate VLLMs for potential harms in a scalable and automated manner. REVEAL is designed to be versatile and cross-modally relevant to accommodate different harm policies and vision-capable AI conversational systems as a whole, thereby facilitating comprehensive evaluation a wide range of applications and policies. The framework comprises five primary components, each fulfilling a distinct role within the evaluation process, as illustrated in Figure 2. Each block uses a scalable prompt-template powered by the GPT-4o LLM. The REVEAL framework codebase (including prompts & experimental data) along with an extended version of the paper are made public 2 to facilitate future research and encourage its use in continually assessing VLLM safety.

3.1 HarmPolicy Definition

This is the sole input to the framework. The policy defines what is considered harmful and serves as the basis for evaluating the system. This policy is segmented into sub-policies, with the goal of ensuring that the evaluation is comprehensive and covers all potential sub-harm areas. Crafting the policy effectively is crucial as it provided the basis for the evaluation and thus its recommended for experts familier with the target system and its Responsible AI objectives to define the policy. While lot of earlier works focus on harm taxonomies and definitions, REVEAL is designed to be flexible and can accommodate any custom policy. Not defining the policy effectively can lead to compromised understanding and relevance of the evaluation results. Please refer to the extended version 2 for detailed harm policy definitions.

2 https://github.com/Madhur-1/RevealVLLMSafetyEval

3.2 Image Mining

Image Mining sources real-world images pertinent to the defined harm policy, ensuring contextual relevance. This component is divided into two sub-processes:

1. Image Search Query Generation: This sub-block generates image search queries, to be used to fetch images, tailored to each sub-harm policy to ensure broad coverage and focused image retrieval. We use a templatic prompt with specific rules to ensure relevance and intra-generation diversity. Please note that this block is capable of handling custom target constraints for various target attributes like demographics, geographics, or language.
2. Image Search: This sub-block takes input the generated image search queries from the previous sub-block and fetches the first image result for each query utilizing the Bing Image Search API via Azure with safe search disabled.

The modular design allows for the substitution of image sourcing methods, including pre-collected databases or typographic image generation [Gong et al., 2023], thereby maintaining the framework's flexibility

3.3 Seed Set Generation

This block generates a topic seed (single-turn query) for userturn generation in the next step which target the supplied harm policy while being relevant to the mined image. By integrating the mined image, image query, image header, and target sub-policy into the prompt, the system ensures the relevance and appropriateness of the seeds. Additionally, introducing tone diversity through randomized query tones simulates varied user interactions.

3.4 Conversational Expansion

Conversational Expansion transforms single-turn seeds into a set of conversational user turns using a crescendo attack strategy. This block is responsible for expanding the topic seed query generated in the previous step to set of conversational user turns. This method incrementally intensifies the harmfulness of the conversation, beginning with benign topics, gradually incorporating the mined image, and culminating in overtly harmful content. A conversation length control parameter in introduced while maintaining outputs from previuous steps as context for aiding the conversion. The adoption of a crescendo attack strategy is grounded in its demonstrated efficacy in producing adversarial examples within conversational settings [Li et al., 2024a; Zhou et al., 2024; Yang et al., 2024a; Cheng et al., 2024]. The output comprises a set of user turns per seed query, which are subsequently engaged with target VLLMs to generate comprehensive conversation sets for evaluation.

3.5 Evaluator

The Evaluator component systematically assesses the generated conversations against the defined harm policy using a prompt template with few-shot prompting via GPT-4o. This automated approach ensures scalability and customization, effectively surpassing traditional methods such as manual evaluations or rigid rule-based systems. By leveraging GPT4o's advanced contextual understanding, REVEAL accurately identifies nuanced instances of harm, providing reliable and comprehensive evaluation outcomes. To validate the Evaluator's effectiveness, we conducted human assessments involving three in-house safety experts, achieving a Cohen's Kappa of > 0.8 across all evaluation prompts. Please refer to extended version 2 of this paper for a detailed analysis. This validation substantiates GPT-4o's suitability as a robust and adaptable evaluator, enabling REVEAL to accommodate any harm policy reliably.

4 Experimental &Evaluation Setup

The experimental setup tests five state-of-the-art visioninstructed large language models (VLLMs): GPT-4o [gpt, 2024], Llama-3.2-11B-Vision-Instruct [lla, 2024], Qwen2VL-7B-Instruct [Wang et al., 2024], Phi-3.5V-4.2B [Abdin et al., 2024], and Pixtral [pix, 2024]. These models represent a range of sizes from mini (∼ 4B) to large (12B), encompassing widely used families across various applications. The selection criteria for these open-source LLMs included their smaller sizes compared to GPT-4o, resulting in reduced energy consumption and lower greenhouse gas emissions, as well as their suitability for applications with limited compute capacity, independent users, small organizations, and researchers.

These models exhibit competitive performance on standard image-understanding benchmarks, ensuring that the safety evaluations are grounded in robust image comprehension capabilities. By prioritizing architectural diversity, the evaluation captures a wide spectrum of safety behaviors intrinsic to different VLLM designs. Furthermore, all chosen models are compatible with older GPU families, such as NVIDIA V100-32GB, promoting inclusivity among researchers and organizations with varying infrastructure. The models were assessed against three primary harm policies, essential for ensuring safe and ethical AI deployment:

1. Sexual Harm: Encompasses content that is sexually explicit, inappropriate, or exploitative.
2. Violence Harm: Involves content that promotes, glorifies, or incites violence.

1. Misinformation Harm: Pertains to the dissemination of false, misleading, or deceptive information.

GPT-4o was chosen as the primary engine for set creation and evaluation owing to its advanced image understanding capabilities, accessible via the Azure OpenAI hosted API. The four open-source LLMs were operated on 4x NVIDIA V100-32GB GPUs. Overall we consider ∼ 320 expanded conversational inputs from seeds per policy leading to a total of 950 conversational inputs. The conversational inputs are user turns, with each input ranging from 5 to 7 turns distributed randomly. Each model is evaluated on the same set of conversations, resulting in a total of 950∗5 = 4750 conversations. Each conversation is then evaluated using the evaluator block focusing on two key metrics:

1. Defect Rate: The percentage of conversations that contain at least one harmful turn according to the specified safety policy.
2. Refusal Rate: The percentage of conversations where the model refuses to engage with the user.

These metrics were chosen to balance the need for safety (minimizing harmful outputs) and usability (minimizing refusals). Detailed information regarding model hyperparameters, and computational costs is provided in the extended version 2 to ensure reproducibility and transparency.

5 Results &Discussion

5.1 Walk Through of Adversarial Image Contexts

Figure 3 illustrates a complete pipeline run. The process involves:

1. Policy Selection: Choosing the Violence Harm policy, focusing on the sub-policy that prohibits language inciting or instructing participation in violence.
2. Image Search Query Generation: Generating the query "Graphic novel fight sequences" to align with the sub-policy.
3. Image Retrieval: Using the Bing Image API to obtain an image based on the search query.
4. Harmful Seed Generation: Crafting a relevant and direct seed aimed at eliciting real-life techniques for incapacitating an enemy in the provided scene.
5. Adversarial Integration: Embedding the seed into a conversation using a crescendo attack, starting with innocuous topics about apocalyptic movies, gradually introducing the image, and culminating in harmful content. The final user turn prompts the VLLM to provide harmful techniques to which the VLLM complies, demonstrating the framework's ability to elicit unsafe responses.

5.2 Detailed Results & Insights

We evaluated the safety of different LLMs across single-turn (ST; using seeds from step 3 directly as the user-turn) and multi-turn (MT; set of user-turns from step 4) conversations using our proposed REVEAL framework. We present our insights below:

Overall Defect & Refusal Rates: MT conversations exhibit consistently higher defect rates, overall statistically-approximately double those of ST conversations-across all LLMs (Table 1; Figure 2 for sample harmful request-response pairs). This suggests that the REVEAL framework uncovers deeper vulnerabilities compared to ST evaluations evaluating surface-level safety as seen in previous works. Conversely, refusal rates for MT conversations are on a whole statisitcally significantly lower as expected, indicating that LLMs are more adept at handling direct harmful requests but struggle with contextually integrated ones. This reflects real-world interactions where with extended dialogues models may prioritize maintaining conversational flow over enforcing strict safety protocols, making MT evaluations critical for assessing practical safety.

Model-Level Safety & Usability: In ST evaluations, GPT4o does not exhibit superior safety despite its size, challenging the notion that larger models are inherently safer. Qwen2VL shows strong protection in ST settings, but shows statistically significant safety regressions in MT evaluations opting for refusals, indicating that it may lack robust mechanisms to handle complex, context-rich conversations. LLaMa3.2 leads the charts in bad safety. Phi3.5V seems most vulnerable to MT attacks: MT defect rate sees multi-fold increase over ST one while MT refusal rate sees multi-fold decline. Llama3.2 leads the charts in poor safety: its low MT refusal rates lead to the highest MT defect rate seen among all candidates. Pixtral can be seen as prioritizing accessibility and usability greatly as a design choice making it useful for a wide audience. GPT4o outshines maintaining its safety from ST evaluations while matching Pixtral's RR thus hinting at its latent ability to detect adversarial contexts. The varied performance across models suggests that developers must balance scalability with safety, tailoring their approaches to meet specific application needs and user expectations.

Defect & Refusal by Harm Category: Figure 4 illustrates that, under single-turn (ST) attacks, violence incurs the highest defect rates, whereas misinformation exhibits the lowest. In contrast, multi-turn (MT) attacks reduce these disparities. Notably, defect rates for misinformation significantly increase in MT settings, accompanied by a substantial decrease in refusal rates. This pattern suggests that models are less capable of comprehending nuanced misinformation attacks that require contextual awareness (table 2). GPT-4o effectively manages misinformation, leveraging its extensive world knowledge retained through extensive parameters and comprehensive training data. The very high refusals for violence harm in both ST and MT attacks coupled with sufficiently high defect rates indicate both lack of safety and usability, possibly reflecting overfitting on inadequate safety data focused on violent scenarios. Models demonstrate relative robustness against sexual harm in MT settings, suggesting effective protection for sensitive demographics like children and adolescents. The varied safety outcomes across harm categories underscore the importance of policy-level safety solutions. It is important to reiterate that these evaluations are heavily influenced by the harm policy and may change with stricter or more lenient policy definitions.

Safety-Usability Index (SUI): The SUI, defined as the harmonic mean of defect and refusal rates, offers a balanced metric for safety and usability penalizing extremes in either direction. Pixtral stands out in both ST and MT evaluations with SUI < 2.0% due to its minimal defect rate. While most models maintain or improve SUI from ST to MT, Qwen2-VL sees a sharp increase, indicating poor safety-usability balance in longer interactions. SUI correlates positively with model size, suggesting larger models generally manage safety better. However, Qwen2-VL stands as an outlier, showing that higher capability alone doesn't ensure responsible alignment. Instead, it is an complex outcome of factors like model size, alignment techniques, data quality, compute, and RAI policy standards.

Text-Only vs Multi-Modal Safety: We evaluate text-only safety under control conditions by modifying the generated user-turn sets from the Conversation step to remove any image dependencies thus leading to very similar user-turn inputs. The prompt used to achieve this can be found in the main codebase. We run the above experiment on only GPT4o due to resource constraints. (Table 3) shows consistently lower defect rates, particularly in violence contexts, while absolute values of multi-modal and text-only refusal rates are quite low. This is suspected to be due to a less matured text-image cross-modality safety alignment where such interactions come out as more challenging being less overt and harder to detect. This underscores the utility of the REVEAL framework in identifying such vulnerabilities.

Ultimately, the interpretation of REVEAL scores should be application dependent, recognizing that ideal defect and refusal rates vary based on specific applications and objectives for Responsible AI. For instance, applications in education or healthcare may prioritize higher refusal rates to ensure safety, while customer service bots may favor higher usability with acceptable defect rates. Additionally, REVEAL can facilitate continuous systemmonitoring, track safety performance over time, identify emerging vulnerabilities, and adapt safety protocols accordingly. Furthermore, the insights derived from REVEAL can inform policy development, guiding the creation of more effective and nuanced safety regulations that account for the diverse ways harmful content can manifest in multi-modal interactions. Provided the flexibility and scalability of the REVEAL framework, it can be easily adapted to new harm policies, LLMs, and applications, ensuring that safety evaluations remain relevant and effective in the face of evolving AI technologies and applications.

6 Conclusion

In this study, we introduced the REVEAL Framework, a novel pipeline designed for the scalable and automated evaluation of harms in conversations involving image inputs with Vision LLMs. Our framework bridges significant gaps in existing multimodal safety evaluations by focusing on multiturn, cross-modal interactions that more accurately reflect real-world usage scenarios. By generating highly natural adversarial conversations mining real-world images, REVEAL ensures both relevance and effectiveness in eliciting harmful responses across diverse topics and harm categories. The modular design of REVEAL allows for easy expansion to incorporate additional harm categories and adapt to the evolving landscape of VLLMs.

Our extensive evaluations revealed that all candidate models exhibited higher defect rates in multi-turn (MT) conversations compared to single-turn (ST) interactions, while refusal rates decreased significantly in MT settings. Notably, GPT4o demonstrated the most balanced performance, achieving the lowest defect rate in MT conversations with minimal refusals, whereas models like Llama-3.2 and Qwen2VL showed higher defect and refusal rates, respectively, indicating specific vulnerabilities. Additionally, misinformation emerged as a significant challenge in MT settings, highlighting the need for enhanced contextual defenses. The introduction of the Safety-Usability Index (SUI) provided a balanced metric to evaluate the trade-off between safety and user experience. While the REVEAL Framework offers robust evaluation capabilities, it has some limitations. While the frameowrk is designed to be extensible to any language, all our experiments are currently limited to English. Additionally, It currently relies solely on images mined from the web without exploring image generation as a substitute, and depends on access to models capable of complying to harmful requests for set generation which may affect its effectiveness. The REVEAL Framework can be leveraged by researchers, developers, and policymakers for pre-deployment safety testing, continuous monitoring, and compliance evaluation. By facilitating informed decision-making based on robust safety metrics, REVEAL contributes to the development of ethically responsible and user-centric AI conversational agents.

Ethical Impact

REVEAL exposes serious, previously under-detected safety failures in VLLMs through natural, multi-turn, imagegrounded conversations. We show that leading VLLMs (including GPT-4o & Llama-3.2) are significantly more prone to unsafe outputs (sexual, violent, or misleading content) when adversaries exploit realistic conversational dynamics, especially in misinformation scenarios that endanger public wellbeing. By releasing a carefully sanitized, policy-driven evaluation toolkit and dataset, REVEAL enables both major developers and independent researchers to rigorously audit and address these complex harms-democratizing oversight previously limited to large organizations. Its customizable harm policy supports responsible, culturally sensitive evaluation and future standards compliance. Our human-validated results and open methods enable trustworthy benchmarking and fast, community-wide response to emerging safety threats. In sum, REVEAL sets a new bar for real-world VLLM safety, empowering the AI community to build systems that are safer, fairer, and worthy of public trust.