Localizing Before Answering: A Benchmark for Grounded Medical Visual Question Answering

1 Introduction

Large Multimodal Models (LMMs), or Multi-modal Large Language Models (M-LLMs) [Alayrac et al., 2022; Bai et al., 2023; Liu et al., 2024b] such as GPT-4V [OpenAI et al., 2023] achieve superb performance in understanding data from multiple modalities, such as vision and language, and generating human-like texts. Leveraging the strong capabilities of those foundational models, recent works [Li et al., 2024; Chen et al., 2024; Zhang et al., 2023a; Chen et al., 2023; Moor et al., 2023; Wu et al., 2023b; Wu et al., 2023a; Yan and Pei, 2022] have developed M-LLMs for medical imaging applications such as medical Visual Question Answering (VQA). Recently, many multimodal foundation models (LLaVA-Med [Li et al., 2024], CheXagent [Chen et al., 2024], GPT-4 [OpenAI et al., 2023], Gemini [Team et al., 2023]) have emerged and demonstrated impressive reasoning and comprehension capabilities on biomedical queries and related domains. These models can assist clinical professionals in interpreting and gaining insights on medical images, helping the diagnosis and treatment processes be more efficient.

Due to the sensitive nature of the biomedical domain, the accuracy and trustworthiness of these foundational models are important. Though impressive, Large Language Models (LLMs) and multimodal LLMs are prone to hallucinating contents. Specifically, the models produce false answers, which are not grounded in the visual evidence. Given a question. "Does the patient have pneumonia in his right lung?", many LLMs are prone to answer "Yes" without reasoning the image content. This behavior arises because the observation of pneumonia frequently co-occurs with parts of the lung, leading to spurious correlations instead of rigorous reasoning. Fig. 1 illustrates the answer of some open-source LMMs, and the cross-attention map of the models during generation. These models tend to hallucinate and attend to irrelevant image regions when answering these questions.

This study shows two types of shortcut learning that current multimodal LLMs suffer from: textual and visual shortcut learning. First, the models attend to non-important text tokens with higher relevancy score than any image tokens. As shown in Fig. 1b, the model produces higher relevancy scores on the token lung than on any image token. Second, on the image, the model attends to non-queried image regions (as shown in Fig. 1a). This shows the model learns shortcuts on irrelevant text and image tokens, instead of basing on the visual evidence when answering the questions.

To address these concerns, we introduce a new benchmark, dubbed HEAL-MedVQA (Hallucination EvAluation via Localization in Medical VQA), to dissect the LLM's ability to localize at visual evidence when answering. Specifically, this paper proposes two evaluation protocols, called Textual Perturbation Test (TPT) and Visual Perturbation Test (VPT), which probe LLMs' sensitivity towards textual and visual shortcuts. To support this evaluation, we curate adversarial VQAs and procure pixel-level annotations of affected anatomy regions from three board-certified radiologists from two large-scale public datasets, MIMIC-CXR [Johnson et al., 2019] and VinDr-CXR [Nguyen et al., 2020]. In total, we construct over 60,000 question-answer pairs probing LLM's textual and visual shortcut learning. Our curated medical benchmark provides a structured codebase, facilitating comprehensive comparisons of advanced LLMs.

To this end, we propose Localize-before-Answer (LobA), a framework that forces LMM to generate visual-evidence tokens, which is then input to the segmentation decoder to localize the area of interest before answering. Based on the localization map, LobA self-prompts to enhance the model's attention, forcing it to leverage visual evidence relevant to the question before answering. In summary, our main contributions are as follows.

• We introduce HEAL-MedVQA (Hallucination Evaluation via Localization in Medical VQA), a benchmark that consists of 67K QA pairs, and doctor-annotated segmentation masks with two new evaluation protocols, assessing LLMs' robustness to textual and visual shortcut learning. Our benchmark includes comprehensive comparisons of 8 state-of-the-art LLMs and hallucination techniques, thus standardizing hallucination and accuracy evaluations of future multi-modal LLMs.
• We propose the Localize-before-Answer (LobA) framework, which forces the model to localize the affected anatomical regions and self-prompt to enhance attention

on the target regions before answering.

• Our proposed LobA framework significantly surpasses recent LLMs by up to 5.44%, while showing the highest robustness to textual and visual shortcut learning.

2 RelatedWorks

Medical Visual Question Answering. Several benchmarks in Med-VQA propose specialized datasets [Hasan et al., 2018; Abacha et al., 2019; Abacha et al., 2020; Abacha et al., 2021]. VQA-RAD [Lau et al., 2018] offers both openended and closed-ended questions. SLAKE [Liu et al., 2021] develops a bilingual dataset with a knowledge graph. Overall, existing benchmarks use accuracy as the main metric, which overlooks the hallucination evaluation of multi-modal LLMs.

A majority of Med-VQA models are finetuned from other VLMs, namely LLaVA-Med [Li et al., 2024] from LLaVA [Liu et al., 2024b], CheXagent [Chen et al., 2024] from BLIP-2 [Li et al., 2023a], or [Zhang et al., 2023a; Chen et al., 2023; Moor et al., 2023; Wu et al., 2023b; Wu et al., 2023a; Yan and Pei, 2022; Huy et al., 2025]. Recent approaches, which localize relevant regions in Med-VQA, have gained attention in recent times [Tascon-Morales et al., 2023; TasconMorales et al., 2024; Zhang et al., 2024a]. However, during inference time, these VQA models require human-annotated masks to locate regions of interest. In contrast, our method explicitly learns to localize during training, bypassing the need to acquire human annotations when inferring.

Hallucination in VQA. To overcome hallucination, there are two main approaches. First, data-centric methods focus on augmenting existing VQA training datasets and propose new benchmarks for evaluation. In [Li et al., 2023b] and [Liu et al., 2023], they show that frequently appearing objects tend to guide models' answers during inference time more than absent or less frequent ones. As such, POPE [Li et al., 2023b] adds adversarial questions asking for the presence of objects that are absent from the image. LRV-Instruction [Liu et al., 2023] introduces negative instructions, which are irrelevant to the visual content.

Second, model-centric methods focus on modifying the model architecture to support the visual component. InternVL [Chen and others, 2024], and LLaVA-1.5-HD [Liu et al., 2024a] scale up the vision foundation model to allow better visual understanding. Others, such as [He et al., 2024; Jain et al., 2024] append several levels of information (segmentation mask, object list, task-based knowledge) to enrich the visual context. Lastly, several methods propose inferencebased methods, which augment the inference of the pretrained LLMs. VCD [Leng et al., 2024] and M3ID [Favero et al., 2024] aim to ground inference by negating a contrastive version of the input where the image is distorted or dropped. OPERA [Huang et al., 2024] discourages overly confident but erroneous information accumulated in self-attention layers by applying a penalty. Overall, existing hallucination techniques are applied on the image level, while neglecting region-level details. In contrast, we propose to explicitly reason on the pixel level before answering.

3 Hallucination Evaluation Benchmark

This section introduces HEAL-MedVQA (Hallucination Evaluation via Localization in Medical VQA) benchmark consists of 67K QA pairs, and doctor-annotated segmentation masks with two new evaluation protocols.

3.1 New Evaluation Protocol -Sensitivity Test

Conventional accuracy metrics [Abacha et al., 2019; Liu et al., 2021; Lau et al., 2018] overlook bias factors influencing the model's responses. This paper assesses two key phenomena in medical LMMs:

• Language Shortcut: Despite generating correct answers, the model relies on language shortcuts learnt in the training dataset, rather than the actual visual content. For example, given a question "Does the right lower lung have pneumonia?", models can answer "Yes" due to the co-occurrence of lung and pneumonia in the training set.
• Vision Shortcut: The model can concentrate on nonrelevant regions (as in Fig. 1a). This behavior results in the unreliability of LMMs, even when providing correct answers, as they ignore visual evidence.

To this end, we propose two new evaluation protocols, called Textual Perturbation Test (TPT) and Visual Perturbation Test (VPT), as shown in Fig. 2.

Textual Perturbation Test. To assess how sensitive a model is to language biases, we propose to swap the key entities, i.e., anatomy or disease, and evaluate the model's sensitivity to the perturbed questions. The evaluation protocol is as follows.

• From the test set of binary questions - "Does anatomy have disease?", we collect True Positive samples having correct "Yes" predictions.
• We randomly replace either the anatomy or disease with another valid co-occurring term. For example, in Fig. 2 (Above), the question "Does left lower lung suffer

from pneumonia?" is replaced by either switching the anatomy to right upper lung, or to another disease.

• The percentage of "Yes" → "No" flips defines the Textual Perturbation Test (TPT) score.

Visual Perturbation Test. We propose to assess whether the model grounds the answer on the anatomical region of interest. Specifically, when querying the presence of a disease in a particular area, e.g., the left lower lung, we take a segmented region of the left lower lung from another image that does not share the same diagnosis as the current one. This segmented region is resized and blended onto the original image. By doing so, we expect the model to focus its attention on the replaced region of interest to produce the correct diagnosis, altering the response from "Yes" to "No". We define the visual perturbation test (VPT) score as the percentage of answers that change from "Yes" to "No" under this protocol.

3.2 Dataset Curation

To assess the LLM's robustness against both textual and visual shortcuts, we introduce a new benchmark called HEALMedVQA (Hallucination Evaluation via Localization in Medical VQA). Built upon VinDr-CXR [Nguyen et al., 2020] and MIMIC-CXR [Johnson et al., 2019], we construct HEALMedVQA by mapping spatial relationships between diseases and anatomical locations, and probing the model's robustness towards disease and location perturbation on both texts and images. Fig. 4 illustrates our overall pipeline of data construction. We extract the disease bounding boxes and anatomical masks from the image. Based on the overlap between disease and anatomy localization, we determine their spatial relationships. Then, several QA templates are used to generate VQA samples. This process can be broken down into four main steps:

Anatomy segmentation. We define 8 main anatomies, e.g. Right lower lung, Heart, etc., where chest diseases commonly occur. On the training set, we obtain the segmentation mask using a model pre-trained on PAXRay++ dataset [Seibold and others, 2022]. On the test set, we consult three board-certified radiologists to annotate the segmentation. By providing doctor-annotated pixel-level segmentation, this research offers a new benchmark to evaluate visual reasoning and robustness to hallucination of current M-LLMs.

Disease extraction. For the disease bounding boxes, the VinDr-CXR dataset provides bounding box annotations for 22 medical findings. We train a standard detection model, YOLOv5 [Jocher et al., 2020], on the dataset and obtain the bounding boxes on MIMIC-CXR. To reduce false positives, we extract the disease labels from the associated medical reports and filter out absent diseases.

Anatomy-disease relation mapping. To get the anatomydisease relation, we measure the overlap between the disease bounding box and the anatomical segmentation mask. Since the disease area is often small compared to the anatomy's, we compute the intersection over the disease area, instead of a union of both. If this IoUdis score between a disease-anatomy pair is over δ = 0.5, then that disease occurs on that anatomy.

Question Generation. After obtaining a list of disease-anatomy mappings, we randomly generate 2-5 QA pairs per image, focusing on two main formats: close-ended and open-ended, as shown in Fig. 4. Close-ended questions are binary: "Does anatomy have disease?", "Is there disease present in anatomy?". Each is either a Positive question, which queries a disease present in the anatomy, or a Hallucinated question, which queries a disease that is not present. Open-ended questions have two types: normal and abnormality-query questions, which respectively probe regions without and with diseases.

In total, our HEAL-MedVQA dataset comprises 45,331 closed-ended questions and 22,573 open-ended ones. We compare our benchmark with those previously reported in Table 1. To the best of our knowledge, we are the first benchmark providing a large-scale dataset with comprehensive hallucination metrics and doctor-annotated masks to evaluate the hallucination of emerging LLMs.

4 ProposedMethod

This section presents our proposed simple-yet-effective framework, Localize-before-Answer (LobA) to (i) localizes the queried regions, and then (ii) reweigh the attention to ground on correct visual evidence before answering. An overview of the framework is shown in Figure 5.

4.1 Grounded Text Generation

Given an image ximg and a question about it xtext. We train LLMs to generate an answer ˆ ytext, while grounding on the queried region, and producing a segmentation mask ˆM. The mask ˆM corresponds to both xtext and ˆ ytext, explaining the response of the model.

We use a <SEG> token, denoting the token that will be used in later steps to obtain the segmentation of the relevant anatomies. Using this token, we instruct LLMs to localize on visual evidence before answering. As such, we format the response as "The location of {relevant areas} is at <SEG>". For example, if the question is "Are there any abnormalities at the right lower lung?", the output of the LMMwill be "The location of the right lower lung is at <SEG>". Note that, in our format, the model has to produce the visual evidence first, before giving the final answers. The LLM is trained to generate the above response:

We use the projected hidden state hseg from the <SEG> token as the query to the segmentation model. This can be formulated as ˆM = Seg(ximg, hseg). Our LLM is trained via the text objectives. Similar to other LLMs [Touvron et al., 2023; OpenAI et al., 2023], we formulate the text generation loss as the cross-entropy loss between the generated text and the ground truth:

We combine BCE loss and DICE loss to train the segmentation module:

Finally, the overall objective can be seen as the weighted sum of the two losses:

4.2 Attention Highlighter via Self-Prompting

While the proposed training paradigm allows the model to localize the region of interest when answering, hallucination can still occur when the model pays little attention to the image compared to the text. To deal with this, we propose to highlight the attention on the queried regions by selfprompting the segmentation map. Our module consists of two functions: attention reweighting and contrastive decoding.

For attention reweighting, patches of interest xhl are added extra weight before feeding through the softmax layer so that those regions get more focus during answer generation. Denote hj,i the original value of the attention logits of patch j to patch i, and β the hyperparameter of added weight, then the new attention weight after reweighting ˆhj,i becomes

Then the attention probability after Softmax is

Here, mi = 1, indicating token i is highlighted, and mi = 0 otherwise. We apply this function to the visual backbone of M-LLM where the patches of interest are interpolated from the mask. Applying the reweighted attention ˜ A, we obtain the highlighted image tokens ˜ximg:

To further alleviate shortcut learning, we perform contrastive decoding [Zhang et al., 2024b], which contrasts the model decisions after and before highlighting regions of interest. Particularly, we compute the difference between the token probability after highlighting phl and probability before highlighting pbh to obtain the decoded probability p:

where α is a hyperparameter controlling the degree of grounding. Applying contrastive decoding minimizes the effect of other tokens on the model output, preventing hallucination.

Discussion. Although Prompt Highlighter [Zhang et al., 2024b] also emphasizes the "highlighting" of regions of interest to reduce hallucination, ours differs in two ways. First, Prompt Highlighter requires human prompting for attention highlighting, which is expensive in medical image analysis. In contrast, ours automatically localizes regions of interest, thus obviating the need for medical professionals to manually localize regions of interest. Second, while Prompt Highlighter only enhances attention on the language decoder, our methodology reweighs attention scores on the visual backbone. The deep intervention on the visual backbone is more effective in mitigating the visual shortcut learning.

5 Experimental Results

In this section, we report the benchmark results of popular medical LMMs and our very own framework, as well as further analysis for our curated adversarial grounding VQA benchmark.

Benchmarks. We conduct a thorough and systematic evaluation of the most popular Medical Large Multimodal Models on our large-scale HEAL-MedVQA benchmark. Specifically, we utilize 7 state-of-the-art multi-modal LLMs for our evaluation protocol, 3 of which are proprietary models: GPT-4o, GPT-4 Vision [OpenAI et al., 2023] and Gemini 1.5 [Team et al., 2023] and 4 are open source M-LLMs of the medical domain: CheXagent [Chen et al., 2024], LLaVA-Med [Li et al., 2024], BioMedGPT [Zhang et al., 2023a] and Med-Flamingo [Alayrac et al., 2022]. We also assess the capabilities of stateof-the-art adversarial VQA methods, including VCD [Leng et al., 2024], Prompt Highlighter [Zhang et al., 2024b], and CRG [Wan et al., 2025].

Implementation Details. Open source multi-modal LLMs are fine-tuned on our training dataset with LoRA [Hu et al., 2022] before evaluation. The learning rate is tuned from the range of values {1e - 4, 1e - 5, 1e - 6}. For LobA framework, we used LLaVA-Med [Li et al., 2024] as the pretrained MLLM and MedSAM [Kirillov et al., 2023] as the segmentation model. Following [Zhang et al., 2024b], we select α = 0.3, βV iT = βLLM = 2 as the set of hyperparameters for the Attention Highlighter module. All models are fine-tuned using Transformer, PyTorch and DeepSpeed frameworks on a single 80GB A100 GPU cluster. During inference, the temperature of all multi-modal LLMs is set to 0.1, and the beam size is set to 1.

Evaluation Metrics. To assess the accuracy of the multimodal LLMs, we utilize LLaMa 3.1-8B to extract information from the M-LLM's output, and categorize them into a set of disease labels. For example, for an open-ended question, if the model returns the answer "The heart suffers from pneumonia, pulmonary fibrosis and nodule/mass" then the extracted labels by the LLM will be pneumonia, pulmonary fibrosis, nodule/mass. We report the multi-label precision, recall, and micro F1 score as the main accuracy metrics. For hallucination sensitivity analysis, we adopt our proposed evaluation metrics, Textual Perturbation Test (TPT), and Visual Perturbation Test (VPT) scores, as discussed in Sec. 3.1. Experimental Results and Discussion. VQA Accuracy. As shown in Table 2, proprietary models achieve sub-optimal accuracy. Most models have accuracy less than 50% on binary Yes/No questions and less than 10% for open-ended questions. This shows the challenging nature of our HEALMedVQA benchmark, requiring the model to be robust against visual and textual shortcut learning. Our proposed framework consistently outperforms recent advances in MLLMs and adversarial VQA techniques even after fine-tuning them. Notably, the proposed LoBA outperforms the state-ofthe-art Prompt Highlighter [Zhang et al., 2024b] by 3.19% and 3.63% on Open-ended questions for VinDr and MIMICCXR, respectively. Injecting grounding prior, as in our proposal, boosts the VQA accuracy.

Textual Perturbation Test. Table 3 reports the sensitivity of each model to textual language bias. The higher the percentage is, the more robust a model is against textual shortcut bias. Our proposed framework achieves the highest sensitivity measure overall, with the ratio of changed answers drastically increasing by up to 3.89% compared to CRG [Wan et al., 2025]. Enforcing visual reasoning via highlighted attention to the queried regions, as in our proposal, alleviates the shortcut learning. Lastly, fine-tuned open-source models are generally more robust than closed-source models like GPT4-O, showing that fine-tuning LLMs on our adversarial data remedies hallucination.

Visual Perturbation Test. Table 4 reports the visual sensitivity score, with the score defined as the number of changed answers when the original image is overlaid with a healthy localized anatomy. Most multi-modal LLMs in our benchmark shift their answer around 40-70% of the time. Our method is the most sensitive to the change in the localized area, increasing the number of visually dependent answers by 1.94% compared to Prompt Highlighter [Zhang et al., 2024b].

Ablation studies. Table 5 reports the effects of each component in our framework: the LLaVA-Med backbone, Grounded Text Generation (GTG) module, and the full framework with self-prompting. Applying GTG improves the model's performance in some categories, showing the effectiveness of localization-aware training. With the self-prompting framework, the accuracy and textual/visual robustness improve greatly, most evidently increasing 4.45% in TPT score, and 2.73% on MIMIC-CXR dataset Yes/No F1 Score.

Qualitative analysis. Fig. 6 showcases our LobA and some other LMM's response on some qualitative examples, as well as the visual attention maps on the input images. We can see that LobA manages to query the attention weights much more intuitively compared to LLaVA-Med or LISA. While others' attentions are scattered into many irrelevant areas, in the first example LobA's main attention weights are allocated around the right lower lung area, and the second example is near the left upper lung. Thanks to the grounding generation module, followed by LobA's self-prompting module to automatically re-weight the attention to the area of interest, the model is able to have better visual reasoning capabilities, leading to more accurate answers compared with other methods.

6 Conclusion

This paper introduces Heal-MedVQA, a new large-scale Medical Visual Question Answering Benchmark with over 67,000 question-answer pairs, which queries diseases at local anatomies, evaluates LMM's capabilities to localize at grounded visual evidence when answering. In addition, we present the Localize-before-Answer (LobA) framework, which trains the LMMs to segment the region of interest and re-adjust their attention for more emphasis on the segmented pathological areas, leading to more reliable answers. Our experimental results showed that our framework LobA outperformed state-of-the-art medical LMMs on Heal-MedVQA, proving its robustness and localization capabilities.

Acknowledgements

We would like to thank Jiangyu Zhou for assisting with the annotation of chest X-ray images for this project.