MedVCoT: Bridging the Modality Gap in Medical VQA Through Latent Visual Reasoning

1 Introduction

Medical Visual Question Answering (Medical VQA) stands as a cornerstone of AI-assisted diagnosis, which needs both high accuracy and rigorous interpretability. The "black-box" diagnosis made to a patient, although it may be correct, will be unconvincing without verifiable visual evidence. However, current state-of-the-art Medical Vision-Language Models (VLMs) come to a fundamental bottleneck: the modality gap. When continuous and fine-grained visual signals (such as the boundaries of a lesion or texture gradients) are forcibly projected into a discrete textual space for reasoning, there is a lot of visual evidence ignored inevitably. This information loss is particularly detrimental in clinical practice, where the subtlest cues in visual content can matter to diagnosis. In that connection, the lack of direct visualization of reasoning has been a barrier to the application of medical AI systems applied to clinical scenarios.

∗Corresponding author.

Prior attempts to address this challenge fall short in large part because they fail to explicitly ground reasoning on visual information. Early medical VLMs, such as LLaVAMed [Liu et al., 2023; Li et al., 2023], mainly focused on aligning the global image features with text through largescale instruction tuning. However, although such models can generate fluent general descriptions, they are fundamentally black box in nature, revealing no information about which particular regions of an image or features led to a particular diagnosis [Mullappilly et al., 2024; Chen et al., 2024; Ning et al., 2025; He et al., 2024; Xu et al., 2025c; Xu et al., 2025a]. This uninterpretability is extremely critical in a clinical scenario, where objective evidence is everything. Other works then attempted to introduce Chain-of-Thought (CoT) reasoning [Wei et al., 2022]

[Hong et al., 2025; Liu et al., 2024; Servantez et al., 2024; Li et al., 2025]. However, these text-based CoTs are limited by the modality gap: since complex, continuous visual information, such as the exact contours of a tumor or subtle texture variations, is forcibly projected into discrete textual descriptions (e.g., merely stating irregular shape), they fundamentally lead to lossy representations of data and therefore also lose precision. Thus, neither paradigm succeeds in providing the pixel-level grounding necessary for high-stakes medical decision-making and so leaves the interpretability gap largely unbridged and clinicians without a reliable way to verify the model's focus.

To address these challenges, we propose MedVCoT, the first approach to incorporate a Visual Chain-of-Thought (Visual CoT) into medical VQA. Different from traditional methods, which only perform reasoning in the text space, our model generates a sequence of continuous latent visual tokens autonomously within a separate thought horizon (denoted as <think> tags) before answering. These tokens act as a bridge between linguistic reasoning and visual perception. In particular, under the guidance of the specialized medical visual expert MedSAM, the VLM learns to generate these tokens as an intermediate reasoning output. These generated tokens can be further mapped with a projector to trigger MedSAM to decode explicit segmentation masks. This mechanism forces the model to "see" and localize the lesion in latent space before "speaking" the diagnosis so as to ensure that answers are causally predicated on verifiable visual evidence.

Our contributions are threefold:

• Pioneering Framework for Latent Visual Reasoning in Medical VQA: We propose MedVCoT, the first medicalVQA-specific framework that performs reasoning through continuous latent visual embeddings instead of discrete text tokens. Through incorporating a Visual Chain-of-Thought mechanism, MedVCoT enables the model to carry out structured visual introspection within a latent thought horizon, ultimately bridging the modality gap that constrains conventional vision-language models.
• Verifiable Interpretability with Latent-to-Mask Projection Projection: Contrary to black-box systems, MedVCoT generates verifiable visual evidence for interpretability in addition to textual diagnoses. The generated latent visual tokens of VLM during reasoning can be projected to trigger MedSAM for decoding pixel-level masks, guaranteeing that decisions are interpretative and clinically reliable.
• State-of-the-art Performance across Multiple Benchmarks: We achieve state-of-the-art performance across multiple medical VQA benchmarks, such as VQA-RAD, SLAKE and PathVQA. The levels of performance of MedVCoT in fine-grained diagnostic tasks are vastly superior to those of currently available open-source and closed-source models.

2 RelatedWork

Evolution of Medical Vision-Language Models. Medical VQA originated from contrastive learning paradigms, including BiomedCLIP [Zhang et al., 2023] and PubMedCLIP [Eslami et al., 2023], which provided strong baselines of retrieval and classification [Radford et al., 2021]. The rise of general VLMs (e.g., GPT-4o [Hurst et al., 2024], Gemini 2.0 [Comanici et al., 2025], Qwen2.5-VL [Bai et al., 2025], InternVL3 [Zhu et al., 2025]) further strengthened specialized medical models, such as LLaVA-Med [Li et al., 2023; Liu et al., 2023], BiMediX2 [Mullappilly et al., 2024], HuatuoGPT-Vision [Chen et al., 2024], UniMedVL [Ning et al., 2025], MedDr [He et al., 2024], EchoGPT [Xu et al., 2025b] and Lingshu [Xu et al., 2025c], these powerful backbones were adapted by domain-specific instruction tuning or data reformatting. Nonetheless, they consider medical VQA as an image captioning problem and lack explicit grounding for lesion localization. In contrast, our method introduces a Visual Chain of Thought, where an expert for medical visual reasoning (MedSAM [Ma et al., 2024]) steers the VLM towards the explicit generation of continuous visual tokens while reasoning. The generated tokens can be decoded into segmentation masks, bridging the modality gap between textual reasoning and fine-grained visual perception.

Evolution of Chain-of-Thought Reasoning. The Chainof-Thought (CoT) paradigm [Wei et al., 2022] has transformed the way that LLMs perform reasoning by embedding intermediate steps in few-shot exemplars. This paradigm has been quickly advanced with progressions including selfconsistency decoding, least-to-most decomposition, and Tree of Thoughts (ToT) [Wang et al., 2022; Zhou et al., 2022; Yao et al., 2023], as well as generalization to medical, legal and coding domains [Liu et al., 2024; Servantez et al., 2024; Li et al., 2025]. GLM-4.1V-Thinking [Hong et al., 2025] uses reinforcement learning with discrete textual chain-of-thought reasoning. Nevertheless, discrete text spaces have difficulty in representing fine-grained visual information for multimodal tasks. To overcome this "modality gap," recent work has turned to latent space reasoning: AURORA [Bigverdi et al., 2025] and Mirage [Yang et al., 2025] introduced perception tokens and machine mental imagery for enhanced visual detail, whereas Chain-of-Visual-Thought (CoVT) [Qin et al., 2025] established an autoregressive framework based on continuous latents for end-to-end visual reasoning. Inspired by these advancements, we propose MedVCoT by introducing this visual chain-of-thought mechanism to the medical domain. Our method bridges visual information and textual diagnostic reasoning, leading to dramatically improved interpretability and accuracy.

3 Method

3.1 ProblemFormulation

Let I denote a medical image, Q a medical question, A the ground truth answer, and M ground truth segmentation masks decoded by a medical visual expert (MedSAM). Then our aim is to model the joint generation probability P(A, Tvis|I,Q). Here, Tvis is a series of continuous latent visual tokens implicitly generated explicitly within a thought horizon (e.g., <think>...</think>). Unlike original VQA, which directly maps P(A|I,Q), MedVCoT forces the Tvis to serve as an interpretable bottleneck, these tokens must be decodable into a predicted mask ˆM ≈Mgt by a visual expert, thus grounding the final textual answer A in verifiable visual reasoning.

3.2 Overall Framework

We propose MedVCoT, a medical vision-language framework that incorporates explicitly visual reasoning in the diagnosis. The architecture consists of three main parts: a fine-tuned Vision-Language Backbone (Qwen2.5-VL-7B), a Latent Visual Projector, and a Medical Visual Expert (MedSAM). The projector is conceived as a lightweight crossattention module where it leverages a linear layer to project the latent visual tokens of a visual expert into keys and values for attention with a set of learnable queries q, yielding aligned visual embeddings. The overall framework and the main workflow of MedVCoT are depicted in Figure 1.

Given a medical image I and a medical question Q, the backbone firstly processes inputs to trigger a Visual Chain-ofThought (Visual CoT) mechanism. Instead of just generating the textual answer, the model can autonomously generate a sequence of continuous Latent Visual Tokens Tvis enclosed within '' tags. These tokens represent a dense and semantically rich representation of the spatial characteristics of the lesion in the latent space.

Subsequently, Tvis is projected by the projector to the prompt embedding space of the frozen MedSAM decoder. MedSAM then decodes these embeddings to a fine-grained predicted segmentation mask ˆM with explicitly visual grounding of the model's focus. Lastly, given the input via context (I, Q) as well as the generated visual rationale (Tvis), the backbone generates the final textual answer A. This

Algorithm 1 Training Process of MedVCoT

Input: Training dataset D = {(Ii, Qi, Ai)}Ni=1 , VLM backbone Mθ, Projector Pϕ (Cross-Attention), Medical Visual Expert (MedSAM) consisting of Encoder Esam and Decoder Dsam.

Parameter: Learnable Query Vectors q (N = 8), Learning rate η, Loss weights λvis

Output: Optimized parameters θ∗ , ϕ∗ , q∗

• 1: Initialize: Load pre-trained weights for Mθ and MedSAM; Initialize Pϕ and q.
• 2: Freeze: Vision Tower ofMθ, Esam, Dsam.
• 3: Trainable: θLoRA, Pϕ, q.

Stage 1: Medical Feature Alignment

• 4: for each batch (I,Q,A) in D do
• 5: Construct prompt: Xin = [I,Q, Esam(I)].
• 6: Forward pass: Apred,Hvis =Mθ(Xin). // Hvis: hidden states of Esam(I)
• 7: Visual projection: Tproj = Pϕ(Hvis, q); ˆM = Dsam(Tproj).
• 8: Get GT masks: Mgt = Filter(MedSAM(I)).
• 9: Lstage1 = Ltext(Apred,A) +λvisLvis( ˆM,Mgt).
• 10: Update parameters: θ∗ , ϕ∗ , q∗ ←Optimizer(Lstage1).
• 11: end for

Stage 2: Visual Reasoning Learning

• 12: for each batch (I,Q,A) in D do
• 13: Generate sequence with visual thoughts: Apred, Tvis = Mθ(I, Q)
• 14: Project thoughts to visual embeddings: Tproj = Pϕ(Tvis, q).
• 15: Decode predicted masks: ˆM = Dsam(Tproj).
• 16: Get GT masks: Mgt = Filter(MedSAM(I)).
• 17: Lstage2 = Ltext(Apred,A) +λvisLvis( ˆM,Mgt).
• 18: Update parameters: θ∗ , ϕ∗ , q∗ ←Optimizer(Lstage2).
• 19: end for

Stage 3: Instruction Tuning with Dynamic VCoT

• 20: for each batch (I,Q,A) in D do
• 21: Sample Bernoulli indicator: k ∼ Bernoulli(p).
• 22: Lstage3 = Ltext +k · λvisLvis.
• 23: Update parameters: θ∗ , ϕ∗ , q∗ ←Optimizer(Lstage3).
• 24: end for
• 25: return θ∗ , ϕ∗ , q∗

makes the diagnosis not just a statistical hallucination but with a causal connection to verifiable visual evidence ˆM, and it consolidates perception, reasoning, and generation into a unified framework.

3.3 Training Pipeline

In order to effectively endow the model with medical-visual reasoning capability, we propose a progressive three-stage training strategy (shown in Figure 2). During all stages, the MedSAM expert is frozen to keep its pre-trained medical seg- Here, LDice guarantees global shape overlap, while LFocal forces the model to focus on hard-to-classify boundary pixels.

Stage 2: Visual Reasoning Learning. In this stage, we integrate visual representation learning and diagnostic reasoning into one training process. We guide the model to autonomously generate explicit visual evidence before generating the ultimate diagnosis. In particular, the model is required to generate a structured output sequence: it first generates continuous latent visual tokens in order to capture pixellevel lesion traits and then uses these variables as rationalegrounded evidence to generate the textual answer. We freeze the vision tower and update our projector and query vectors, as well as LoRA parameters. Meanwhile, we employ a joint optimization strategy in order to balance between visual quality and diagnostic accuracy. The overall loss for Stage 2 is given by:

where λvis balances the two objectives (set to 1.0 by default). The Text Loss Ltext (Cross-Entropy) ensures the model follows the instruction to generate the correct reasoning path and final answer. This hybrid objective enforces a strict causal dependency: the model must "see" the lesion correctly (via Lvis) to answer the question correctly (via Ltext).

Stage 3: Instruction Tuning using Dynamic Visual CoT. In this stage, to facilitate the model for application in complex clinical scenarios and improve its robustness, we conduct instruction tuning for MedVCoT on fine-grained medical clinical VQA datasets. We introduce a dynamic visual CoT ap-

mentation ability.

Stage 1: Medical Feature Alignment. In this stage, we want the visual encoder's representations to be aligned with the LLM's semantic space. The model processes medical image-text pairs, where the input question accompanies the special tokens (e.g., <sam pad>). The output hidden states from the LLM are divided into two streams: (1) Textual Stream: The states for textual answer tokens are processed by the language model head to compute the Text Loss Ltext using a common Cross-Entropy loss. (2) Visual Stream: The states for the special visual tokens are extracted and mapped by the projector to prompt the MedSAM decoder, and then segmentation masks are generated. Such predictions are guided by the Visual Loss Lvis with respect to groundtruth masks. This dual-task objective effectively provides a way to align visual features and diagnostic semantics in the beginning.

The total loss for Stage 1 is:

Here, Ltext is the standard Cross-Entropy loss for next-token prediction, and λvis is a hyperparameter balancing the textual and visual objectives. Simultaneously, the Visual Loss Lvis supervises the quality of the generated visual tokens by comparing the decoded masks ˆM with the ground-truth masks Mgt(automatically distilled from MedSAM with stabilitybased quality filtering):

proach mechanism with a Bernoulli-driven strategy. This procedure stochastically decides whether it should generate the full sequence of visual reasoning tokens or if it will opt out of that for straight text inference. This promotes the model to be flexible about whether or not explicit visual grounding is desired, while preserving the opportunity for direct diagnosis if required. The optimization target for this stage incorporates a dynamic indicator function k ∼ Bernoulli(p), where p represents the probability of activating the visual reasoning path.

When k = 1 (Keep): The model generates visual tokens, and the loss includes both text supervision Ltext and visual reconstruction supervision Lvis. When k = 0 (Drop): The visual generation path is deactivated, and the model is optimized solely via Ltext, simulating scenarios where visual evidence is implicit or reasoning is purely linguistic.

4 Experiments

4.1 Datasets and EvaluationMetrics

We evaluate our method on four diverse medical VQA benchmarks: VQA-RAD [Lau et al., 2018], a manually curated radiology dataset containing 315 images and 3,515 QA pairs verified by clinicians; SLAKE [Liu et al., 2021], a bilingual dataset with 642 annotated images (CT/MRI/X-ray) and 14k QA pairs, incorporating rich semantic labels; PathVQA

[He et al., 2020], the first large-scale pathology VQA dataset with 4,998 images and over 32k open-ended QA pairs derived from medical textbooks; and VQA-Med 2019 [Ben Abacha et al., 2019], a radiology-focused challenge dataset with 4,200 images and 15k QA pairs covering modality, plane, organ, and abnormality categories. For all datasets, we adopt a standard random split, using 80% of the data for training and 20% for testing. All experiments were conducted on a server equipped with two NVIDIA A800 GPUs.

Primary metrics include overall ('All') and open-ended ('Open') accuracy. To ensure robust evaluation against medical terminology variations, we utilize GPT-4o as a semantic judge, explicitly instructing it to "determine if the prediction is semantically consistent with the ground truth, prioritizing core meaning over phrasing." This protocol effectively handles clinical synonyms and ignores irrelevant surface-level discrepancies.

4.2 Main Results

We benchmarked MedVCoT against 14 leading models, including closed-source (GPT-4o, Gemini 2.0) and open-source (Qwen2.5-VL, LLaVA-Med, Lingshu, etc.) baselines, across four medical VQA datasets. For a fair comparison, all baseline models were evaluated using their official pre-trained weights directly on the test sets under a unified evaluation protocol.

As shown in Table 1, MedVCoT achieves superior overall performance with an average accuracy of 68.1%, outperforming all competitors, including the previous SOTA open-source model Lingshu-7B (+7.2%) and the closed-source Gemini 2.0 (+37.5% relative improvement). This advantage is particularly pronounced in tasks requiring fine-grained visual perception. On SLAKE, which features rich semantic annotations, our model achieves an Open accuracy of 83.5%, significantly surpassing Lingshu (78.9%) and UniMedVL (70.8%). Notably, on the challenging PathVQA dataset, our Open accuracy reaches 35.9%, a substantial 21.1% improvement over the runner-up Lingshu (29.6%). These results strongly demonstrate that incorporating an explicit Visual Chain-ofThought effectively bridges the gap in lesion localization and fine-grained feature recognition inherent in traditional VLMs. Although slightly outperformed by HuatuoGPT-Vision-34B on VQA-Med (-2.2%), this marginal gap is primarily attributable to the significant disparity in model parameter scale (7B vs. 34B), yet our model maintains exceptional competitiveness in open-ended reasoning tasks even with a much lighter architecture.

4.3 Ablation Study

Based on the ablation study visualized in Figure 3, we analyze the cumulative impact of our three-stage training strategy compared to the baseline [Bai et al., 2025; Qin et al., 2025]. The baseline model, lacking domain-specific adaptation, achieves a modest overall accuracy of 43.3%.

Stage 1 (Feature Alignment) provides the most significant initial boost, propelling overall accuracy to 59.8% (+16.5%). This sharp rise is particularly evident in PathVQA (+22.9%) and VQA-Med (+11.4%), indicating that establishing a foundational mapping between the VLM latent space and medical visual features is crucial for basic domain competency.

Stage 2 (Visual Reasoning Learning) introduces the Visual CoT mechanism, yielding steady gains (overall +1.3% to 61.1%). While improvements on VQA-RAD and PathVQA confirm the benefit of reasoning capabilities, a slight dip in VQA-Med suggests that learning to reason without taskspecific tuning may introduce complexity that temporarily affects performance on simpler tasks.

Stage 3 (Medical Tuning) delivers the final qualitative leap, surging overall accuracy to 68.1% (+7.0%). The most dramatic improvement is observed on SLAKE (+14.6%), where instruction tuning and robust training strategies (e.g., random token dropping) fully unlock the model's potential for complex, semantically rich queries. This trajectory (43.3% → 59.8% → 61.1% → 68.1%) confirms that while feature alignment lays the groundwork, the integration of explicit visual reasoning and domain-specific tuning is essential for achieving state-of-the-art performance.

4.4 Fine-grained Capability Analysis

To rigorously assess robustness and generalization boundaries, we extend beyond aggregate metrics to conduct a finegrained capability analysis based on imaging modalities. We categorize test samples into 14 distinct sub-categories , spanning macroscopic imaging (e.g., Dermoscopy, Endoscopy), microscopic pathology (e.g., Histopathology), diverse radiological modalities (e.g., CT, MRI, X-ray), and a miscellaneous 'Others' class. This stratified evaluation comprehensively profiles the model's consistency across varying imaging mechanisms and anatomical structures, elucidating its cross-modal strengths.

As illustrated in Figure 4, MedVCoT (dark blue line) demonstrates superior comprehensive performance, with its envelope area significantly encompassing all baselines. Specifically, our model maintains a decisive lead in common modalities such as MRI, CT, and Fundus, while exhibiting exceptional stability in challenging tasks like Histopathology.

Although a slight performance dip is observed in the 'Others' category, this is attributable to its extreme data scarcity (accounting for only 0.22% of the dataset), which has negligible impact on overall clinical utility. These results strongly suggest that the explicit Visual Chain-of-Thought effectively enhances perceptual sensitivity to diverse features, providing reliable decision support across multi-disciplinary diagnostics.

4.5 Case Studies

To qualitatively assess the reasoning logic in authentic clinical scenarios, we select three representative and challenging cases covering anatomical localization, injury differentiation, and specimen identification, comparing MedVCoT against two strong baselines. As illustrated in Figure 5, baseline models, despite incorporating textual CoT capabilities, still lack explicit visual grounding mechanisms and tend to rely on surface-level feature matching or hallucinations. For instance, in the first case, both baselines failed to correctly distinguish the anatomical orientation of the four-chamber view, consistently misidentifying the structure as an atrium rather than the ventricle. In the second case, they misclassified obvious lacerations as petechiae or provided vague descriptions lacking diagnostic precision. Furthermore, in the specimen identification case, both baselines hallucinated reproductive organs (uterus or testicular tissue) instead of correctly recognizing the abdominal tissue.

In contrast, MedVCoT, by activating a Visual Chain-ofThought (Visual CoT) within the latent space, first generates precisely aligned anatomical segmentation tokens. This process acts as a visual introspection prior to answering. This mechanism enables the model to explicitly localize key evidence and structurally connect it with global semantics, resulting in correct predictions across all cases. These qualitative results robustly validate that incorporating visual reasoning steps in the latent space not only improves diagnostic accuracy but also significantly enhances the interpretability and transparency of the model's decision-making process.

5 Conclusion

This study successfully develops and validates MedVCoT, a medical VQA framework that bridges modality gap via a Visual Chain-of-Thought. By generating continuous latent visual tokens, MedVCoT enhances both diagnostic accuracy and clinical interpretability through verifiable segmentation masks. Our model demonstrates state-of-the-art performance across four authoritative benchmarks. Future work will explore scaling MedVCoT to larger-parameter models to verify its universality and integrate diverse visual experts to optimize the Visual CoT.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work is supported by the Science and Technology Project of Liaoning Province (2023JH2/101700363, 2024JH2/102600027), in part by the National Natural Science Foundation of China under Grant 62072073, 62106034, in part by the Fundamental Research Funds for the Central Universities under Grant DUT24ZD124, in part by the Dalian Innovation Fund 2021JJ12GX016, and the Science and Technology Project of Dalian City (2024JJ12GX025, 2023JJ12SN029 and 2023JJ11CG005).