MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias

1 Introduction

Multimodal AI systems are moving into real-world use. They help diagnose diseases, moderate content, and assist drivers. This growing deployment raises a pointed question: can we trust what they see? Imagine a radiologist reviewing a chest X-ray with an AI assistant. The system receives both the image and the radiologist's preliminary notes. If those notes contain an error, will the AI catch it by looking at the image? Or will it simply echo the mistake?

Unfortunately, the evidence points to the second outcome. Show a model an image of a red car alongside text claiming

∗ Corresponding author.

"the car is blue," and most MLLMs will confidently answer "blue" [Liu et al., 2025b; Liu et al., 2025a; Wu et al., 2024]. They trust the text over their own visual perception. This pattern appears across different model families [Liu et al., 2023b; Dai et al., 2023; Bai et al., 2025b; Bai et al., 2025a; Chen et al., 2024] and benchmarks [Zhang et al., 2025b; Jia et al., 2025; Qian et al., 2024]. In safety-critical applications [Bannur et al., 2024; Hou et al., 2025], such blind trust in text could cause serious harm.

Why does this happen? Prior work describes the behavior but not the mechanism [Liu et al., 2025b; Zhang et al., 2025b]. The failure could arise because visual information is poorly encoded, lost during cross-modal fusion, or correctly processed but later overwritten. Understanding the failure mode matters: different causes require different solutions.

To find out, we trace how a model's predictions evolve across its layers. Following [Elbayad et al., 2020; Schuster et al., 2022], we project hidden states at each layer to output probabilities. This lets us observe what the model "prefers" at each depth, and what we find is surprising. Surprisingly, in many conflict failures, the model assigns higher probability to the correct visual answer in intermediate layers. It then reverses course and outputs the text answer (Figure 1(a-b)).

We term this phenomenon late-layer textual override. To measure it, we introduce Modal Dominance Ratio (MDR), which quantifies vision-versus-text preference at each layer. In failure cases, MDR starts positive (visual) and flips negative (textual) in later layers. The visual information was there; it simply didn't survive the full forward pass.

Not every late-layer shift is harmful. Such prediction shift occurs in successful cases too [Wang et al., 2025b], not just failures. What distinguishes the two outcomes is the direction of the shift. We quantify this using the change in MDR across the transition point. As shown in Figure 1(c), in failures, 85% of transitions move toward text. In successes, 89% move toward vision. Late layers sometimes help; they sometimes hurt. The key is knowing which.

This directional asymmetry points to a practical solution. If a model holds a confident visual prediction at the transition layer but abandons it by the final layer, we suspect harmful override. We capture this with two signals derived from our analysis. Anchor confidence measures how certain the transition-layer prediction is. Prediction retention captures whether it survives to the output. High anchor confidence paired with low retention is the signature of harmful override.

Recovering what was known. These observations motivate Conflict-Aware Layer Reference Decoding (CALRD), a training-free method that restores intermediate predictions when override is detected. Our core insight is that predictions at the transition layer still carry the vision-grounded answer before it gets overwritten. By blending transition-layer logits back into the final distribution, we can recover it. CALRD first locates the layer where the output distribution shifts most sharply. It then uses anchor confidence and prediction retention to set correction strength: when both signals point to harmful override, CALRD blends in the transition-layer logits; otherwise, it leaves the output unchanged.

CALRD is straightforward. The model already has the right answer in its intermediate layers; we are helping it remember. No external knowledge or modules are involved. This points to a shift in perspective: improving MLLM reliability may be less about teaching models to see better and more about helping them retain what they have already seen.

We evaluate CALRD on five MLLMs, from InstructBLIP [Dai et al., 2023] to Qwen3-VL [Bai et al., 2025a]. The results suggest that late-layer override is not tied to a specific architecture. Our contributions are as follows:

• We introduce Modal Dominance Ratio to trace layer-wise modal preference and uncover late-layer textual override as a characteristic failure pattern. We show that transition direction, not mere occurrence, predicts correctness.
• We propose CALRD, a training-free method that detects harmful overrides through complementary signals and applies adaptive correction, preserving beneficial processing.
• Experiments show CALRD achieves up to 9.4% gains on conflict tasks while largely preserving standard performance. We also contribute Conflict-VQA, a diagnostic benchmark with explicit visual-textual annotations for mechanistic analysis.

2 RelatedWork

Multimodal knowledge conflicts. When information sources disagree, which should a model trust? Prior work has explored this question for text-only language models [Chen et al., 2022; Xu et al., 2024; Xie and others, 2023]. The problem becomes more pronounced in multimodal settings, where visual perception and textual claims can point to different answers. A number of recent benchmarks document this challenge. HallusionBench [Guan et al., 2024], AutoHallusion [Wu et al., 2024], and PhD [Liu et al., 2025a] report that MLLMs often follow textual context even when it contradicts what is visible in the image. Subsequent analyses suggest this behavior is systematic rather than accidental: models tend to rely on language priors or internal knowledge when facing conflicting evidence [Liu et al., 2025b; Nguyen et al., 2025; Deng et al., 2025]. More targeted benchmarks explicitly construct multimodal conflicts and measure which modality dominates, further confirming this tendency [Jia et al., 2025; Zhang et al., 2025a; Zhu et al., 2024]. While these studies clearly establish the phenomenon, they mostly describe it at the output level without explaining how conflicts are resolved inside the model. Our work addresses this gap by examining how visual and textual signals interact across layers, showing that many failures occur after visual information has already been encoded correctly.

Hallucination mitigation in MLLMs. Various inferencetime methods have been proposed to reduce hallucinations. Contrastive decoding is a common strategy: VCD suppresses tokens preferred under distorted images [Liu et al., 2023a], ICD contrasts standard versus perturbed instructions [Wang et al., 2024c], and OPERA penalizes attention over-trust patterns [Huang and others, 2024]. Recent variants refine these ideas through explicit attention steering [Wang et al., 2025c], multi-stage contrast with selective visual inputs [Park et al., 2025], and probabilistic detection that avoids a second forward pass [Fieback et al., 2025]. A separate line of work exploits model depth. DoLa compares predictions from shallow and deep layers to recover factual signals [Chuang et al., 2024]. DeCo, SHIFT, and related decoders rely on the observation that intermediate layers preserve stronger visual grounding [Wang et al., 2025a; Wang et al., 2025b; Tong et al., 2025]. Training-based approaches reduce hallucinations through preference optimization or contrastive learning but require supervision [Wang et al., 2024a; Yu et al., 2024; Jiang et al., 2024]. Most methods correct uniformly, without distinguishing helpful from harmful late-layer processing. Our finding that transition direction predicts correctness enables selective intervention only when needed. As we show in Section 5, uniform correction methods like DoLa can degrade performance when applied indiscriminately.

Layer-wise Analysis ofMultimodal Models. Recent studies examine visual information flow in MLLMs, finding that visual signals peak in intermediate layers and weaken in deeper ones [Wang et al., 2024b; Wang and others, 2025; Yin et al., 2025]. Others identify failure modes such as attention drift and modality imbalance that reduce visual grounding [Jiang et al., 2025; Chen et al., 2025]. These findings help explain why models capture visual details early but produce text-driven answers at the end. Our work extends this line of research. We show that the direction of layer-wise transitions predicts final correctness, allowing us to intervene only when late-layer processing undermines visual evidence.

3 Diagnosing Late-Layer Textual Override

To understand why models favor text over vision under conflict, we need to trace how their predictions evolve during the forward pass. Does textual bias arise because visual information is never properly encoded? Or is it encoded but subsequently discarded? This section presents a layer-wise analysis that reveals a surprising answer.

3.1 Experimental Framework

Task and data. We study visual question answering where inputs consist of image 𝐼, context 𝐶, and question 𝑄. Let 𝑤vis and 𝑤text denote answers supported by the image and context respectively. A knowledge conflict occurs when 𝑤vis ≠ 𝑤text. We categorize samples into three groups based on context type and model output. Non-Conflict: the context is consistent with the image and the model answers correctly. Conflict-Correct: context contradicts image but the model follows visual evidence. Conflict-Incorrect: the context contradicts the image and the model follows textual claims.

Existing conflict benchmarks lack the (𝑤vis, 𝑤text) annotations required for layer-wise analysis. To address this, we construct Conflict-VQA: 5,969 samples with paired consistent and conflicting contexts. We source images from VrRVG [Liang et al., 2019] and questions from TDIUC [Kafle and Kanan, 2017], covering six question types: object presence, color, attribute, counting, spatial reasoning, and activity recognition. All contexts are generated using GPT-4o and manually verified. Construction details are provided in Appendix A. For mechanistic analysis, we filter each model's competent subset, defined as samples answered correctly under consistent context across five trials. This filtering ensures that observed failures under conflict reflect conflict resolution process rather than basic visual perception errors.

Layer-wise projection. Following early-exit methods [Elbayad et al., 2020; Schuster et al., 2022], we project hidden states at each layer 𝑙 to the output vocabulary via the language model head 𝑊head: 𝑃 (𝑙) 𝑡 = softmax(𝑊headℎ (𝑙) 𝑡 ), where ℎ (𝑙) 𝑡 is the hidden state at the answer token position and 𝑃 (𝑙) 𝑡 (𝑤) is the scalar probability assigned to vocabulary token 𝑤. This lets us observe the model's "preference" at each depth.

3.2 Visual Predictions Emerge, Then Fade

When models fail under conflict, is visual information absent from their representations, or present but ultimately ignored? To answer this, we introduce Modal Dominance Ratio (MDR), quantifying modal preference at each layer:

Positive MDR indicates preference for the visual answer; negative indicates textual preference.

Figure 3 shows MDR trajectories averaged across sample categories. Non-Conflict and Conflict-Correct samples maintain positive MDR throughout all layers, reflecting consistent visual preference. Conflict-Incorrect samples tell a different story: MDR is positive in early and middle layers, then reverses to negative in late layers.

This pattern indicates that in failure cases, models assign higher probability to the visual answer in intermediate layers before shifting toward the textual answer. In other words, the visual information is present in intermediate representations but does not survive to the final output. We term this phenomenon late-layer textual override. This observation suggests that the failure mechanism lies in late-layer processing rather than in early-layer encoding.

3.3 Transition Direction Predicts Success

The late-layer reversal in Conflict-Incorrect samples might suggest a straightforward fix: simply use intermediate-layer predictions. But prediction transitions also occur when models answer correctly. In these cases, late-layer processing appears to refine the answer rather than override it. To intervene selectively, we need a way to tell these two situations apart.

Detecting transitions. Layer-wise predictions show high entropy in early layers and lower entropy in deeper layers as predictions converge. We define the stability onset 𝐿start as the layer with the largest entropy drop:

where 𝐻 (𝑙 ) 𝑡 = - � 𝑤 𝑃 (𝑙) 𝑡 (𝑤) log 𝑃 (𝑙) 𝑡 (𝑤). Across our models, 𝐿start falls between 25% and 40% of model depth.

After stability onset, we detect abrupt distributional shifts using Jensen-Shannon divergence between adjacent layers:

where 𝑀 = 1 2 (𝑃 (𝑙) 𝑡 + 𝑃 (𝑙+1) 𝑡 ). The transition layer is: 𝐿trans = argmax𝑙∈[𝐿start ,𝑁-1] JSD (𝑙) .

Transition direction distinguishes success from failure. Figure 4 shows that both Non-Conflict and Conflict-Incorrect samples have JSD spikes at similar depths, so transitions alone do not distinguish the two groups. What does differ is the direction of the shift. We quantify this via MDR change across the transition:

where bars denote averages over pre/post-transition layers.

Finding. In 85%of Conflict-Incorrect cases, ΔMDR is negative, indicating a shift toward text. The pattern reverses for correct predictions: 89% of Conflict-Correct and 91% of Non-Conflict samples show positive ΔMDR, shifting toward vision or holding steady. Transitions occur in all groups, but only the direction correlates with correctness.

4 Conflict-Aware Layer Reference Decoding

The analysis in Section 3 shows that harmful overrides are characterized by a late-layer shift from visual to textual preference, and that the direction of this shift distinguishes failures from successes. However, the MDR analysis relies on ground-truth (𝑤vis, 𝑤text) labels, unavailable at inference. Can we detect harmful overrides without labels? In this section, we develop practical detection signals and a decoding strategy that recovers overridden predictions without requiring such labels.

Specifically, we propose Conflict-Aware Layer Reference Decoding (CALRD), a training-free method. CALRD first identifies when a confident transition-layer prediction is being suppressed, then adjusts the output distribution to restore it. Since prediction transitions also occur in correct outputs, the intervention must be selective: we modulate its strength based on how strongly the override signature is present.

4.1 Detecting Harmful Override

The MDR analysis shows that Conflict-Incorrect samples hold confident predictions at 𝐿trans that do not survive to the final layer (section 3). This observation motivates two simple signals. Let 𝑤 ∗ = argmax𝑤 𝑃 (𝐿trans ) 𝑡 (𝑤) be the top prediction at transition. Our detection signals are:

Anchor confidence. A higher transition-layer probability for 𝑤 ∗ indicates a stronger intermediate preference:

Prediction retention. To quantify survival, we use the final-to-transition probability ratio; 𝜌 ≈ 1 indicates preservation, while 𝜌 ≈ 0 indicates override:

Figure 5 shows how these signals distribute across sample types. Conflict-Incorrect samples cluster in the high𝐷conf, low-𝜌 region: confident predictions that get overridden. Non-Conflict and Conflict-Correct samples maintain high 𝜌, meaning their predictions persist. This separation allows us to detect harmful overrides at test time.

4.2 Adaptive Intervention

Not all prediction transitions are harmful. As shown in Section 3.3, transitions toward vision correlate with correct outputs, while transitions toward text correlate with failures. To avoid disrupting beneficial late-layer processing, intervention strength must adapt to override severity. We define:

The multiplicative form ensures that 𝜆 is substantial only when both conditions hold: a confident transition-layer prediction exists (𝐷conf high) and it is being suppressed (𝜌 low). When 𝐷conf is low, there is no preference to recover, so 𝜆 stays small regardless of retention. When 𝜌 is high, the prediction survives to the output, so intervention is unnecessary. Only when a confident prediction gets discarded does 𝜆 become large enough to affect decoding. This design is motivated by the asymmetry in Section 3.3: harmful cases are likely to exhibit confident intermediate predictions that are later suppressed, whereas correct cases tend to preserve them.

4.3 Transition-Layer Guided Decoding

Given 𝜆, we adjust the output logits by combining final-layer and transition-layer predictions. Let 𝜙 (𝑙) 𝑡 =𝑊head · ℎ (𝑙) 𝑡 denote the unnormalized logits at layer 𝑙. The adjusted logits are:

with output distribution 𝑃 ′ 𝑡 = softmax(𝜙 ′ 𝑡 ). When 𝜆 = 0, this reduces to standard decoding. As 𝜆 increases, the transitionlayer prediction receives more weight, recovering the visiongrounded answer that would otherwise be lost. For samples without override signatures, 𝜆 remains near zero and the output is largely unchanged.

CALRD works at each token position independently. For multi-token generation, we recompute 𝐿trans, 𝐷conf, and 𝜌 at each step, allowing the intervention strength to vary across the sequence. The logit-level operation makes CALRD compatible with greedy, beam search, and nucleus sampling without modification. Note that CALRD does not introduce external knowledge or require additional training. The correction comes entirely from the model's own intermediate representations. As our analysis shows, the vision-grounded answer is often already present at the transition layer; CALRD simply helps it survive to the output.

5 Experiments

We evaluate CALRD on two questions: (1) Does it improve conflict resolution? (2) Does it hurt non-conflict scenarios? Experiments cover five MLLMs with different architectures and capability levels. For each model, we report results on conflict benchmarks where context contradicts the image, as well as standard hallucination benchmarks.

5.1 Experimental Setup

Models. We evaluate five MLLMs: InstructBLIP-7B [Dai et al., 2023], LLaVA-1.5-7B, LLaVA-1.6-7B [Liu et al., 2023b], Qwen2.5-VL-7B [Bai et al., 2025b], and Qwen3VL-8B [Bai et al., 2025a]. These models span different architectures and capability levels.

Benchmarks. We evaluate on two complementary benchmark groups. (1) For conflict resolution, we use ConflictVQA (Section 3.1), which contains 5,969 samples requiring open-ended answers, and PhD-icc [Liu et al., 2025a], the incorrect-context subset of PhD containing 16,844 yes/no question pairs. Both benchmarks include contexts that contradict visual evidence, and we report accuracy. (2) For standard hallucination without explicit conflicts, we use POPE [Li et al., 2023] and CHAIR [Rohrbach et al., 2018]. POPE probes object hallucination by asking "Is there a <object> in the image?" across random, popular, and adversarial splits. It covers 500 MSCOCO images with six questions per image for each split, and we report F1 score. CHAIR measures caption hallucination against ground-truth object annotations, reporting instance-level C𝐼 and sentence-level C𝑆 (lower is better). Following [Huang and others, 2024], we evaluate on 500 MSCOCO images with the same captioning prompt "Please describe this image in detail." for consistency.

Baselines. We compare against four representative methods. DoLa [Chuang et al., 2024] contrasts logits from later layers against earlier layers to surface factual knowledge. VCD [Leng et al., 2024] contrasts outputs from original and distorted visual inputs to reduce over-reliance on language priors. OPERA [Huang and others, 2024] penalizes overtrust patterns in self-attention and uses rollback to re-select tokens. DeCo [Wang et al., 2025a] adaptively selects preceding layers where visual information is stronger and integrates their predictions into the final output. We test CALRD with greedy, beam search, and nucleus sampling. Since not all baselines are designed for every decoding strategy, we evaluate each method under the decoding regimes supported by its official implementation and compare CALRD against the same vanilla decoder in each regime. All experiments are run on NVIDIA H100 GPUs.

5.2 Main Results

Conflict resolution. Table 2 shows results on conflict benchmarks. CALRD outperforms all baselines across models and decoders. Gains are largest on PhD-icc: LLaVA1.5 improves from 27.72% to 36.35% (+8.63%), LLaVA-1.6 from 28.23% to 36.52% (+8.29%), and InstructBLIP from 41.08% to 49.62% (+8.54%). On Conflict-VQA, improvements reach up to 7.4%.

Larger gains on PhD-icc. The two benchmarks differ in question format: PhD-icc uses binary yes/no questions, while Conflict-VQA requires open-ended answers. We suspect the binary format creates conditions where textual bias is more pronounced. When context strongly suggests "yes" or "no," the model may be more easily swayed away from visual evidence. In contrast, open-ended questions require generating specific content, making it harder for context alone to override perception.

Baseline accuracy and improvement. An interesting pattern emerges when we rank models by their vanilla accuracy on PhD-icc: LLaVA-1.5 (27.72%) < LLaVA-1.6 (28.23%) < InstructBLIP (41.08%) < Qwen2.5-VL (51.30%) < Qwen3VL (60.32%). The gains from CALRD roughly follow the inverse order, with weaker models seeing larger improvements. This pattern is expected: stronger models already resolve more conflicts correctly in their late layers, leaving fewer overridden predictions for CALRD to recover. The adaptive nature of our intervention naturally accommodates thiswhen predictions survive to the output (𝜌 remains high), 𝜆 stays small and CALRD leaves the output largely unchanged. For Qwen3-VL, the +3.36% absolute gain corresponds to an 8.5% relative error reduction, indicating that even wellcalibrated models contain recoverable failures.

Comparison with other layer-based methods. DeCo and DoLa also leverage information from earlier layers, but their performance is less consistent. DoLa improves some configurations but hurts others: on InstructBLIP, Conflict-VQA accuracy drops from 39.61% to 32.72%. DeCo shows more stable behavior but still underperforms CALRD across the board. One possible explanation is that these methods apply corrections more uniformly, without distinguishing cases where late-layer processing is beneficial. CALRD's detection signals allow it to intervene selectively, reducing the risk of disrupting predictions that were already on track.

Non-conflict scenarios. Textual override is not identical to all hallucinations, so we also test CALRD on standard objectcentric hallucination benchmarks without explicit conflicts. Amethod that improves conflict resolution but degrades standard performance would have limited practical value. Tables 3 and 4 show that CALRD largely avoids this trade-off.

POPE results (Table 3). CALRD maintains or improves F1 across all configurations. Under greedy decoding, InstructBLIP improves from 80.0 to 85.4, and Qwen3-VL from 88.5 to 91.6. POPE probes object hallucination without explicit visual-textual conflicts, so these results suggest that late-layer override may occur more broadly than just in conflict settings. The key is that CALRD does not intervene indiscriminately: when 𝜌 is high, indicating the prediction is stable, 𝜆 stays near zero and the output remains unchanged.

CHAIR results (Table 4). Captioning differs fromVQA in that it requires extended sequences rather than short answers. Hallucinations accumulate over multiple tokens, making it a useful stress test. CALRD reduces both C𝐼 and C𝑆 in most configurations. InstructBLIP shows the largest improvement: C𝐼 drops from 23.7 to 14.6 (38% relative reduction) and C𝑆 from 58.8 to 40.2. These results indicate that recomputing detection signals at each token position allows CALRD to adapt throughout the sequence, rather than a fixed correction.

5.3 Analysis

Ablation study. Table 5 examines the contribution of each component using LLaVA-1.6 with greedy decoding. Removing 𝐷conf drops Conflict-VQA accuracy from 49.5% to 47.3%, while removing 𝜌 causes a larger drop to 45.1%. This suggests that retention is the more informative signal for detecting harmful override, which makes sense: a prediction can be confident at the transition layer for various reasons, but a sharp drop in retention specifically indicates that something changed in late-layer processing.

Using a fixed layer at 75% depth instead of dynamic 𝐿trans detection reduces accuracy to 45.9%. The value corresponds to the median transition layer observed in our analysis (Section 3.3), so this comparison uses the most representative fixed position rather than an arbitrary choice. The accuracy drop indicates that transition points vary substantially across samples, and a single fixed layer cannot capture this variation. We also observe that POPE follows a similar pattern, with both signals contributing to the overall improvement.

These results support the multiplicative design 𝜆 = 𝐷conf · (1 - 𝜌): both signals provide useful information, but neither is sufficient alone. The product ensures that substantial intervention occurs only when a confident prediction exists and is being suppressed by late-layer processing.

Efficiency. Figure 7 compares latency and memory usage across methods using LLaVA-1.5 on a single H100 GPU. CALRD adds modest overhead: 19.8 ms/token compared to 18.3 ms/token for vanilla decoding, with only 80 MB additional memory. VCD nearly doubles latency due to its dual-stream requirement, and OPERA is approximately 10× slower because of iterative rollback. DeCo falls in between. These results suggest CALRD is practical for deployment scenarios where efficiency matters.

6 Conclusion

We studied why multimodal models favor text over vision when the two conflict, and found that the problem is not poor visual encoding. In many failure cases, models assign higher probability to the correct visual answer in intermediate layers, only to override it with the textual answer in deeper layers. We call this late-layer textual override. Importantly, the direction of prediction change distinguishes failures from successes: shifts toward text correlate with incorrect outputs, while shifts toward vision correlate with correct ones. This observation motivates CALRD, a training-free decoding method that restores intermediate predictions when override signatures are detected. Experiments on five MLLMs show up to 9.4% improvements on conflict benchmarks while largely preserving performance on standard tasks. Our results suggest that improving reliability under conflict may require not better visual encoding, but better preservation of visual information through the forward pass.

Limitations and future work. Our work opens several avenues for future research. A natural next step is to examine how late-layer dynamics interact with model scale and pretraining data composition-questions that require controlled experiments with access to training corpora, beyond the scope of our mechanistic study. Additionally, exploring whether similar override patterns emerge across other modality pairs (e.g., audio-text) may reveal general principles of multimodal integration. We view our mechanistic analysis as a foundation for such investigations.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 62306331, 62407037, and CAAI Youth Talent Lifting Project under Grant CAAI20232025QNRC001.