Cause-Effect Driven Optimization for Robust Medical Visual Question Answering with Language Biases

1 Introduction

Medical visual question answering (Med-VQA) has recently garnered significant attention [Liu et al., 2021a; Liu et al., 2022]. Med-VQA [Yang et al., 2016; Yu et al., 2017; Kim et al., 2018; Chen et al., 2022; Do et al., 2021] aims to bridge the gap between medical images and corresponding clinical questions, enabling artificial intelligence systems to predict plausible answers. This automated diagnostic process offers significant advantages, including reduced ing counterfactual samples to reduce the model's dependence on language biases and refocus attention on the target information. However, this approach significantly disrupts the original data distribution, thereby compromising the robustness necessary for Med-VQA.

∗Corresponding author: Bingzhi Chen.

Addressing language bias requires a comprehensive analysis of its causes and consequences. As illustrated in Figure. 1, one major cause is data imbalance: frequent answer categories receive disproportionate emphasis during training, leading to overexpansion of their feature space. This amplifies spurious correlations between certain question types and answers. During training, these spurious correlations are encoded into the network through gradients. Therefore, the model disproportionately relies on the question type for predictions, neglecting critical image features. The question modality gets large gradient updates [Guo et al., 2021], thus becoming a shortcut for generating answers. This phenomenon, driven by language bias, is referred to as modality shortcut. By identifying these challenges, this work strives to propose a novel solution to the challenges posed by inherent biases, improving the robustness of Med-VQA models.

To address these challenges, this paper proposes a novel Cause-Effect Driven Optimization (CEDO) framework, which aims to alleviate language biases from their cause and effect. Specifically, the proposed CEDO framework innovatively incorporates three mechanisms: Modalitydriven Heterogeneous Optimization (MHO), Gradientguided Modality Synergy (GMS), and Distribution-adapted Loss Rescaling (DLR). On the one hand, MHO and GMS are seamlessly integrated to address modality shortcut bias. The primary purpose of MHO is to achieve adaptive optimization for different modalities by adjusting their learning rates. Leveraging the multi-learning rate strategy, it strengthens weaker modalities while preventing dominant modalities from monopolizing the prediction process, thereby mitigating the risk of a single modality becoming a shortcut. Meanwhile, GMS fosters coordinated optimization between modalities through the Pareto method and gradient orthogonality. The Pareto method identifies a steep gradient direction that benefits all objectives (optimizing each modality), allowing the system to converge to a balanced trade-off state. Gradient orthogonality removes gradient conflicts, thereby preventing excessive updates to any single modality and preserving the reasoning capability of other modalities. On the other hand, DLR mitigates data imbalance bias. It ensures balanced learning across all ground-truth answers by adjusting the loss magnitude for each answer category. Based on statistical data distributions, DLR assigns adaptive weights to rescale individual losses, preventing the model from overemphasizing frequent answer categories at the expense of rare ones. Our main contributions are summarized as follows:

• Our work systematically addresses language biases by targeting their cause and effect. Three innovative mechanisms, i.e., MHO, GMS, and DLR, are proposed to comprehensively address shortcut biases originating from modalities and imbalance biases within datasets.
• MHO and GMS cooperate to reduce shortcut biases. MHO enables modality adaptive training, while GMS
• facilitates coordinated updates between modalities to ensure balanced optimization.
• DLR leverages a dynamic loss rescaling strategy to counteract dataset imbalance. By assigning adaptive weights to individual loss, DLR ensures equitable attention across categories.
• Two bias-sensitive Med-VQA datasets, SLAKE-CP and VQA-RAD-CP, are built to evaluate the debiasing performance. Extensive experiments on five datasets demonstrate the effectiveness and generalization of our CEDO method, achieving state-of-the-art performance.

2 RelatedWork

2.1 Visual Question Answering

Medical VQA. Existing Med-VQA research [Yu et al., 2017; Yang et al., 2016; Kim et al., 2018; Chen et al., 2022; Do et al., 2021; Liu et al., 2021a; Liu et al., 2022; Ben Abacha et al., 2019] typically applies prevalent VQA models to the Med-VQA task, with a primary focus on introducing multimodal feature fusion modules. However, due to the scarcity of medical data, these direct adaptations are often hindered by severe overfitting. To mitigate this challenge, Nguyen et al. [Nguyen et al., 2019] propose a hybrid enhanced visual feature (MEVF). Despite these efforts, many Med-VQA datasets [Liu et al., 2021b; Lau et al., 2018] attempt to balance medical images to alleviate inherent bias dependencies, yet they fail to address the detrimental effects of OOD data, thus falling into the trap of language biases.

Robust Medical VQA. While research on Med-VQA debiasing has only recently gained traction, studies on bias mitigation in general VQA have flourished. Recent robust VQA approaches focus on introducing bias models [Han et al., 2021; Han et al., 2023] or problem-specific branches [Cadene et al., 2019; Clark et al., 2019] to learn and mitigate the inherent biases in modalities or datasets. However, due to the scarce medical data, these methods often demonstrate suboptimal generalization performance in the medical domain. As a first attempt, DeBCF [Zhan et al., 2023] specifically targeted this issue and constructed a bias-sensitive dataset to assess debiasing performance. DeBCF employs pre-generated counterfactual samples and counterfactual causal effects to mitigate bias. However, the additional samples cause a distribution shift in the original data, thereby undermining the goal of achieving robust Med-VQA.

2.2 Ensemble-BasedMethods

Addressing the elusive and multifaceted nature of bias in VQA tasks, prior research has introduced a question-only model to explicitly capture inherent biases in the dataset and subsequently subtract them from the base VQA model's predictions [Cadene et al., 2019; Clark et al., 2019; Ramakrishnan et al., 2018; Han et al., 2021; Han et al., 2023]. These approaches isolate the question modality to identify and eliminate shortcuts learned by the model. In addition to these explicit bias removal techniques, data re-weighting methods have emerged as a promising solution. These methods adjust the contribution of individual classes by assigning different weights during loss calculation. A widely used approach is to prioritize tail classes (i.e., less frequent categories) by assigning them higher weights, while head classes (i.e., more frequent categories) receive lower weights, ensuring balanced learning across the dataset. For example, Focal Loss [Ross and Doll´ar, 2017] introduces a dynamic weighting mechanism that emphasizes hard-to-classify and misclassified samples, making it particularly effective in imbalanced datasets. Furthermore, Guo et al. [Guo et al., 2021] propose a loss rescaling strategy tailored to the dataset's distribution, further mitigating the overemphasis on dominant classes.

3 Methodology

3.1 Preliminaries

The Med-VQA task can be viewed as a multi-label classification problem. Without loss of generality, given a batch of data samples D having N samples, each consisting of an image v ∈ V, a question q ∈ Q and a ground-truth answer a ∈ A, the goal of the Med-VQA model is to optimize a mapping function f : V ×Q → R|A| to predict the answer corresponding to a given image-question pair. The base MedVQA model embeds the two features to obtain a joint representation and uses an answer classifier c to generate the logits f. Thus the problem can be formulated as follows:

where ev and eq are the image encoder and question encoder, respectively, and g is a multi-modal feature fusion network. The Med-VQA model can be trained by minimizing the Cross-Entropy (CE) loss function:

In this paper, we integrate [Basu et al., 2023] as our base model, and the total loss is denoted as Lt.

3.2 Modality-driven Heterogeneous Optimization

In Med-VQA, different modalities exhibit distinct complexities and learning requirements. The question modality is prone to biases [Guo et al., 2021], requiring a lower learning rate to avoid overfitting, while the image modality, especially in medical contexts, demands a higher learning rate to capture complex features [Khan et al., 2022; Liu et al., 2021c; Tajbakhsh et al., 2020; Hesamian et al., 2019]. Besides, the classifier tends to be insufficiently trained [Guo et al., 2021]. Traditional single learning rate strategies fail to address these differences, leading to over-reliance on simpler modalities and underutilization of complex ones. To tackle this, we propose MHO, which assigns tailored learning rates to each modality, ensuring synchronized and effective learning aligned with their inherent complexities.

Given a multimodal model F , its parameters θ are grouped into three distinct sets corresponding to the modalities:

where θq, θv, θc represent the parameters of the question modality, the image modality, and the classifier, respectively. To perform heterogeneous optimization, we define a learning rate ηk for each modality k ∈ {q, v, c}. The total parameter update for a training step is expressed as:

where L denotes the loss, and ηq, ηv, ηc are the hyperparameters. The MHO mechanism enhances convergence by addressing the inherent differences in modality complexity. By assigning learning rates tailored to each modality, the algorithm prevents overfitting to modality biases (e.g., question shortcuts) and encourages robust learning from underrepresented modalities (e.g., medical images).

3.3 Gradient-guidedModality Synergy

A robust Med-VQA model is expected to possess well-trained question and image modalities with aligned optimization directions. However, the pervasive issue of modality shortcut bias has been extensively explored [Cadene et al., 2019; Han et al., 2021; Han et al., 2023], where dominant modalities often exhibit disproportionately larger gradient norms [Guo et al., 2021], leading to imbalanced learning dynamics. Inspired by [Sener and Koltun, 2018; Wang et al., 2024], we propose the GMS module that applies the Pareto method and gradient orthogonality to promote synchronized updates between modalities, effectively mitigating the shortcut bias.

Multimodal Learning. The Med-VQA task can be viewed as multimodal learning, where models are expected to produce correct predictions by integrating information from multiple modalities. However, only utilizing such joint loss to optimize all modalities together could result in the optimization process being dominated by one modality, leaving others being severely under-optimized [Peng et al., 2022; Huang et al., 2022]. To overcome this imbalanced multimodal learning problem, introducing unimodal loss, which targets the optimization of each modality, is widely used and verified effective for alleviating this imbalanced multimodal learning problem [Wang et al., 2020]. Therefore, we introduce a question branch and an image branch using CE loss, denoted as Lq and Lv, respectively:

where fq denotes the logits in question branch. fv and Lv are computed in the similar way. Thus, the final loss function is:

Pareto Integration. In multimodal systems, the relationships between multimodal loss and unimodal losses are intricate, as they are highly interdependent but often exhibit gradient conflicts. These conflicts stem from the fact that optimizing one modality's loss might detract from the performance of another, especially when modalities vary in representational strength. The gradients of these losses can be expressed as:

where Rt represents the joint features, Rq and Rv correspond to the unimodal features. Resolving how to integrate Gk effectively without introducing additional conflicts is critical.

The Pareto method, widely utilized in multi-task learning, provides a principled approach for managing these competing gradients [Sener and Koltun, 2018]. It adaptively assigns weights to gradients at each iteration and combines them into a single gradient vector. This ensures that the optimization direction benefits all objectives, facilitating convergence to a Pareto-optimal state. At Pareto-optimality, further improving any single objective is impossible without sacrificing the performance of others. Integrating Pareto optimization into multimodal learning frameworks naturally aligns with the need to reconcile multimodal and unimodal gradients. The corresponding optimization problem can be formalized as:

where ∥·∥ denotes the L2-norm. This formulation seeks to minimize the norm of the weighted gradient combination within the convex hull of the gradient family {Gk}k∈{t,q,v}. The theoretical properties of this optimization are particularly compelling. As shown in [D´esid´eri, 2012], there are two key outcomes: (1) if the minimum norm equals zero, the parameters are Pareto-stationary, satisfying a necessary condition for Pareto-optimality; (2) otherwise, the solution identifies a descent direction that is beneficial for all learning objectives.

Gradient Orthogonality. Under an ideal unbiased setting, the Pareto method identifies the optimal gradient direction for each modality, ensuring efficient and balanced updates. However, the inherent bias present in OOD datasets significantly influences the model's learning process, increasing the likelihood of the Pareto method conforming to this bias rather than mitigating it. To address this limitation and suppress shortcut biases, we introduce gradient orthogonality as a corrective mechanism. Taking Gq and Gv for instance, the cosine similarity S between two gradients can be written as:

If Sqv > 0, it indicates that the two modalities are wellaligned during training, with optimization progressing without bias. Conversely, a negative Sqv reveals that Gq and Gv are oriented in opposing optimization directions. In such cases, the modality with a larger gradient norm tends to dominate the training process, leading to shortcut biases. To this end, we employ gradient orthogonality to recalibrate the optimization directions, ensuring unbiased learning dynamics. The projection of Gv onto Gq can be denoted as P Gv Gq :

Therefore, the biased gradient norms in the question modality can be dislodged:

The gradient norms G ′v and G ′t are obtained in the same manner. In the context of the general case, the ultimate gradient G can be represented as follows:

This approach ensures effective optimization of the modalities, mitigating shortcut bias arising from the question modality. Notably, the method can also be extended to the image modality or fusion modules, though it necessitates careful consideration of the trade-off between computational resource requirements and potential performance gains.

3.4 Distribution-adapted Loss Rescaling

Data imbalance is a primary factor contributing to language biases. Frequent answer categories disproportionately impact the loss function, leading to excessive focus and spurious correlations. To address this issue, inspired by [Guo et al., 2021], we propose an interpretable weighting mechanism that dynamically assigns category-specific weights based on the distribution of question types, ensuring balanced learning across all answer categories. The weight wj i is obtained via:

whereMj is the number of samples under question type qtj, and mji is the number of answer ai under qtj. To achieve a finer balance between rare and common answers, we incorporated the softplus function into the weighting process:

The softplus function not only smooths extreme weight values but also enhances the stability of the training process. This weighting mechanism is specifically applied to the supervised contrastive loss [Khosla et al., 2020] in [Basu et al., 2023], addressing the adverse effects of imbalanced data on feature spaces in contrastive learning:

where i is the index of the current sample in a mini-batch of size I of fused features denoted as {x1, x2, ..., xI}. ai and qtj are the ground truth and question type of the sample xi, respectively. The set of positive examples in the mini-batch is represented as Pi : {p ∈ I s.t. ap = aj}, and the set of negative examples is denoted by Ni : {n ∈ I s.t. an ̸= ai}. The temperature τ is set to 1 following [Basu et al., 2023].

The proposed mechanism tackles data imbalance by ensuring balanced contributions across all answer categories, effectively mitigating distributional shifts and improving robustness in OOD scenarios. Additionally, this mechanism maintains training stability through moderate weight adjustments while remaining computationally efficient, enabling seamless integration into clinical workflows and widespread adoption without disrupting operational efficiency.

4 Experiments

4.1 Datasets

We evaluate our approach on two classical Med-VQA datasets, SLAKE [Liu et al., 2021b] and VQA-RAD [Lau et al., 2018], along with two OOD constructed medical benchmark evaluation protocols, SLAKE-CP and VQA-RAD-CP. To address the scarcity of medical data, we further test our method on a large-scale OOD natural benchmark, VQA-CE [Dancette et al., 2021], to verify the scalability and generalization of the proposed approach. All experiments follow the standard VQA evaluation metric [Antol et al., 2015]. Implementation details can be found in the supplementary material.

4.2 Bias Reconstruction

Following [Zhan et al., 2023], we propose a novel method to construct two biased Med-VQA datasets, SLAKE-CP and VQA-RAD-CP, designed to serve as valuable benchmarks in future research. Similar to [Zhan et al., 2023], the training and test sets were first merged. Each question was labeled based on its question type, determined by its initial words. Samples with binary answers ("Yes" or "No") were categorized as "Yes/No". Finally, samples sharing the same question type and answer were grouped into distinct clusters. As illustrated in Figure. 3, the samples are redistributed to introduce controlled biases. For each question type, the most frequent answers are allocated to the training and test sets in a 39:1 ratio, while the second most frequent answers are split in a 1:39 ratio. The remaining samples are divided at a 3:1 ratio to ensure the training set contains twice as many samples as the test set, maintaining consistency with the VQA-CP v2 [Agrawal et al., 2018] ratio.

4.3 Comparisons with State-of-the-art

Evaluation onMedial Language Biases Benchmark. Table 1 demonstrates the superiority of the CEDO framework on the SLAKE-CP and VQA-RAD-CP datasets, specifically designed to evaluate sensitivity to language biases in the medical domain. Key observations include: 1) Most methods exhibit a substantial performance decline on the biased SLAKE-CP and VQA-RAD-CP datasets compared with SLAKE and VQA-RAD. 2) The proposed model outperforms all the state-of-the-art approaches, achieving improvements of 2.85% and 2.62% on SLAKE-CP and VQA-RAD-CP, respectively, underscoring its robustness in addressing medical bias challenges. 3) Debiasing models designed for natural scene datasets [Cadene et al., 2019; Liang et al., 2021; Han et al., 2021] yield suboptimal results, highlighting their limited generalizability to the medical domain.

Evaluation on Medial Standard Benchmark. Table 2 highlights the significant performance improvements achieved by our CEDO method on the SLAKE [Liu et al., 2021b] and VQA-RAD [Lau et al., 2018] datasets. CEDO

demonstrates notable gains of at least 1.98% and 0.67% over state-of-the-art approaches, respectively. It is important to note that while many existing methods perform well on in-distribution data, they exhibit pronounced performance degradation when exposed to varying prior conditions.

Evaluation on Natural Standard Benchmark. To evaluate the generalization capability of our CEDO approach in realistic application scenarios, we conduct experiments on the challenging OOD benchmark, VQA-CE. This benchmark is designed to test the robustness of VQA models under distributional shifts. As shown in Table 3, the proposed CEDO achieves a minimum improvement of 1.23%, demonstrating What diseases are included in the picture?

(b) Rare diseases distribution

Figure 5: Visualization analysis of CEDO. Our proposed approach combines robust reasoning capabilities for rare diseases with an effective strategy for mitigating bias-related challenges.

its effectiveness in handling OOD conditions.

4.4 Ablation Study

To assess the effectiveness of each component, ablation studies were conducted on SLAKE-CP, as presented in Table 4. The results are summarized as follows: 1) Baseline: We integrate [Basu et al., 2023] as the base model. 2) Baseline w/ MHO: Adding the MHO mechanism improves performance by 1.59%, demonstrating its effectiveness in tailoring learning rates for different modalities. This mechanism mitigates language bias while enhancing the learning of complex visual features, leading to better multimodal integration. 3) Baseline w/ GMS: The GMS component yields a performance gain of 1.91%, effectively facilitating collaborative optimization between modalities, thus significantly reducing shortcut bias. 4) Baseline w/ DLR: The addition of DLR achieves a notable performance improvement of 2.52%, highlighting its capability to prevent sparse categories from being neglected and to mitigate data imbalance bias. 5) Baseline w/ CEDO: The full CEDO model achieves the best performance, demonstrating its robustness in tackling language bias comprehensively from both cause and effect sides.

4.5 Parameter Analysis

We conduct an in-depth parameter analysis of the proposed CEDO method by exploring its behavior under various hyperparameter configurations. Our investigation focuses on three key hyperparameters: ηq, ηv, and ηc, as specified in Eqn. (4). Through systematic experimentation and detailed analysis, as depicted in Figure. 4, we observe that the model achieves peak performance when ηq = 0.002, ηv = 0.003, and ηc = 0.003. This analysis highlights that an optimal combination of these hyperparameters can achieve superior performance of the CEDO model.

4.6 Visualization Results

From the analysis in Figure 5, our CEDO method demonstrates dual advantages in Med-VQA by prioritizing critical visual information. It achieves outstanding debiasing performance, reducing language biases and modality shortcuts, while ensuring high diagnostic accuracy for rare diseases-an often overlooked yet vital aspect of medical AI. Additionally, Figure 6 demonstrates that our CEDO method significantly alleviates medical language bias by shifting the focus from biased patterns to essential information within the data. This enables the model to accurately predict unbiased answers, even in challenging scenarios where language shortcuts or distributional biases may otherwise dominate.

5 Conclusion

In this paper, we identified the cause and effect of medical language biases, i.e., shortcut bias and imbalance bias. To overcome these challenges, we proposed an innovative Cause-Effect Driven Optimization (CEDO) framework that addresses language biases from a causal perspective. The proposed method introduces a multi-learning rate strategy to heterogeneously optimize each modality, then applies the Pareto method and gradient orthogonal technology to achieve intermodality coordination, thereby mitigating shortcut bias, combined with the loss rescaling mechanism to alleviate the imbalance bias. Two bias-sensitive Med-VQA datasets are constructed to evaluate the debiasing performance.

Ethical Statement

There are no ethical issues. Code is available at https://github.com/bvyih3/CEDO

Acknowledgments

This work was supported in part by the Shenzhen Fundamental Research Fund (No. JCYJ20240813105900002), in part by the Guangdong Basic and Applied Basic Research Foundation (No. 2025A1515010225), and in part by the National Natural Science Foundation of China (No. 62302172).