CABIN: Debiasing Vision-Language Models Using Backdoor Adjustments

1 Introduction

Vision-Language Models (VLMs) such as CLIP [Radford et al., 2021] unify visual and textual representations, enabling them to excel in a broad range of downstream tasks. Training on large-scale image-text pairs allows these models to learn a joint embedding space where visual and linguistic concepts align effectively, exhibiting strong generalization even without extensive task-specific fine-tuning [Zhao et al., 2024]. Their ability to learn generalisable representations makes VLMs attractive for numerous real-world applications, from web-scale image searches [L¨ulf et al., 2024] to automated content filtering [Hong et al., 2024].

Despite their impressive performance, recent studies have shown that VLMs inherit stereotypical biases in their training data [Birhane et al., 2021]. Such biases may manifest as performance disparities across sensitive attributes (e.g. gender, race and age). For instance, a VLM might misclassify or underrepresent certain demographic groups in tasks, e.g. face recognition or occupational image classification [Parraga et al., 2023]. For image retrieval, such biases could cause search results to reflect stereotypical associations. For example, querying "nurse" primarily returns female-presenting images. Such bias adversely impacts both the inclusivity and fairness of VLM-driven systems.

Efforts to mitigate bias in machine learning encompass several intervention strategies. Pre-processing methods, e.g. data augmentation [Choi et al., 2020; Lim et al., 2023], and in-processing techniques, e.g. adversarial learning [Zhang et al., 2018] and feature disentanglement [Locatello et al., 2019], generally require retraining the entire model. This process consumes substantial computational resources and often involves counterfactual data [Zhang et al., 2022; Lee et al., 2023], which can generate unrealistic or distorted results. Post-processing strategies involve adjustments applied after training and include techniques such as feature projection [Chuang et al., 2023; Ratzlaff et al., 2024] and prompt engineering [Friedrich et al., 2023; Qiao et al., 2025]. Although these methods do not require fine-tuning or retraining, they typically focus only on single modalities, thus failing to use VLMs' shared embedding spaces. Furthermore, many existing approaches rely on training datasets that are closely related to their test datasets, often partitioned from similar face-centric datasets (e.g., FairFace [Karkkainen and Joo, 2021]), which limits their generalizability and hinders consistent performance across diverse datasets.

To address these limitations, we propose CABIN, a novel backdoor adjustment framework that mitigates the influence of sensitive attributes on VLM predictions (Fig. 1). CABIN uses a shallow neural network "mapper" to transform text embeddings into image embeddings, allowing us to manipulate sensitive attribute features within the shared embedding space without altering the image. This is a key advantage, as it will enable us to directly address biases within the VLM's core representation. Trained on data similar to the VLMs, this mapper avoids overfitting without requiring annotated facecentric datasets. By manipulating sensitive attribute features in the image embedding through text, CABIN fully leverages the VLMs' shared space. We may identify biased attribute subspaces within this shared embedding space by finding features that vary under different sensitive attributes and locating the features of the image embedding that encode sensitive attributes. By neutralizing these subspaces, e.g. removing these attribute-related components, CABIN makes the image representations less sensitive to these attributes. Importantly, CABIN does not require model retraining for different test sets. By neutralizing these subspaces, CABIN enables debiasing via backdoor adjustments, a causal inference technique that allows us to estimate the causal effect of the input on the prediction by controlling for confounding factors (i.e., sensitive attributes), facilitating accurate causal estimation. This procedure is independent of domain-specific architectures and task-specific data, making it applicable to various tasks, including zero-shot classification and image retrieval.

Our key contributions are as follows. (1) We develop CABIN, a new debiasing framework grounded in causal inference. CABIN uses a lightweight mapper to manipulate sensitive attributes within image embeddings by leveraging text, avoiding overfitting specific datasets and ensuring effective utilization of both image and text modalities. (2) We introduce a task-agnostic structural causal model for VLM inference. This model explicitly represents the confounding influence of sensitive attributes on the model's predictions, allowing us to identify and address the relevant backdoor paths for debiasing. (3) We conduct extensive experiments across diverse image classification and retrieval datasets. The results demonstrate that CABIN effectively mitigates bias, as quantified by multiple fairness metrics, while maintaining competitive performance.

2 RelatedWork

VLMs Bias Evaluation. VLMs, such as CLIP, excel in unifying visual and textual modalities, enabling robust zero-shot generalization across downstream tasks [Parraga et al., 2023].

These models effectively align image and text embeddings effectively, supporting applications such as image classification and retrieval. However, their reliance on large-scale, uncurated datasets often perpetuates societal biases [Birhane et al., 2021], raising fairness concerns. Early fairness evaluations primarily relied on datasets containing human images [Wang et al., 2021; Weng et al., 2024], with face-centric datasets, FairFace [Karkkainen and Joo, 2021] being a common choice [Zhang et al., 2023; Berg et al., 2022]. Textimage pair datasets with sensitive attribute annotations were also used to assess the fairness of VLMs [Jung et al., 2024]. Debiasing VLMs. Efforts to mitigate biases in multimodal models typically adapt single-modality techniques such as adversarial training [Zhang et al., 2018], feature disentanglement [Locatello et al., 2019], and data augmentation [Choi et al., 2020; Lim et al., 2023]. However, extending these methods to VLMs is challenging due to the complexity of joint embedding spaces and the high cost of retraining. Postprocessing approaches, such as feature projection [Chuang et al., 2023; Ratzlaff et al., 2024] and prompt engineering [Friedrich et al., 2023], avoid retraining but often fail to address cross-modal biases. Other methods [Jung et al., 2024; Seth et al., 2023] disentangle visual and textual features but rely on curated datasets, risking overfitting and limiting generalization. Our approach leverages post-processing by manipulating embeddings without altering image pixels or model architecture. Trained on open-source image-text pairs [Schuhmann et al., 2021], CABIN reduces overfitting while identifying sensitive attribute subspaces in the joint embedding space, which generalises across tasks, including classification and retrieval.

Debias using Causal Inference. Causal inference offers an approach to disentangle spurious associations from genuine causal relationships by introducing interventions that modify variable distributions while keeping others unchanged. Recent research [Weng et al., 2024; Patil et al., 2023; Zhang et al., 2023] apply causal inference to debias machine learning models but often target specific tasks or single modalities, limiting generalizability. We propose a task-agnostic framework for modelling causal relationships in VLM predictions. Our approach removes spurious correlations without task-specific adaptations by treating sensitive attributes as confounders and applying backdoor adjustment. Experiments demonstrate its effectiveness in mitigating bias across diverse tasks while preserving model performance. Table 1 summarizes the main characteristics of existing compared to ours.

3 Preliminaries

Vision-Language Models (VLMs) such as CLIP [Radford et al., 2021] represent a class of foundational models trained on large-scale image-text pairs. These models learn a shared latent space where both images and text are embedded into a common representation. VLMs are trained to align these embeddings such that related image-text pairs are close in the joint embedding space.

Bias in Vision-Language Models. Although VLMs demonstrate strong zero-shot capabilities, they may inherit biases from their pre-training data [Birhane et al., 2021; Gustafson et al., 2023]. Such biases may manifest as systemic performance disparities across different demographic attributes. Consider a set of m sensitive attribute groups A = {A1,A2, . . . ,Am}, and each Ai = {a1i , a2i , . . . , ani } is a set of categories corresponding to a particular sensitive attribute dimension where n is the number of sensitive attributes in sensitive attribute group Ai. Let D = {(x1, {aji }mi=1 ), (x2, {aji }mi=1 ), . . . , (xk, {aji }mi=1 )} be an image dataset with sensitive attribute annotations, i.e., each image xk has a corresponding annotation set {aji }mi=1 . For instance, if A1 corresponds to race with categories {a11 , a21 , a31 } (e.g., Black, Asian, White) and A2 corresponds to age with categories {a12 , a22 } (e.g., young, old), then annotation {a21 , a12 } might indicate that xi is an image of a young Asian person. We focus on the event where the VLM's top similarity score for an image xi is at least a certain threshold ϵ. Among all the textual prompts T used to compute similarity, let:

where I(·) is an indicator function. The binary result M(k) indicates whether the model's confidence, as measured by the maximum similarity to any class, surpasses the threshold ϵ for the image xk. This may be interpreted as bias if the proportion of images meeting this criterion differs significantly across categories within an attribute group. Alternatively, ϵ could be tuned as a hyperparameter to balance the tradeoff between the number of high-confidence results and the sensitivity of bias measurement. To obtain high-confidence results for the model, we set ϵ to 0.5.

Fairness Metrics. To quantify bias more interpretably, we examine how high-confidence predictions are distributed across the categories of each attribute group Ai ∈ A by measuring the relative representation of these predictions for each category within an attribute group. This approach enables us to identify disparities in how the model treats different categories, such as age or race. Consider one attribute group Ai = {a1i , a2i , . . . , ani }, let the subset of the dataset D containing aji attribute beDaji . For each category aji inAi, where 1 ≤ j ≤ n, define:

as the proportion of images in D that belong to category aji of the i-th attribute group. Similarly, let

be the proportion of images that both belong to category aji and meet the confidence criterion (M(k) = 1) among all high-confidence classifications. We compute the disparity using the Skew value for each category aji :

A positive Skew(aji ) indicates that the model is disproportionately favouring category aji relative to its baseline frequency in the dataset. In contrast, a negative Skew(aji ) suggests that aji is underrepresented or disadvantaged.

For a given attribute group Ai, we summarise the skew values across all categories {aji } using aggregate metrics as follows. The MaxSkew is defined as maxj Skew(aji ), which captures the most favoured category within the attribute group. Conversely, the MinSkew, defined as minj Skew(aji ), identifies the most disfavored category. Additionally, we use MaxSkew@k, which represents the average MaxSkew values of the top-k results within Ai, providing insights into which categories are most favoured collectively. Similarly, MinSkew@k computes the average MinSkew values of the top-k results, highlighting the categories that are most disfavored on aggregate.

4 CABIN Framework

This section describes our debiasing approach grounded in causal inference to mitigate stereotyped biases in VLMs. First, we formulate the causal inference framework for VLMs to make predictions. To effectively mitigate biases in VLMs, we employ a causal inference strategy known as backdoor adjustment in Sec. 4.1 and introduce a lightweight mapper that alters image embeddings based on text prompts introduced in Sec. 4.2. By combining the mappers and backdoor adjustment, we effectively identify biased features in embedding VLMs, described in Sec. 4.3.

4.1 Causal ProblemFormulation

Our method selectively incorporates only those causal factors relevant to understanding and mitigating bias, omitting other causal relationships that do not pertain to the debiasing objectives. To define the relationships between different variables in a VLM-based prediction task, e.g., zero-shot image classification, we use a Structural Causal Model (SCM) to model the causal relationships. The model includes three primary variables: X, the input to the VLM, exemplified by an image; Y, the prediction of the VLM, which could be the predicted class label; and B, which represents the sensitive attributes such as gender and age. The causal relationship between these three variables may be graphically represented using SCM in Fig. 2(c).

The VLM learns complex associations from large-scale image-text data. Thus, the input data X may naturally encode sensitive attributes B when making predictions. The model will predict the stereotypical bias inherited from training data. Thus, B also affects the prediction of Y. Since B affects both X and Y, we identify B as a confounding sensitive factor using a backdoor path X ← B → Y, which may render the prediction biased. Specifically, the model may favour certain demographic groups or produce systematically different predictions for images with different sensitive attributes. To mitigate this bias, we isolate the causal effect of X on Y by removing the influence of the confounding factor B. Specifically, we identify a set of variables, known as a backdoor adjustment set, that blocks all non-causal paths between X and Y. By conditioning on this set, we eliminate spurious correlations introduced by B, enabling the computation of the genuine causal effect of X on Y:

where P(Y | set(X)) denotes the interventional distribution of Y when X is set at a particular value.

4.2 Text Embedding to Image EmbeddingMapper

We use backdoor adjustment to block the backdoor path X ← B → Y. Applying the backdoor adjustment formula requires that X and B be independent so that conditioning on B removes the confounding influence. However, B is typically visibly encoded in the image X in vision-based tasks. For example, facial features in a nurse's image may reveal gender information, implying that X and B are not independent.

To resolve that X and B are not independent, we propose training a mapper that transforms from text modality to image modality. This mapper is trained on a large-scale textimage pairs dataset LAION-400-MILLION [Schuhmann et al., 2021]. Since the mapper learns the transformative relationship between text and image embedding of the same sense, we only need a neural network with a small number of parameters, such as Bottleneck [He et al., 2016], to leverage the text encoder to generate embeddings for variants of textual descriptions that differ only in their sensitive attribute terms. By applying the mapper to these textual embeddings, we simulate how changes in sensitive attributes affect image embeddings in the VLM's joint space.

Here, we define the mapping process for aligning sensitive attributes. Let ϕ : I → Rd and ψ : T → Rd represent the image and text encoders, mapping inputs x ∈ I and t ∈ T to d-dimensional embeddings ϕ(x) and ψ(t), respectively. The mapper M : Rd → Rd is trained so that M(ψ(t)) ≈ ϕ(x) for corresponding image-text pairs. Once trained, we identify dimensions or subspaces of the image embedding space that are responsible for encoding the sensitive attribute by analyzing how substituting one attribute (e.g., male) for another (e.g., female) changes M(ψ(t)).

As shown in Fig. 2(a), we design a training loss with a combination of two loss terms: the first term Lalign measures the alignment between text embeddings and image embeddings, while the second contrastive term Ldiff encourages mismatched image-text pairs to be pushed farther apart, enhancing the discriminative ability of the learned mapper. To balance two loss items, we use λ to adjust the weights of loss items. In order to find a proper λ to balance two terms in the loss function, we use PCGrad [Yu et al., 2020], which dynamically adjusts the gradient directions by projecting conflicting gradients onto orthogonal subspaces to reduce interference between losses and ensure a more stable optimization process. This approach aligns with our requirement to simultaneously enhance the mapper's alignment capacity and discriminative ability. Let (xk, tk) be a matched image-text pair. We define an arbitrary prompt that does not correspond to xk in the dataset as tuk = {t ∈ T | t ̸= tk} and let (xk, t′ ) be a non-matching pair where t′ ∈ tuk. We propose the following loss function:

where λ is a learnable parameter determined by PCGrad.

Alignment Loss (Lalign): The primary goal is to ensure that M(ψ(t)) aligns closely with ϕ(x). Intuitively, we minimize the distance measure between text and image embeddings:

Difference Loss (Ldiff): Additionally, to guide the mapper to distinguish matched text-image pairs from mismatched pairs, we introduce a contrastive term:

where µ > 0 is a contrastive loss margin. Incorporating Ldiff into Lmapper ensures that matched pairs are mapped closer together than mismatched pairs, fostering a more discriminative mapping.

By analyzing pairs of textual descriptions that differ only in their sensitive attribute categories, we identify embedding directions corresponding to specific attribute groups. For a given attribute group Ai (e.g. gender), we compute all pairwise differences among the categories (e.g. male and female) to form a set of directional vectors:

The SAi defines a subspace that captures the variations across categories within the attribute group, effectively characterizing biased directions.

To ensure the independence of X and B so that the image embedding does not contain information related to sensitive attributes, we neutralize these attribute-specific components from the image representation:

where Π∆(·) denotes the projection onto the subspace defined by all difference vectors ∆ in SAi . After this transformation, ˜ ϕ(x) is less dependent on sensitive attributes, moving us closer to a scenario where X and these sensitive attributes are independent.

4.3 Applying Backdoor Adjustment

As we have ensured that X and the sensitive attributes are approximately independent by neutralizing the corresponding attribute subspaces in the image embeddings, we apply the backdoor adjustment to estimate the causal effect of X on Y, which represents the genuine causal relationship, i.e., the unbiased prediction.

To implement the backdoor adjustment, we first compute a generalised outcome distribution conditioned on the neutralized embedding ˜ ϕ(X) under each category aji ∈ Ai for every attribute group Ai ∈ A. This involves providing the VLM with the neutralized visual representation ˜ ϕ(X) as well as the textual inputs {ψ(t)} that vary depending on the downstream task. The VLM's inference function f(˜ ϕ(X), {ψ(t)}, θ), parameterised by θ, then produces a conditional outcome distribution. For each aji ∈ Ai:

where Norm(·) is a normalizing function (e.g., softmax for categorical outcomes) chosen according to the requirements of the downstream task. Next, the probabilities P(aji ) for each category aji are estimated from the distribution of attributes in the test data Dtest:

where Daji test represents the subset of Dtest containing aji attribute.

Finally, we aggregate over all weighted conditional outcomes for each attribute group. For a single attribute group Ai:

Extending to multiple attribute groups, each Ai is adjusted independently. Fig. 2(c) illustrates the process of applying backdoor adjustment. This approach is agnostic to the specific downstream task, relying only on the availability of textual inputs {ψ(t)} to guide the VLM's prediction. By performing a backdoor adjustment on the neutralized embeddings, the outcome reflects the genuine causal relationships free from confounding sensitive attributes.

5 Experiments

We structure our experimental study around two key research questions (RQs): How effectively does our method reduce bias? (RQ1) and How does our approach balance debiasing with preserving task performance? (RQ2). We focus on two representative tasks, image classification and image retrieval, to provide a holistic perspective on how bias arises and is mitigated in VLMs.

Baseline and Architecture. All experiments use Vanilla CLIP encoders with ResNet50 and ViT-B/32 backbones as baselines. DeAR [Seth et al., 2023] and SFID [Jung et al., 2024] serve as comparative methods. We report Skew-based metrics mentioned in Sec. 3.

Data and Training. As mainstream VLMs such as CLIP's training data have not been published, we use a largescale open-source image-text dataset LAION-400-MILLION [Schuhmann et al., 2021] to train our mapper M. We randomly sampled 10 million paired image-text data from the dataset to ensure M(ψ(t)) approximates ϕ(x) accurately.

Evaluation Datasets. We use FACET [Gustafson et al., 2023], PATA [Seth et al., 2023], and Flickr30K [Plummer et al., 2015] to evaluate our debiasing method. Traditional facecentric datasets such as FairFace [Karkkainen and Joo, 2021], MS-COCO (MS) [Lin et al., 2014], and Pascal-Sentence (PS) [Rashtchian et al., 2010] are also used to show our method applies to various ranges of datasets and tasks.

6 Results and Discussions

We present analyses of our proposed approach to address research questions (RQ1 and RQ2) outlined in Sec. 5. We further include a parameter sensitivity analysis demonstrating how our choice of hyperparameters influences both bias mitigation and overall performance.

Evaluation of Skew-based Fairness (RQ1). Table 2 presents Skew-based metrics across diverse datasets and sensitive attributes. Our method consistently reduces MaxSkew and MaxSkew@k compared to the vanilla CLIP model (VA) and strong debiasing baselines (DeAR, SFID). Additionally, it shows comparable performance in image-to-text retrieval on Flickr30K while effectively lowering Skew values for sensitive attributes.

Figure 3 further illustrates these improvements through visual examples, where we use GradCAM [Selvaraju et al., 2017] to reveal how our method shifts focus away from sensitive attributes such as faces to more contextually relevant features when we use the CLIP model to classify images with different sensitive attributes in different scenarios.

Evaluation of Accuracy and Recall-based Performance (RQ2). Although our primary goal is bias mitigation, preserving task performance is also essential. Table 3 reports zero-shot classification accuracy and image retrieval Recall@k before and after debiasing, comparing our method with unmodified CLIP models and two baselines (DeAR, SFID). Our method maintains competitive performance while achieving improved fairness.

Parameter Analysis. Intuitively, a loss function containing only Ldiff is unsuitable, as a mapper trained solely on mismatched image-text pairs cannot accurately transform text embeddings into image embeddings. The weighting factor λ balances the alignment loss Lalign and the contrastive difference loss Ldiff, influencing how the mapper trades off embedding fidelity against discriminative power. While PCGrad [Yu et al., 2020] is used to mitigate gradient conflicts between these two objectives, λ itself remains a fixed hyperparameter. To study its impact, we evaluate three settings (λ = 0, λ = 0.5, and λ = 1). As shown in Figure 4, different λ values yield distinct trade-offs between bias mitigation and task performance. When λ is near zero, the model prioritizes alignment. The mapper lacks the ability to distinguish between matched and mismatched pairs, resulting in weak attribute direction signals extracted from attribute pairs. Conversely, at λ = 1, the contrastive term dominates: attribute subspaces become more sharply defined, yielding stronger bias mitigation but at the cost of embedding alignment and occasional drops in performance. An intermediate setting (λ = 0.5) achieves moderate bias reduction while retaining reasonable task accuracy. However, as shown in Table 2 and Table 3, even λ = 0.5 falls short of the stable bias-performance balance achieved when using PCGrad to dynamically reconcile both losses; manually tuning λ across fixed values is also computationally expensive for only marginal gains.

7 Conclusion

We present CABIN, a novel debiasing framework that systematically mitigates stereotypical biases in Vision-Language Models (VLMs) through causal inference while preserving the models' original capabilities. By using a lightweight mapper to identify and neutralize sensitive attribute directions in the joint embedding space and applying backdoor adjustment to remove confounding influences, CABIN substantially reduces Skew-based bias across demographic groups on downstream tasks (e.g., image classification and text-to-image retrieval) without requiring retraining. CABIN focuses on debiasing the model rather than specific downstream tasks, such that it can be easily extended to other applications. Experiments and ablation studies on multiple benchmarks demonstrate that CABIN achieves a favourable balance between fairness enhancement and performance retention, underscoring its adaptability to diverse real-world scenarios.

Acknowledgements

This research was supported by the Our Voices study (https://ourvoices.auckland.ac.nz), funded by the Ministry of Business, Innovation and Employment (2019-2025) under Endeavour grant UOAX1912, and a University of Auckland Masters scholarship awarded to the first author. The authors also gratefully acknowledge the Centre for Machine Learning for Social Good at the University of Auckland for supporting the first author's travel to present this work. The authors are especially grateful to the rangatahi / youth who participated in the Our Voices study, designed by Susan M.B. Morton and the Our Voices study team.