Empowering Vision Transformers with Multi-Scale Causal Intervention for Long-Tailed Image Classification

1 Introduction

Real-world data typically follows long-tailed distributions, resulting in models that primarily optimize for head classes and demonstrate limited generalization to tail classes [Zhang et al., 2023]. Existing CNN-based long-tailed algorithms including class balancing methods [Cui et al., 2019; Ren et al., 2020], data augmentation [Wang et al., 2024a; Ahn et al., 2022], enhanced training strategies [Wang et al., 2021; Du et al., 2023]. With the development of Transformers, ViT employs an attention-based global feature extraction approach, facilitating the capture of finer-grained features relative to CNN architectures. However, this does not change the essence of the model's reliance on statistical information from the data, leading to an overall performance gain but a persistent gap between head and tail class performance, leaving the long-tail problem unresolved.

∗Corresponding author

Long-tail image classification in ViT can be improved using two main approaches: parameter-efficient fine-tuning strategies and information enhancement. The former method [Li et al., 2024a] mainly leverages pre-trained knowledge to enhance the generalization of tail classes. LPT [Dong et al., 2023] designs visual prompts for group-wise categories to improve the learning of unique representations for tail classes, while LIFT [Shi et al., 2023] adopts a partial parameter fine-tuning approach to enhance the discriminative ability for tail classes. However, they struggle with tail classes that have high intra-class complexity by fine-tuning with a limited number of parameters. Information enhancement methods aim to augment the information for tail classes. VLLTR [Tian et al., 2022] leverages textual features to enhance the learning of image features, and DeiT [Rangwani et al., 2024] extracts information from pre-trained CNNs through knowledge distillation. These methods often struggle to obtain accurate knowledge of tail classes, leading to semantic confusion. The key to solving these problems lies in learning highly relevant visual features and mitigating spurious correlations caused by long-tailed distribution, which can be achieved through causal inference methods. However, directly applying existing causal methods [Tang et al., 2020; Zhu et al., 2022] to ViT fails to yield performance gains akin to CNN-based models due to they are difficult to model the spurious association between the fine-grained features and the predictions by calibrating the logits with the estimated category consistency bias. This leads to the problem that existing causal methods struggle to eliminate the spurious associations between a woman's image and related categories like "girl" or "table," with most of the misclassified tail categories being confused with similar categories, as shown in Figure 2.

To address these issues, this paper proposes a two-stage causal modeling framework by multi-scale causal intervention termed TSCNet, as shown in Figure 1. To enhance the model's fine-grained causal representation and mitigate the spurious associations on logits, we design two stages: hierarchical causal representation learning (HCRL) and counterfactual logits bias calibration (CLBC). HCRL enhances the model's fine-grained causal representation for tail classes by introducing class-independent semantic information such as background at both the patch-level and global feature-level. This enables the model to focus on class-relevant regions through hierarchical interventions. CLBC calibrates the spurious associations in label predictions caused by domain distribution from counterfactual perspective. By counterfactual generation and adaptively refining the intensity of counterfactual augmentation to construct different distributions, we effectively model category relationships and calibrate logits' bias caused by long-tailed distribution. The two-stage causal modeling method enables independent interventions on multiple biases, allowing TSCNet to maintain head class performance while improving tail class accuracy.

Experiments were conducted on two datasets, including performance comparison, ablation study, case study, and other in-depth analyses. The results confirm that TSCNet effectively enhances causal representations for tail data, while mitigating the logits bias caused by long-tailed distributions. The main contributions of this paper are:

• This paper points out that due to the different feature extraction, existing long-tailed causal methods face challenges when applied to transformer architectures.
• This paper proposes a two-stage causal framework with multi-scale interventions. To the best of our knowledge, it's the first causal framework that uses backdoor adjustment to remove various biases and is applicable to ViT.
• Experiments show that TSCNet effectively mitigates semantic and distributional biases, reducing errors in both head and tail class predictions.

2 RelatedWork

2.1 Long-tailed Image Classification

Previous studies addressing the negative impacts of longtail distributions have focused on three distinct aspects: class balancing methods, which enhance optimization for tail classes by designing resampling strategies [Cui et al., 2019; Dang et al., 2023], reweighting loss functions [Ren et al., 2020; Zhou et al., 2024], and adjusting logits [Hong et al., 2021; Liao et al., 2024]; data augmentation, which improve the information scarcity of tail classes through curriculum learning for image augmentation[Ahn et al., 2022; Wang et al., 2024a], transfer learning from head classes to tail classes [Chen and Su, 2023; Dang et al., 2024a], and feature enhancement for tail classes [Qi et al., 2023; Li et al., 2024b]; improving training strategies [Du et al., 2023; Fu et al., 2025], which typically involve decoupling representation learning from classifiers [Kang et al., 2019] and employing ensemble learning strategies [Zhou et al., 2020; Cui et al., 2022; Dang et al., 2024b; Dang et al., 2025] to further optimize both head and tail classes. Causal methods [Tang et al., 2020; Zhu et al., 2022] have shown remarkable performance improvements in CNNs by calibrating the logits bias induced by data distribution through backdoor adjustments. Transformer-based long-tailed methodologies[Xu et al., 2023; Zhu et al., 2024] primarily focus on finetuning strategies for ViT, including prompt tuning to enhance shared prompts for tail classes[Dong et al., 2023; Li et al., 2024a], parameter-efficient fine-tuning techniques[Shi et al., 2023] to facilitate the learning of tail classes. Some methods further enhance the representation of tail classes by incorporating external knowledge through visual-language contrastive learning[Tian et al., 2022] and knowledge distillation techniques[Rangwani et al., 2024].

2.2 Causal Inference in Image Classification

Causal inference and counterfactual reasoning have received increasing attention in a variety of tasks in computer vision, including scene graph generation[Sun et al., 2023], image recognition[Meng et al., 2025; Guan et al., 2023], and video analysis[Wang et al., 2024d; Dang et al., 2024c]. Causal methods in the field of image classification demonstrated significant performance improvements. Existing research has implemented backdoor adjustment strategies by designing causal classifiers[Liu et al., 2022], using attention mechanisms[Yang et al., 2023] to identify and mitigate the interference of confounding factors[Zhang et al., 2024]. Moreover, prevailing front-door adjustment strategies involve designing local-global feature attention mechanisms[Wang et al., 2024c] to extract distinguishable causal features. Additionally, causal invariant representation learning methods [Mao et al., 2022; Liu et al., 2024] utilize style generation models or Fourier transform techniques in conjunction with invariant loss functions[Lv et al., 2022] to improve the identification of causal factors. However, existing causal methods in image classification are inadequate for addressing the longtailed distribution problem, as they overlook the influence of long-tailed bias on causal graph construction and the limited effectiveness of interventions in tail classes with sparse data.

3 ProblemFormulation

The long-tailed dataset is represented as D = {x, y}. Let nj denote the number of training sample for class j, and let n = ∑Cj=1 nj be the total number of training sample and n1 ≫ nC. Conventional methods extract visual features : Fv = Mv(x), where Mv(·) denotes the feature extractor. Then, predicting the category of the sample, i.e. P = classifier(Fv). In the first stage, we initially extract confounder S = [s1, s2, ...sn] and perform causal interventions P(Y |do(X)) at both the token level and the global feature level, thereby obtaining a causally enhanced model M(·). In the second stage, we construct a counterfactual balanced distribution x through counterfactual data augmentation F(x,Lec ) while adaptively adjusting the intensity Lec to perform causal interventions P(Y |do(D)). This process can get predictions after calibration Pc = M(x).

4 Method

This study proposes a two-stage debiasing method for longtail learning, called TSCNet. The method constructs a structured causal graph to analyze the interfering factors in the inference path. Specifically, TSCNet consists of two main stages, as shown in Figure 4. The Hierarchical Causal Representation Learning (HCRL) stage enhances the model's finegrained causal representation through debiasing at the patch and global feature levels eliminating semantic confusion. The Counterfactual Bias Calibration (CLBC) stage utilizes counterfactual data augmentation and refinemention strategies to reduce the logits' bias caused by data distribution.

4.1 Causal View at Long-tailed Classification

We use the structural causal model to model the variable relationships of complex spatiotemporal data in long-tailed image classification tasks. As illustrated in Figure 3, it is a directed acyclic graph G = {N,E} in which the nodes N denote variables and edges E denote association between variables. The SCMG includes four variables: image X, semantic confounder S, data distribution confounder D and prediction Y . The correlations in graph G are as follows:

• S -→X. This path indicates that similar semantic factors such as the background influence the composition of image content and tend to affect the tail classes where data is sparse.
• S -→ Y . This path indicates that the predicted label distributions(logits) follow their own training domain prior.
• D -→X. This path indicates that an image is sampled according to the selected data distribution,e.g., imbalanced data distribution is prone to head classes.
• D -→ Y . This path indicates that the predicted label distributions follow their own training domain prior.
• X ← S -→ Y , X ← D -→ Y . This two back-door path contribute spurious correlation between X and Y , where S and B acts as confounder.

4.2 Hierarchical Causal Representation Learning

To eliminate the semantic confusion between tail classes and head classes, we propose a hierarchical causal representation learning method. By extracting class-agnostic information and performing causal interventions at both the patch and global feature levels, the method enhances the model's finegrained representation learning for tail classes.

Mitigating the bias caused by S is to intervene on X, ensuring that class-agnostic semantic information contributes equally to the image classification. We extract class-agnostic semantic information and intervene on X:

where do(X) denotes intervene on X, the path in Fig. 3 from S to X is cut-off. Due to the inability to combine all classagnostic information with the image, only approximate interventions are possible. We propose a hierarchical intervention strategy to strengthen the intervention for tail class and improve the fine-grained causal representation of tail classes.

Patch-level Intervention

At the patch-level intervention, we introduce class-agnostic patch information alongside the original image patches, leveraging the encoder's attention mechanism to finely uncover the causal regions within the image. Specifically, we can apply this method on CNNs to merge the original image with theclass-agnostic information.

Confounder Dictionary S We extract class-irrelevant semantic information from the training set and construct a confounder dictionary. Given an image xi, it uses off-the-shelf methods such as Grad-CAM [Selvaraju et al., 2020] to detect the main subject of the image. Then, we apply an inverse transformation to obtain the class-agnostic information mask Mi. Then we paste the mask onto the original image by Mi ⊙ xi to obtain a class-irrelevant image si.Then, TSCNet constructs the confounder dictionary S = [s1, s2, ..., sn].

Patch-level intervention requires multiple backpropagations over all irrelevant information. To reduce training cost, we assume a uniform distribution of confounding patches and apply a random sampling strategy:

where sk is randomly sampled from the confounder dictionary S, E(·) is the patch embeddings of the image, and f(x, sk) is the function that indicates stacking and concatenation on the sequence length dimension, the length of the sequence can be changed by random sampling from E(sk).

Then,we can get the causal-enhanced representation:

where φ(.) is the visual backbone.

Feature-level Intervention

To further mitigate the effects caused by class-agnostic semantics on the feature distribution and enhance the model's generalization ability for tail classes, TSCNet introduces a global feature causal intervention module.

We construct a confounder prototype dictionary Sp = [c1, c2, ..., cl] structured as an l ∗ d matrix to systematically address these feature-level factors. Where l is the dictionary size and d is the feature dimension using the pre-trained backbone. We apply k-means++ to derive ci from the confounder dictionary S, each ci is the average feature of its cluster. To implement the theoretical interventions in Eq 2 and reduce computation, we use Normalised Weighted Geometric Mean [Xu, 2015] to approximate the results expected from the above feature layers:

We parameterize the network model to approximate the conditional probability of Eq.6, inspired by[48], as follows:

where Wa ∈ Rdm×da and Wb ∈ Rdm×d are learnable parameters. We approximate Ec[g(c)] as a weighted integration of all background prototypes:

where µi represents the important weight coefficient measuring the interaction between each ci and the feature Fv.

4.3 Counterfactual Logits Bias Calibration

Although we obtained features of causal enhancement in the first stage, the model relies on this long-tail distribution prior, leading to bias in label prediction. Therefore, we propose a model-agnostic counterfactual intervention method, which generates a balanced distribution through counterfactual augmentation and refinement to adaptively calibrate the logits:

where d = 0 denotes the imbalanced data distribution, and d = 1 denotes the balanced data distribution.

Counterfactual Generation

Simple balanced sampling is not applicable as it leads to the model's overfitting to tail classes. We use Fourier transformation [Lv et al., 2022] to perform counterfactual augmentation by disturbing non-causal factors to construct a counterfactual balanced distribution:

where A(x), P(x) denote the amplitude and phase components respectively. We then perturb the amplitude information via linearly interpolating between the amplitude spectrums of the original image x and an image x′ sampled randomly:

where λ ∼ U(0, Lec ) and Lec controls the strength of perturbation, which adjusts the perturbation strength for class c during an epoch e. Then we can obtain the counterfactual augmented image. We can obtain the counterfactual augmented image:

Counterfactual Refinement

The counterfactual refinement module adaptively adjust the strength Lec of counterfactual data augmentation to enable the model to progressively adjust the logits bias from easy to difficult. At epoch e, we define a computation function Pl for each class to adaptively update the perturbation strength:

where γ is threshold hyperparameter, M(·) is the model from stage 1. We can update VLoL as follows:

where Acc is a function which outputs the number of correctly predicted examples by the model fθ.

After updating Lec , TSCNet control the intensity of generating the non-causal factor ˆ A(xo ) and conterfactual augmented image xa to construct a balanced dataset x = (xB , yB )∼τ B . Then, We can get Pz = M(x).

4.4 Training Strategies

The training of TSCNet follows two steps: the first step is de-confounded training for HCRL. After obtaining a causal representation-enhanced model, counterfactual fine-tuning is subsequently applied for CLBC. The details are as follows:

The HCRL stage uses causal intervention modules at the patch and feature levels. The process is constrained by:

The CLBC construct and refine counterfactual distribution x to mitigate the long-tail bias. The process is constrained by:

where αgf is the weight factor, M(.) is the model from step 1, x′ is a counterfactually augmented sample of x.

5 Experiments

5.1 Experiment Settings

Datasets

Experiments are conducted on two datasets: CIFAR-100LT and the more challenging VireoFood-172 [Chen and Ngo, 2016] of 66,071 training and 33,154 test images.

Evaluation Protocol

For CIFAR-100LT dataset, we evaluated Top-1 accuracy under three different imbalance ratios: 100/50/10. For VireoFood-172, we evaluated Top-1 accuracy under an imbalance ratio of 50. We followed TDE [Tang et al., 2020] and CMLTNet [Li et al., 2024c] to test the performance of the head, middle, and tail classes in the CIFAR100 dataset and VireoFood-172 dataset.

Implementation Details

For CIFAR100-LT, we use warm-up scheduler for fair comparisons. All models were trained by using SGD optimizer with momentum µ = 0.9 and batch size 64. The learning rate was decayed by a cosine scheduler from 0.01 to 0.0 over 200 epochs for the ResNet50 and 40 epochs for the ViT and VPT . For VireoFood-172-LT, all models were trained by using Adam optimizer with momentum µ = 0.1 and batch size 64. The learning rate is chosen in the range of 1e-4 to 5e5. The learning rate decays every 4 epochs, with each model decaying 3 times by a factor of 0.1.

5.2 Performance Comparison

We conducted a comprehensive comparison involving 3 visual modal backbones, 3 causal methods and 8 long-tailed methods: ResNet50 [He et al., 2016], ViT [Alexey, 2020], VPT[Jia et al., 2022], CCIM [Yang et al., 2023], GOAT [Wang et al., 2024b], CaDeT [Pourkeshavarz et al., 2024], TDE [Tang et al., 2020], xERM [Zhu et al., 2022], PLOT [Zhou et al., 2024], LiVT [Xu et al., 2023], Gpaco [Cui et al., 2023], H2T [Li et al., 2024b], LPT [Dong et al., 2023]. To make a fair comparison, the hyper-parameters of all models are chosen in above section. The following observations are drawn from Table 1:

• TSCNet achieved significant improvements across different vision networks especially in the Transformers. This is due to multi-level causal interventions that enhance the model's fine-grained causal representations, along with the introduction of a model-agnostic counterfactual bias calibration strategy.
• TSCNet generally achieved better performance than other algorithms in both datasets. The two-stage causal debiasing framework has been validated, demonstrating significant improvements in tail class performance while maintaining head class performance.
• Causal methods for image classification enhance head class performance but offer limited gains for tail classes. They enhance causal representation for data-rich head classes, but fail to provide fine-grained

causal representations for tail classes and do not address decision boundary bias in long-tail distributions.

• Long-tail algorithms boost tail class performance but degrade head class. Existing long-tail algorithms perform poorly on the imbalanced VireoFood-172 test set due to spurious associations, which cause a trade-off between head and tail class performance.

5.3 Ablation Study

In this section, we further studied the working mechanism of each module of TSCNet, as shown in Table 2:

• Hierarchical causal interventions(+I+F) effectively improve the feature representation for tail classes. Incorporating patch-level intervention (+I) and featurelevel intervention (+F) can significantly improve ViT's performance for tail classes, but there is still a significant gap compared to the head classes.
• Counterfactual generation (+C) helps further optimize the decision boundary. Training the model with a counterfactual balanced distribution (+C) leads to a significant improvement in tail class performance. However, the uncontrollable counterfactual strength leads to performance decline in the head classes.
• Counterfactual refinement (+R) enhances learning for both head and tail classes. By adopting a difficultyprogressive refining strategy (+R), it simulates different data distributions to finely model the spurious correlations between head and tail logits.

5.4 In-depth Analyses

Effectiveness Analyses of the Dictionary Sp

As delineated in Table 3, we scrutinized the efficacy of the class-non confounder dictionary Sp within the BD module. Experimental results show that replacing Sp with a random dictionary or a zero dictionary significantly deteriorates performance. Using average Image features as a confounder dictionary is less effective than class-agnostic confounder features, indicating that random dictionaries and class averages, among others, are insufficient as confounder.

Effect of the counterfactual enhancement parameter Lec As shown in Figure 5, compared to the multiple fixed parameters (0.36, 0.64), the adaptive adjustment of the enhancement parameter Lec achieves performance improvements on most categories of the VireoFood-172 dataset. This effectively demonstrates that the designed adaptive enhancement parameter Lec can progressively increase the counterfactual data strength from easy to difficult and assign appropriate enhancement intensity to each class.

5.5 Case Study

Visualization of the Causal Representation by HCRL

To assess the efficacy of HCRL, we conducted a comparative evaluation against CCIM [Liu et al., 2022] on the CIFAR100 validation set, emphasizing causal visual information in long-tail data using GradCAM [Selvaraju et al., 2020] heatmaps as shown in Figure 6. CCIM demonstrated weak causal perception for tail-class images such as "wolf" and "woman." In contrast, the proposed patch-level intervention method (PCLT) enriched fine-grained causal representations, enabling the model to capture intricate features, such as facial details in "woman" and "mouse," as well as edge features in "flowerpot," effectively mitigating the interference of irrelevant information. Furthermore, HCRL improved the model's attention, enhancing its intervention on tail-class.

Visualization of the Decision Boundary by CLBC

We utilized tSNE visualization to illustrate the feature distribution and decision boundaries of three commonly confused head-tail categories in the CIFAR100 dataset, as shown in Figure 7. We observed a clearly separable boundary for the three categories in the feature space through HCRL. However, due to the confounding effect of the data distribution, a significant number of tail-class samples were misclassified as head-class samples. By incorporating the CLBC module and constructing a counterfactual balanced distribution, the model's bias towards tail classes at the decision boundary was HCRL+CLBC

Figure 8: The error confusion analysis of TSCNet, and the comparison of xERM and LPT in terms of similar-FP and non-similar-FP in CIFAR100.

effectively mitigated. Notably, the CLBC module further enhanced the model's representation of long-tail data.

Error confusion analysis of TSCNet

We compared the error confusions of different models with our method, as shown in Figure 8. The error confusions were categorized into two types: similar-class confusion and nonsimilar-class confusion. The results demonstrate that TSCNet effectively alleviates error confusions in similar classes by leveraging fine-grained causal representations and bias calibration, particularly in the middle and tail classes, with a more significant improvement observed in the middle classes. In contrast, xERM and LPT faces challenges in modeling the relationship between feature regions in ViT and model predictions, leading to a concentration of errors in similar classes.

6 Conclusion

To effectively mitigate the biases induced by long-tail distributions and tackle the challenges associated with applying existing causal methods to ViT, this paper introduces TSCNet. It strengthens the model's fine-grained causal representation through hierarchical causal representation learning. Furthermore, TSCNet employs a model-agnostic counterfactual logits bias calibration stage to adaptively eliminate the prediction biases caused by long-tail distributions. Experimental results indicate that the synergistic interaction of the two stages significantly enhances long-tailed image classification performance across various backbones. Future work will focus on exploring the use of causal methods in large models to further improve long-tail performance.

Acknowledgments

This work is supported in part by the Shandong Province Excellent Young Scientists Fund Program (Overseas) (Grant no. 2022HWYQ-048).