MVP-CBM: Multi-layer Visual Preference-enhanced Concept Bottleneck Model for Explainable Medical Image Classification

1 Introduction

Deep neural networks have achieved remarkable success in various visual recognition tasks [He et al., 2016], yet a key challenge they confront is the internal "black box" decisionmaking process, lacking the desired model transparency and poses safety concerns especially for life-critical medical image classification [Vuong et al., 2014], e.g., cancer diagnosis. Consequently, transforming medical image classification models into more interpretable "white boxes" is both a critical and challenging endeavor [Hou et al., 2024].

In recent years, the Concept Bottleneck Model (CBM) has emerged as a promising interpretable neural network [Koh et al., 2020], providing explanations in terms of high-level, human-understandable concept attributes. Its key idea is to learn an intermediate bottleneck layer, in which each element corresponds to an interpretable concept, and constrain the model to make decisions based on this bottleneck layer. Therefore, the model's decision factors can be explicitly explained based on the concept activation values in the bottleneck layer. The early CBM paradigm is based on bottleneck-layer supervised learning of manually annotated concepts [Kim et al., 2023; Zarlenga et al., 2022], which leads to labor-intensiveness and concept-incompleteness. Recent advancements in large language models have paved the way for automatically generated concepts [Yang et al., 2023; Oikarinen et al., 2023; Shang et al., 2024], reducing the dependency on manual labeling and improving interpretability. Generally, Fig. 1-(a) displays a common paradigm [Gao et al., 2024] that most existing CBM methods are based on. It first extracts the image features of the last standard layer of a pre-trained visual encoder, then calculates the similarities between the extracted visual features and concept textual features that are automatically constructed as the bottleneck layer activations, and finally utilizes the activations to make classification predictions.

However, we argue that the existing CBM paradigm typically ignores the so-called concept preference variation, that is, visual features in different layers may exhibit varying association abilities to the concepts. Only leveraging the visual output of the last standard layer weakens the model's ability to accurately correspond to concepts, hence impairing interpretability. As shown in Fig. 1-(c1), based on the existing state-of-the-art CBM-based skin disease diagnosis model [Gao et al., 2024], we show the activation degree distribution of different visual layers and concepts, where each index on the vertical axis denotes a concept and the horizontal axis denotes the concept activation amplitude. For example, for the concept of '1strawberry pattern, glomerular vessels' (index 18), the visual features of the 7th layer have a higher activation than those of the 9th and 11th layers, indicating that it prefers to recognize and explain this concept. It is generally understood that the shallow-to-deep layers in the visual extraction process are usually considered to encode different levels of information [Yao et al., 2024], which also leads to preference variance in correspondence with conceptual attributes [Zhang et al., 2022]. Moreover, as depicted in Fig. 1(c2), we evaluate the accuracy of concept prediction using layer-specific features, which further proves our motivation that the default standard output layer may not be optimal. As a result, ignoring these layer-specific preferences may hinder the accurate discovery of visual concepts and compromise the transparency of decision-making processes.

To address the above limitations, we propose a novel multilayer visual preference-enhanced concept bottleneck model (MVP-CBM), which explicitly accounts for the concept preferences of each visual layer, and integrates concept activation fusion across layers to comprehensively enhance both the model's interpretability and accuracy, as in Fig. 1-(b). Specifically, our method contains two modules: 1) Intra-layer Concept Preference Modeling (ICPM), which takes into account the preferences of different visual layers for different concepts. Rather than treating all layers equally, this approach recognizes that different layers may focus on different aspects of a concept, from low-level features to high-level semantic representations. 2) Multi-layer Concept Sparse Activation Fusion (MCSAF), which is designed to sparsely aggregate multi-layer concept activations to enhance model performance and interpretability. Instead of relying solely on the activations from the final output layer or a single concept bottleneck layer, MCSAF captures the concept-specific information at various intermediate layers, each of which may focus on different aspects of visual or semantic features.

Our contributions are summarized as follows:

• To the best of our knowledge, we are the first to discover the concept preference variance across visual layers in the CBM and leverage this to improve model interpretability.
• We propose a novel method, MVP-CBM, which captures the preferences of different concepts across various visual layers and sparsely aggregates multi-layer concept activations to enhance model performance.
• Through extensive experiments on several public medical classification benchmarks, MVP-CBM achieves state-of-the-art accuracy and interpretability, outperforming existing CBM methods and black-box models.

2 RelatedWork

Deep neural networks have shown exceptional performance in medical image classification, but their black-box nature limits trust among healthcare professionals and patients [Hou et al., 2024]. To improve interpretability and reliability, Concept Bottleneck Models (CBMs) have been developed to map complex visual features to human-understandable concepts, providing explanations for model decisions.

Recent advancements in CBMs have primarily focused on addressing the challenge of manual concept annotation. For example, PCBM [Yuksekgonul et al., 2022] leverages ConceptNet [Speer et al., 2017] to generate a concept library and calculates the projection distance between CLIP text concept embeddings and visual representations to define concept bottlenecks, reducing the need for manual labeling. Building upon this, LaBo [Yang et al., 2023] enhances CBMs by employing large language models (LLMs) to generate candidate concepts and utilizing sub-model optimization blocks to select the most relevant ones. Explicit [Gao et al., 2024] further advances this line of work by leveraging LLMs and Q-former mechanisms to improve automation and flexibility in medical concept alignment [Zhang et al., 2023a].

Despite these advancements, existing CBM methods primarily match visual representation with concept embeddings at the standard output layer, overlooking the fact that different concepts may prefer different visual encoder layers, which can compromise decision-making transparency. Our work addresses this limitation by introducing two key innovations: (1) constructing concept preference for each visual sub-layer, which ensures that each concept is aligned with the most relevant features of the model, and (2) applying multi-layer concept sparse fusion, which aggregates concept activations from multiple layers to improve transparency and interpretability.

3 Methodology

3.1 Overview of MVP-CBM

The overall framework of our proposed Multi-Layer Visual Preference Concept Bottleneck Model (MVP-CBM) is shown in Fig. 2, which aims to incorporate human-understandable concepts into the decision-making process, thus making the 'black box' model more transparent and comprehensible for humans to increase trust in the model's predictions. Formally, the input data is denoted as D = {(x, y)}, where x represents the image and y ∈ Y is its corresponding disease category. The disease y is generally defined with a set of diagnostic attributes A = {Ai}mi=1 , where each attribute Ai = {aji }kj=1 comprises k distinct concepts, which can be generated by large language models (e.g., ChatGPT) or human experts.

As we discussed in the Sec. 1, existing CBMmethods typically predict based on the bottleneck layer activations, i.e., calculating between the image feature v extracted on the last standard layer of a pre-trained visual encoder and the predefined concepts as:

where Ψ is a linear layer to perform label prediction.

However, our preliminary investigations, as illustrated in Fig. 1 (c1) and (c2), reveal that different visual layers exhibit distinct preferences for various aspects of concepts. These preferences range from low-level features to high-level semantic representations. Relying solely on the visual features of the standard output layer may result in inconsistencies between concepts and visual data, potentially compromising the fidelity and interpretability of the model. Therefore, we propose a novel framework that first models the preferences of each layer for different concepts and then integrates concept activations across multiple layers. Firstly, we define the concept preference function for a given layer ℓ as follows:

where pℓ,i denotes the captured preference degree between the ℓ-th visual layer's feature vl and the i-th concept attribute.

Then, the multi-layer preference-enhanced concept bottleneck layer activations are calculated to make prediction as:

in which ℓ ∈ [1,L], i ∈ [1,m], and L denotes the total number of visual layers, and agg(·) represents the fusion of concept activations across all layers. This design ensures that contributions from all layers are appropriately and comprehensively considered, leading to more accurate and interpretable concept bottleneck model.

3.2 Key Components of MVP-CBM

Visual Representation

Different from previous CBM methods only using the last standard layer of the pre-trained visual encoder, we extract each visual layer feature of the encoder as:

Diagnosis Criteria ( LLM-Driven/Human Experts )

where the class token vcls ℓ ∈ Rd and patch tokens Vpatch ℓ = {vpatch ℓ,1 , vpatch ℓ,2 , . . . , vpatch ℓ,Np } ∈ RNp×d denote global and local features, respectively, in a specific ViT model layer ℓ ∈ [1,L]. vpatch ℓ,n ∈ Rd denote the n-th patch token from layer ℓ, and Np is the total number of patches. Same with the existing method [Gao et al., 2024], the visual encoder is adopted as the BioMedCLIP model [Zhang et al., 2023b].

Concept Representation

As shown in Fig. 3, to define concept representation in medical imaging diagnosis, we start with identifying key diseaserelated concepts that are crucial for diagnosing specific conditions. These concepts are derived from medical literature, clinical guidelines, and expert knowledge [Gao et al., 2024], ensuring they capture important diagnostic criteria. Specifically, the pre-trained BiomedCLIP's text encoder is used to extract the representation of each concept aji as:

where tji ∈ Rd represents concept-based feature. m is the number of diagnostic attributes, k is the number of concepts per attribute, and d refers to the dimension of text features.

Intra-layer Concept Preference Modeling (ICPM)

Here, in contrast to existing methods that ignore the variance of the corresponding preferences between different visual layers and concepts [Gao et al., 2024], our goal is to explicitly reveal the degree of concept preference at each visual layer. Considering that the concepts within an attribute reflect the same characteristic, we propose to quantify the preference value through the semantic correspondence between the global concept attribute feature and the visual layer features, i.e., the degree to which the visual layer tends to explain the concept attribute.

Specifically, we concatenate all concepts within the same attribute to form an input that allows us to obtain an attribute global-level feature Ti for each Ai = {aji }kj=1 . Formally,

where Ti ∈ Rd encapsulates the collective semantic information of all concepts under a diagnostic attribute.

Then, in the intra-each visual layer, we compute the semantic correspondences between different attribute features {Ti}mi=1 and the visual layer feature vcls ℓ as:

where σ(·) denotes the Sigmoid activation function.

The value of pℓ,i reflects the probability of association ability between the ℓ-th visual layer and different attributes, in which the larger pℓ,i is, the more helpful the visual layer is in preferentially explaining the i-th concept attribute. Moreover, to further amplify the preference degree between different attributes, a learnable scaling parameter τ1 is introduced. Therefore, the preference degree is calculated as follows:

where pℓ,i is designed to emphasize the preference for different attributes within the ℓ-th layer.

Multi-layer Concept Sparse Activation Fusion (MCSAF)

To fully exploit inter-layer concept preferences while reducing redundancy introduced by inter-layer concept activations, we propose a multi-layer concept sparse activation fusion method. First, we compute fine-grained concept activation scores, weighted by the visual layer preferences. Next, to enhance the effectiveness of the most relevant visual layers for each concept, we calculate sparse activation weights for different visual layers. Finally, we aggregate the most representative concept activation scores after applying sparse activation to make reliable decisions.

Firstly, to alleviate the issue of an excessive number of local visual patch tokens in each visual layer, for the i-th diagnostic attribute, we only aggregate fine-grained semantic features related to the corresponding attribute from different layers, which is calculated as:

in which fℓ,i denotes the adaptive average pooling that downsamples the local visual patch tokens. vpool ℓ,i ∈ Rd represents the pooled semantic feature corresponding to the i-th diagnostic attribute in the ℓ-th visual layer.

The activation score of the concept within the i-th attribute across all layers is expressed as:

where the attribute preference pℓ,i is exploited to weight the concept activation scores. This ensures that each concept activation score is adjusted according to the corresponding attribute preference degree. That is, the concept activation, whose attribute has a higher association preference, is amplified, and vice versa.

Afterward, since different attributes exhibit varying preferences for visual layers, and not all concepts within a layer contribute equally to the final prediction, we propose a method to sparsify the less relevant visual layers. This maximizes the utilization of the most effective layers while reducing potential redundancy, improving the model's robustness. Concretely, we compute the concept contribution weights across all layers as follows:

To sparsify these weights, we introduce a thresholding mechanism that adaptively adjusts the activation weights of concepts across different visual layers.

where eτ2(·) serves as an adjustment factor to scale w(i,j),ℓ to introduce sparsity. Here, τ2 is a learnable scaling parameter. The threshold θℓ is adaptively adjusted based on the importance of each concept's activation weights, as defined by: θℓ = σ(K)(wℓmax -wℓmin) +wℓmin, where K is a learnable parameter that modulates the threshold within the range of [wℓmin,wℓmax].

Furthermore, we define a binary hard mask M(i,j),ℓ for each layer ℓ to filter out insignificant activation weights:

Then, the sparse activation weights are obtained by applying this binary mask:wsparse (i,j),ℓ = M(i,j),ℓ ⊙ wadjust (i,j),ℓ , (14)

where activation weights with absolute contributions below the threshold θℓ are compressed towards zero to effectively avoid their interference.

Finally, we aggregate the concept activation scores for each concept across all layers:

where sagg (i,j) denotes the fused concept activation, which comprehensively and sparsely combines the contributions from multi-layers.

To encourage sparsity in the activation weights, we introduce a sparse loss term defined as the mean of the binary masks across all layers:

which penalizes the average number of active concepts, thereby promoting fewer active concepts and enhancing the model's interpretability.

Classification and Training

The fused concept activations of all concepts sagg (i,j) are concatenated into a vector sagg c ∈ R1×(m·k) , facilitating subsequent classification layer to obtain predicted disease logits ˆ y:

where Ψ : Rd → Y denotes a linear classifier.

The model is trained end-to-end by minimizing the combined loss:

where y and yc are the disease label and concept label, respectively. sc ∈ sagg c is the specfic concept activation score.

λ1 and λ2 are hyperparameters that balance the importance of accurate concept prediction and sparsity in concept activations. By jointly optimizing these parameters, the model learns to identify and emphasize the most relevant concepts across layers while maintaining sparsity, thereby enhancing both interpretability and predictive performance.

4 Evaluation

Datasets: (1) ISIC2018 [Tschandl et al., 2018] contains 10015 dermoscopic images with seven skin lesion categories for skin cancer classification. (2) NCT-CRCHE(NCT) [Kather et al., 2018] includes 100k patch-based histological images of human cancer with nine tissue classes for classification. (3) IDRiD [Porwal et al., 2018] consists of 455 retinal fundus images annotated with 5 severity level grading of diabetic retinopathy. (4) BUSI [Al-Dhabyani et al., 2019] dataset contains 780 ultrasound images of breast masses categorized into normal, benign, and malignant classes for breast cancer classification. (5) CMMD [Cui et al., 2021] includes 1,775 x-ray images with benign and malignant. (6) Cardiomegaly (CM) [Johnson et al., 2019] is used for binary classification, which is the x-ray image in IMIC-CXR. (7) SIIM-ACR Pneumothorax(SIIM) [Zawacki et al., 2019] has 12,047 x-ray images for binary classifcation.

Implementation Details: We prompt GPT-o1 to query domain knowledge and diagnostic criteria. The details are listed in appendix. MVP-CBM and Explicd are implemented based on BioMedCLIP for all datasets. The fine-tuning of MVPCBM and Explicd involves optimizing visual encoder, visual concept learning module, and the final linear layer with AdamW optimizer, while keeping the text encoder fixed. For the hyperparameters in MVP-CBM, the initial values are set as follows: τ1 and τ2 are both initialized to 0.2, and K is initialized to 0. All experiments are conducted using PyTorch with an Nvidia 4090 GPU. To reduce randomness, we run 5 times and report mean and standard deviation. Given class imbalance in medical data, we report both balanced accuracy (BMAC) and accuracy (ACC) for comprehensive evaluation.

4.1 Comparison with State-of-the-arts

We compare our method against several baselines: (1) Zeroshot Models, including CLIP and specialized biomedical VLMs MedCLIP and BiomedCLIP [Wang et al., 2022; Zhang et al., 2023b], which perform classification without task-specific fine-tuning; (2) Supervised Black-box Models, such as ResNet50 [He et al., 2016] and Vision Transformer(ViT)-based models [Dosovitskiy, 2020], both fine-tuned for the classification task; and (3) Explainable Models, including LaBo [Yang et al., 2023], PCBM [Yuksekgonul et al., 2022], and the state-of-the-art Explicd [Gao et al., 2024], which utilizes predefined diagnostic criteria for interpretable medical classification.

As shown in Table 1 and Table 2, Vision-Language Models (VLMs) like CLIP, MedCLIP, and BiomedCLIP perform poorly in zero-shot settings, with CLIP near random levels. Although BiomedCLIP achieves higher accuracy on the CMMD dataset due to its medical pretraining, it still lags behind supervised models such as ResNet50 and ViT-Base on other datasets, highlighting limited generalization.

In the explainable setting, LaBo shows lower accuracy because it relies on well-aligned VLMs, highlighting the challenge of balancing accuracy and explainability. PCBM improves upon LaBo by utilizing a robust concept bank but still falls short of Explicd, which integrates human knowledge and visual concept learning. In contrast, MVP-CBM integrates concept preferences from multiple layers of the visual encoder, aligning diverse visual features with diagnostic criteria to create more comprehensive representations. This multilayer approach enables MVP-CBM to outperform state-ofthe-art models like Explicd and other black-box methods across all evaluated datasets.

4.2 Ablation Study

We conduct an ablation study on the ISIC2018 dataset to evaluate the effectiveness of various components and modules within the MVP-CBM, as shown in Table 3. No. 2-4: Removing key components such as Intra-layer Concept Preference Modeling (ICPM), concept loss, and sparse loss led to significant declines in both BMAC and ACC, underscoring their essential roles in maintaining model performance.

For the impact of modifying specific modules. No.5: Replacing the adaptive pooling mechanism with a Q-former in the Multi-layer Concept Sparse Activation Fusion (MCSAF)

results in a notable performance degradation, likely due to increased complexity and potential overfitting. Similarly, No.6: Substituting the hard mask with a soft mask in the MCSAF allows for more concepts to be activated simultaneously and introduces noise, diluting the focus on the most relevant concepts. No.7-8: Replacing sparse concept activation fusion with Layer Adaptive Weighted Fusion (LAWF) and a MultiHead Attention Mechanism (MHAM) in the MCSAF further impairs performance. LAWF tends to overemphasize certain layers while neglecting others, leading to incomplete or imbalanced concept representations. MHAM fragments attention across different heads, hindering the coherent consolidation of concept activations.

For hyperparameter study. No.9-10: Removing the learnable parameters τ1 and τ2 from the ICPM and MCSAF results in a significant decline in model performance. In the ICPM, τ1 adjusts the importance of concept preferences, thereby enhancing the model's ability to understand and differentiate high-level semantic concepts. Meanwhile, in the MCSAF, τ2 determines which concepts are activated in the multi-layer concept sparse activation fusion, thereby achieving sparsity.

4.3 Qualitative Analysis

Concept preference in ICPM. Fig. 4 shows how concept preferences vary across layers of the model, highlighting the differential preference of various concepts at each layer. For example, in the left plot, the concept 'Dermoscopic Patterns' exhibits the highest preference in the lower layers (Layer 8), whereas 'Border' and 'Shape' show lower preference across the early layers but tend to increase in higher layers (Layer 12). This indicates that certain concepts are more dominant at specific layers, supporting the idea that different concepts align better with different levels of the visual representation. In the right plot, 'Border' shows a relatively consistent preference across all layers, while concepts like 'Texture' and 'Symmetry' peak around Layer 8, indicating a stronger concept preference to intermediate layers. The varying preferences of these concepts across layers demonstrate that the final output layer does not necessarily contain the most optimal preference for all concepts.

Sparse concept activation in MCSAF. Fig. 5 compares the original concept activation and sparse concept activation across different layers of the model. On the left, the Original Concept Activation shows a dense activation pattern, where many concepts are activated across all layers, indicating a broad and less focused distribution of activations. In contrast, the Sparse Concept Activation on the right displays a more focused pattern, where only a few concepts are activated at each layer, leading to more concentrated and sparse activations. This difference highlights the effect of sparse activation, which reduces noise by activating only the most relevant concepts at each layer, thus improving interpretability by focusing on the most relevant information for predictions.

4.4 Interpretability

We visualize the concept bottleneck layer of both MVP-CBM and the state-of-the-art method Explicd [Gao et al., 2024], focusing on how they rely on different concepts to make predictions. For each image across six datasets, we first normalize the activation scores of all concepts, then display the top 5 concepts with the highest activation scores in the concept bottleneck layer.

As shown in Fig. 6-(f), MVP-CBM accurately diagnoses a Benign condition, as evidenced by the mammogram showing characteristics typical of benign masses such as 'high density' and 'well-defined borders'. High contribution values for concepts like 'High Density' and 'well-defined borders' further reinforce this diagnosis. In contrast, Explicd incorrectly labels the condition as Malignant, placing undue emphasis on 'irregular Calcifications'. This misinterpretation overlooks the clear, smooth borders and absence of irregular calcifications that are hallmarks of benign lesions. Similar discrepancies between the two methods are observed across other datasets. Given that both the CM and SIIM datasets consist of chest X-ray images, we focus the visualization on the SIIM dataset.

5 Conclusions

To the best of our knowledge, this is the first work to focus on the variance of concept preferences across visual layers within Concept Bottleneck Models (CBMs) to improve both model performance and interpretability. In this paper, we introduce the Multi-Layer Visual Preference Enhancement Concept Bottleneck Model (MVP-CBM), which incorporates two key components: Intra-layer Concept Preference Modeling (ICPM) and Multi-Layer Concept Sparse Activation Fusion (MCSAF). ICPM captures the varying preferences for concepts across different visual layers, allowing the model to leverage the most relevant layers for each concept. Meanwhile, MCSAF sparsely aggregates concept activations across layers, reducing noise and improving accuracy. Experimental results on multiple datasets demonstrate that our approach outperforms existing methods, setting a new standard for both model transparency and predictive performance.

Acknowledgments

This work is supported by the National Natural Science Foundation of China under Grant 62271465, the China Postdoctoral Science Foundation under Grant 2024M763178, and the Suzhou Basic Research Program under Grant SYG202338.