ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition

1 Introduction

The emergence of deep neural networks (DNNs) has created unprecedented achievements in machine learning, thanks to their powerful capabilities of learning representations [Talaei Khoei et al., 2023]. However, the lack of transparency hinders DNNs' wider applications in areas requiring trace-

Why is this bird classified as a Painted Bunting?

able and understandable decisions. To further explore the interpretability of DNNs, many researchers propose various approaches to promote the advancement in explainable artificial intelligence (XAI) [Gao and Guan, 2023]. Among these methods, ProtoPNet [Chen et al., 2019], inspired by the human vision system, has attracted increasing research interest and many follow-up studies with its self-explanatory property for XAI. Considering the example image in Fig. 1, we can identify this sample as a Painted Bunting by comparing the features of its beak, wings, and feathers with existing bird species, even without expertise. Similarly, ProtoPNet aims to precisely perceive and recognize discriminative parts of objects with category-specific prototypes. As shown in Fig. 1 (a), the highest activated prototypes capture features of the bird's head and wings. By making predictions through the linear combination of prototypes' similarity with image patches, ProtoPNet is inherently interpretable and can be analyzed by visualization when post-processing.

ProtoPNet and its variants are mainly developed on convolutional neural networks (CNNs). While recent years vision Transformer (ViTs) [Dosovitskiy et al., 2021] have been introduced into computer vision, challenging the domination of CNNs with their superior performance due to the ability to model long-range dependencies [Liu et al., 2021; Chen et al., 2022a]. Naturally, we want to investigate whether prototype-based methods can be utilized for ViT-backed models with their case-based reasoning process. Disappointingly, directly applying ProtoPNet to a ViT (removing the class token) leads to a "distraction" problem: the learned prototypes are prone to obtain high activation scores in the background and show scattered and fragmented activation scores in the foreground, as shown in Fig. 1 (b). The "distraction" problem stems from the homogeneity of different units in ViTs. Fig. 2 shows that the ViT has significantly higher unit similarities than the CNN. The unit similarities are the average similarities between all pairs of units or tokens in the feature map or visual sequence, which indicates that the same prototype may have high similarity with many units in the same token sequence and generate scattered visualization results in the ViT (more quantitative analysis experiments of unit similarities are provided in the the supplementary material). The unsatisfactory visualization results violate the idea that makes prototypes point out critical visual evidence for each case in ProtoPNet. The lack of intrinsic inductive bias makes prototypes of ViTs focus less on prototypical parts and more on long-range dependency.

To solve the aforementioned limitations, we propose prototypical part transformer (ProtoPFormer) to appropriately and effectively apply the prototype-based method with ViTs for interpretable image recognition in two steps, as shown in Fig. 1 (c). ProtoPFormer proposes global and local prototypes to concentrate the "distracted" prototypes on the discriminative parts to build a self-interpretable ViT-backend model. As reported by [Raghu et al., 2021], the class token (i.e., the global branch) of ViTs progressively aggregates information from all the image tokens and produces the highlevel abstraction of targets; on the contrary, image tokens (i.e., the local branch) remain strong similarities to their corresponding spatial locations in inputs. Hence the global and local prototypes are designed to compute similarity scores with output embeddings of the class token and image tokens for capturing and highlighting holistic and partial features of targets and fully capitalizing on the built-in architectural characteristics of ViTs. Next, we gradually perform a two-step concentration process to solve the "distraction" problem with the proposed two types of prototypes. In the first step, the global prototypes perceive the holistic features of targets and devise a foreground preserving (FP) mask to guide the local branch to selectively keep foreground-related image tokens and eliminate the influence of the background. In the second step, A prototypical part concentration (PPC) loss is designed to promote inter-prototype divergence and centralize scattered similarity scores, encouraging local prototypes further focus on diverse prototypical parts as visual explanations. Fig. 2 shows that ProtoPFormer significantly decreases the unit similarities of ProtoPNet on the ViT backbone, which mitigates the homogeneity of different units and makes the prototypes more concentrated. Finally, the predictions from the local and global branches are combined to make final decisions jointly.

Our experiments have proved that combining the two types of high-level abstracted features can mutually correct each other's decisions from their exclusive views. Moreover, extensive experiments have demonstrated that our ProtoPFormer not only enjoys superior performance on accuracy and quantitative interpretability results [Kim et al., 2022b], and faithfully reasons the decision-making processes from both global and local perspectives, appropriately and efficiently resolving the limitations of previous prototype-based methods in ViTs. Besides, more interpretability analysis, ablation studies and visualization results are presented in the supplement. In conclusion, our contributions are summarized as follows:

• Based on the architectural characteristics of ViTs, we propose ProtoPFormer with global and local prototypes to capture and highlight the holistic and partial features of target objects complementarily.
• A two-step process is performed to progressively solve the "prototype distraction" problem and point out visual evidence associatively from global and local perspectives.
• Extensive experiments have demonstrated that ProtoPFormer can achieve superior performance and transparently reason the decision-making processes, benefiting from the strategy of mutual correction and joint decision of global and local prototypes.

2 RelatedWork

2.1 Interpretability with CNNs

The interpretability of the inherent decision-making process of DNNs has become a grand challenge in computer vision. In general, previous works of model interpretability can be divided into two groups: self-interpretable models and post-hoc analysis [Gao and Guan, 2023]. Selfinterpretable models are elaborately designed neural networks that have transparent reasoning processes with regularization techniques [Subramanian et al., 2018; B¨ohle et al., 2022] or accountable components [Zhang et al., 2019; Zarlenga et al., 2023]. The post-hoc analysis methods focus on undermining interpretable information of a welltrained DNN with various techniques like visualization [Bychkov et al., 2018], saliency analysis [Chefer et al., 2021a; Hu et al., 2023] and gradients [Selvaraju et al., 2020; Singla et al., 2019]. ProtoPNet [Chen et al., 2019] combines the characteristics of both schools with faithfully reasoning the decision-making process by the linear combination of the prototype's similarity scores and visualizing the importance of discriminative parts as post-hoc analysis. Many follow-up studies extend ProtoPNet to many areas like medical image processing and explanatory debugging [Keswani et al., 2022; Nauta et al., 2021; Donnelly et al., 2022; Bontempelli et al., 2022; Gautam et al., 2023; Huang et al., 2023]. While these CNN-based methods tend to obtain unsatisfactory results both on classification accuracies and visualization of prototypes when directly applying to ViTs [Dosovitskiy et al., 2021]. Besides, recently [B¨ohle et al., 2022] propose selfinterpretable classifiers using patch-wise similarity, which requires enormous labor to define critical patches and interact with machines while only conducting experiments with CNNs. Therefore, we introduce ProtoPFormer to point out visual evidence for ViTs automatically.

2.2 Interpretability with ViTs

Transformer[Vaswani et al., 2017] is introduced into computer vision filed and has achieved impressive success [Touvron et al., 2021a; Liu et al., 2021; Zhang et al., 2023]. With the wide applications of ViTs, some approaches are proposed to explore their interpretability. The most intuitive way is to analyze the attention weights [Vaswani et al., 2017; Abnar and Zuidema, 2020]. Nonetheless, this simple assumption may not be a fail-safe indicator [Serrano and Smith, 2019]. For ViTs, some reasons the decision-making process via gradients [Yao et al., 2022; Chen et al., 2022b], attributions [Chefer et al., 2021b; Yuan et al., 2021] and redundancy reduction [Pan et al., 2021]. While these methods can only attend to the global features, leaving out discriminative parts. In particular, ViT-Net [Kim et al., 2022a] and ConceptTransformer [Rigotti et al., 2021] include visual prototypes/concepts into ViTs as visual explanations. While ViTNet merely adopts ViTs as backbones to extract features. The interpretability mainly comes from the neural tree that also introduces many parameters and ignores the architectural character of ViTs. ConceptTransformer adds user-defined concepts (e.g., attribute annotations) to enforce an additive relation between token embeddings and concepts. The userdefined concepts require expensive and time-consuming human labeling. Comparatively, ProtoPFormer is designed precisely for ViTs and capitalize on the built-in class token and image tokens in Transformer with category-specific prototypes that can be automatically learned when training.

3 Preliminaries

3.1 ProtoPNets

A typical ProtoPNet is composed of three sequential modules: (1) a backbone network maps an input image to a sequence X ∈ Rn×d , where n is the length of the visual sequence and d is the embedding dimension; (2) a prototype layer Proto(X) transforms X into a similarity score vector s ∈ Rm, where m denotes the number of learnable prototypes; (3) a fully connected (FC) layer FC(s) makes the prediction with s. In the prototype layer, particularly, the i-th similarity score si is the max pooled value from the similarity map Si = Sim(X, p(i) ), where p(i) ∈ P is the i-th prototype and Sim(·, ·) computes the similarity between the given prototype pi and all visual tokens, defined in [Chen et al., 2019]. Generally, ProtoPNets assign k prototypes equally for each class and therefore m = kC (C is the number of classes).

Vision transformers. This paper focuses on ViTs adopting the attention mechanism as the original Transformer [Vaswani et al., 2017]. ViTs firstly embed disjoint image patches as a sequence of image tokens. Then they are appended with a class token and fed into multiple encoder layers composed of a multi-head self-attention (MHSA) module and a multilayer perception (MLP). Given a visual sequence X ∈ Rn×d , according to [Vaswani et al., 2017] the MHSA layer can be rewritten as

where Ah = softmax(Ψh) is the normalized self-attention matrix by row-wise softmax of head h (Ψh ∈ Rn×n ), H is the head number, and Wh ∈ Rd×d is the linear projection matrix. In particular, the global information is gradually aggregated to the class token solely for the final classification by each MHSA layer.

3.2 ViT-backed ProtoPNets

To directly implement a ViT as the backbone, we remove the class token from the sequence and feed remaining tokens to the following prototype layer. However, unlike CNNs, which are stacked with local perception units, e.g., convolutional and pooling layers, the MHSA mechanism aggregates global information to every visual token, which is consequently incompatible with prototypes representing local visual parts. Furthermore, as experiments demonstrates in Section 4.1, the ViT-backed ProtoPNet and its variants show notably performance degeneration compared with CNN-backed ProtoPNets. This phenomena can be explained by visualizing the learned prototypes shown in Fig. 5, revealing a "prototype distraction" problem that the tokens with high similarity scores are distributed irregularly and spread the whole similarity map, including background positions. Hence, not only the classification accuracy but also the interpretability coming with the semantic meaning of prototypes suffers from the incompatibility between the long-range perception of ViTs and the local visual dependency of ProtoPNets.

4 ProtoPFormer

To tackle the abovementioned problem, we propose explicitly constraining the activated similarity map in the local branch so that every prototype is enforced to represent an individual visual part as intended. Moreover, employing the global information aggregated to the class token, the global branch is proposed to model inter-class and intra-class differences contributing complementarily to final predictions along with the local prototype branch.

The overall architecture is illustrated in Fig. 3, in which a class token tc ∈ R1×d and a feature sequence Xf ∈ R(n-1)×d are split from the visual sequence X = [tc,Xf ] and fed into a global prototype branch and a local prototype branch respectively. Similar to the CNN-backed ProtoPNets settings, the global branch has mg learnable prototypes Pg = {p(1) g , . . . , p(mg) g } (mcg for each class), and the local branch has ml learnable prototypes Pl = {p(1) l , . . . , p(ml) l } (mcl for each class). The predicted logits of the two branches are weighted summed for generating the complementary result, formulated as

where zg and zl are outputs from the FC classification layer of global and local branches in respective, and λ{g,l} are weighted coefficients for each branch.

4.1 Concentration on the Foreground

The first step is to concentrate local prototypes on the foreground and eliminate the influence of the background via an adaptive binary mask, named foreground preserving (FP)

mask, to selectively preserve foreground-related image tokens and screen background-related image tokens. The global branch provide guidance to the local branch via the FP mask, capitalizing on the built-in class token and image tokens in ViTs. As shown in Figure Fig. 3, we use the rollout method [Abnar and Zuidema, 2020] to generate the FP mask from the rollout matrix of the class token. For a ViT model, the attention rollout matrix at the l-th layer (l ≥ 1) is defined recursively as ˜ A(l) r = A(l) ˜ A(l-1) r , where A(l) is computed based on the attention matrices A(l) h at this layer:

The initial attention rollout matrix A(0) is predefined as the identity matrix In. As mentioned in [Abnar and Zuidema, 2020], ˜ A1,i is the influence score of the i-th token to the class token, which help us distinguish how likely it could be as a foreground token as visualization in Figure Fig. 3.

Given a backbone ViT with L encoder layer, we extract the rollout matrix at the (L - 1)-th layer for filtering out background tokens. Let ˆac ∈ Rn-1 be the rollout attention values to the class token, we only preserve top-K foreground tokens for computing the L-th encoder layer. More specifically, let γ ∈ {0, 1}n-1 denote a binary foreground preserving (FP) mask (γi = 1 represents that token i is a preserved foreground token). To remove selected background tokens, we modify the softmax normalization applied to Ψ in Eq. (1) as

Eq. (4) cuts off the connection between the background and the foreground, thus avoiding changing information at this step. In the following prototype layer, only the similarity scores generated from the foreground-related tokens are preserved for further class prediction. Generally, this step tentatively concentrates local prototypes in the foreground with the proposed FP mask.

4.2 Concentration on Prototypical Parts

With selected foreground tokens, the local prototypes can be forced to concentrate on heterogeneous visual parts with explicit supervision. For achieving such a purpose, we model the similarity map (reshaped to a two-dimension array like image patches) with regard to each local prototype as a bivariate Gaussian function:

where x ∈ R2 represents the position on the similarity map; µ ∈ R2 and Σ ∈ R2×2 are the parameters controlling the center position and dispersion in respective. On the one hand, by minimizing the eigenvalues of Σ, we are able to dissolve the distraction problem. On the other hand, to promote the divergency of prototypes, we can supervise the prototypes by pushing the centers away from each other.

Gaussian fitting. When achieving N data points D = {(xi, si)}Ni=1 (xi represent the position of the similarity value si on the heatmap S) along with the FP mask γ, we are able to estimate the parameters of the Gaussian model Gaussian(·|µ,Σ). Precisely, by removing background tokens, µ and Σ can be estimated by

Then we propose a prototypical part concentration (PPC) loss to make local prototypes concentrate on different and centralized representative parts for each class. By constraining tr(ˆΣ), the trace of ˆΣ, the PPC loss minimizes the sum of eigenvalues of ˆΣ. In the meantime, this loss also encourages prototypes belonging to the same class to have diverse ˆµ. The PPC loss can be computed as LPPC = λµLµPPC + λσLσPPC, where LµPPC is defined as

and LσPPC can be written as

Here tµ and tσ are two predefined thresholds to guarantee that the PPC loss only penalizes too close center coordinates and encourages small covariance values for concentrating c-class private local prototypes to learn and make decisions on different distinctive visual concepts. λµ and λσ represent their factors. The final optimization objection of ProtoPFormer is to minimize L = LCE + LPPC, LCE is the conventional crossentropy loss.

5 Experiments

5.1 Experimental Settings

Datasets. We conduct experiments on three widely-used datasets including CUB [Welinder et al., 2010], Dogs [Khosla et al., 2011] and Cars [Krause et al., 2013]. All images are resized to 224 × 224 pixels without cropping.

Backbones. Three popular vision Transformers: DeiTTi [Touvron et al., 2021a], DeiT-S [Touvron et al., 2021a], and CaiT-XXS-24 [Touvron et al., 2021b], are adopted as the ViT backbones, initialized with the official pre-trained weights on ImageNet-1k [Russakovsky et al., 2015].

Parameters. All models are trained for 200 epochs with AdamW optimizer [Loshchilov and Hutter, 2019] and cosine LR scheduler. The weighted coefficients λµ, λσ are set to 0.5, 0.1, and the two thresholds tµ, tσ are 2 and 1. We use 10 local prototypes for all datasets, and 10, 5, and 5 global prototypes for CUB, Dogs and Cars datasets, respectively. K is 81 for CUB and Dogs, and 121 for Cars. FC layers are non-trainable in ProtoPFormer.

Baselines. We compare the proposed ProtoPFormer with the classic and state-of-the-art (SOTA) prototype-based approaches. (1) Base represents the vanilla ViT model, serving as the non-interpretable counterpart of our method. (2) ProtoPNet [Chen et al., 2019] is the first work that interprets DNNs' decisions through a linear combination of similarity scores of prototypes. (3) ProtoTree [Nauta et al., 2021] combines prototypes with decision trees for hierarchical reasoning. (4) TesNet [Wang et al., 2021] introduces a transparent embedding space with class-aware basis concepts. (5) Def. ProtoPNet [Donnelly et al., 2022] designs spatially flexible prototypes for handling images with pose variations. (6) CT [Rigotti et al., 2021] stands for ConceptTransformer which utilizes attributes as visual concepts. This method cannot be applied to Dogs and Cars which lack visual attribute labels. (7) ViT-Net [Kim et al., 2022a] integrates ViTs and trainable neural trees based on ProtoTree, which only employs ViTs as feature extractors without fully exploiting their architectural characteristics. For fair comparison, we rerun all baselines with three ViT backbones and use grid search to turn their hyperparameters.

5.2 Performance Comparison

In Section 4.1, we report the top-1 accuracy and parameter numbers of ProtoPFormer and our competitors on involved datasets with three ViT backbones. Generally, it can be observed that our proposed method consistently achieves superior performance compared to baseline methods and economizing parameters. Specifically, Base, as the noninterpretable counterpart, steadily shows upper-middle performance than other self-interpretable baselines, which implies that previous SOTA prototype-based methods are confronted with the trade-off between accuracy and transparency for ViT-backed models. ProtoPNet, TesNet, and Def. ProtoPNet suffer from varying degrees of performance degradation with Transformer backbones on three datasets, indicating that these approaches, designed based on CNN models, are unsuited for learning prototypical parts from token embeddings in Transformers. CT obtains worse results than our method with visual concepts defined by user-defined attributes on CUB, which require time-consuming labeling and rely on human judgment. In contrast, our prototypes are automatically learned along with the training process, faithfully reflecting the discriminative information for the decisions of ViTs. ViT-Net has comparable accuracy with our method but introduces significant extra parameters with the neural trees. Comparatively, ProtoPFormer solely adds small additional parameters, the global prototypes, to achieve the necessary improvement for ViTs. In summary, extensive experiments have verified that our ProtoPFormer outperforms seven baselines on adopted datasets with three ViTs, meanwhile avoiding increasing too many overheads.

5.3 Visualization Analysis

Reasoning Process. Fig. 4 shows a typical reasoning process of ProtoPFormer. The global and local branches make complementary predictions contributing to the final decision. Prototypes of each branch compute corresponding similarity scores with token embeddings and produce the final points with linearly combined scores. In this case, when classifying an Indigo Bunting, global prototypes mistakenly have a slightly larger response with the holistic features of the Boat Tailed Grackle class than Indigo Bunting. Meanwhile, local prototypes successfully discover that the test bird reveals many close parallels with the Indigo-Bunting's exclusive prototypes: the indigo features covered the bird's head, belly, and black feet. Pointing out prototypical visual evidence helps local prototypes make sound judgments and correct the global branch. With the mutual correction and joint decision strategy, global and local prototypes capture holistic and partial features of target objects in a complementary way, benefiting the final decisions of ProtoPFormer.

Visual Comparison. In this subsection, we analyze the interpretability of ProtoPFormer through visualizing local prototypes after training and compare the results with five baselines, illustrated in Fig. 5. The bounding box covers the top 5% similarity scores in the same map, and heat maps are generated by up-sampling and mapping the activation maps to the pixel space, following the same visualization process in ProtoPNet. It can be observed that previous prototype-based competitors obtain unsatisfied visualization results: similarity scores are distributed irregularly and spread throughout the whole similarity map while paying excessive attention on the background. For example, two prototypes of ProtoPNet show high similarity with the background when classifying

Why is this bird classified as a Indigo Bunting?

Total points to the Indigo Bunting: 32.24

Total points to the Boat Tailed Grackle: 24.36

Joint Decision Indigo Bunting

Figure 4: The reasoning process of our ProtoPFormer in classifying the species of a bird with DeiT-Ti, where ⊕ denotes summation of similarity scores.

the Red Winged Blackbird, which dramatically impairs its self-explanatory. Nevertheless, ViT-Net and Def.ProtoPNet train prototypes to capture features of the greensward for identifying two dog breeds, as dogs often appear accompanied by the grass. We ascribe this phenomenon to the vulnerability of previous CNN-based baselines to ViT backbones, which mistakenly learn the misleading information related but not congruent with objects and ignore targeting cues. By contrast, ProtoPFormer preciously captures diverse discriminative prototypical parts with the two-step concentration. Notably, the high activated prototypes of ProtoPFormer show significant activations with a central tendency, e.g., the birds' head and under-surface and the dogs' head and belly.

5.4 Analysis of Interpretability for ProtoPFormer Interpretability Evaluation of Prototypes with HIVE. We

follow HIVE [Kim et al., 2022b], a interpretability evaluation method, to quantitatively evaluate the visual interpretability of the learned local prototypes through two evaluation tasks, i.e., a agreement and distinction task, on four SOTA methods and ProtoPFormer. Specifically, the agreement task is used to examine users' confidence in model predictions based on their explanations provided by prototypes, and the distinction task is used to identify the correct predictions based on explanations provided by five methods. We separately compare three sets of 30 examples and 10 examples for the agreement and distinction task with 186 participants (remaining anonymous) on CUB with DeiT-Ti. As shown in Table 2, our pro-

(b) Distinction task results.

Table 2: Interpretability evaluation (%) of local prototypes.

posed ProtoPFormer obtains the highest correct scores on the agreement task compared to other methods, indicating that the local prototypes in our method can make local prototypes pay attention to different local parts and capture the representative object parts with Transformer-backed models. Moreover, in the distinction task, we present the activation maps of the five prototype-based methods for the participants and require them to choose the correct explanations for the test images. Table 2 also shows that the activation maps provided by ProtoPFormer are considered as the most discriminative explanation among the competitors by participants.

6 Conclusion

In this paper, we propose ProtoPFormer for appropriately and effectively applying the prototype-based method with ViTs for interpretable image recognition. Experiments have demonstrated that our ProtoPFormer can achieve the superior performance and capture the representative visual evidence through learned prototypes, benefiting Transformerbacked models with the self-explanatory characteristic. In future work, we plan to investigate the potential applications of ProtoPFormer in other critical areas like model debugging and medical image diagnosis.

Acknowledgments

This research was supported by "Pioneer" and "Leading Goose" R&D Program of Zhejiang (2024C01167, 2024C01114), Zhejiang Provincial Natural Science Foundation of China (LQ24F020020, LD24F020011), Zhejiang Province High-Level Talents Special Support Program "Leading Talent of Technological Innovation of Ten-Thousands Talents Program" (No. 2022R52046) and National Science and Technology Major Project (2022ZD0119103).

Contribution Statement

Mengqi Xue and Qihan Huang equally contribute to this work. Canghong Jin is the corresponding author.