Self-calibration Enhanced Whole Slide Pathology Image Analysis

1 Introduction

Pathology images are regarded as the "gold standard" for cancer diagnosis and treatment due to their rich microscopic cellular and tissue characteristics. These images exhibit a superlarge size and a wealth of features, necessitating that pathologists frequently zoom in and out to examine both global structural elements and localized details for accurate diagnosis. However, the extensive size and complexity of these images can result in time-consuming evaluations and may lead to misdiagnoses or missed diagnoses due to the inability to scrutinize every detail.

∗Corresponding author. Email: zunleifeng@zju.edu.cn

Some researchers [Xiang et al., 2022; Wang et al., 2018] analyzed pathological thumbnails due to the challenges existing deep learning models face in processing large images. Others [Thandiackal et al., 2022; Shao et al., 2021] employed Multi-Instance Learning with randomly selected patches for whole slide image analysis; however, the former often lacks detailed tissue and cellular features, while the latter fails to capture global structural information.

Consequently, multi-layer pyramid features are employed for pathology image analysis [Chen et al., 2021b; Chen et al., 2022]. However, these pyramid features increase computational time required for processing gigapixel pathology images. Furthermore, extensive mixed features often contain limited task-related features alongside significant irrelevant ones, which can impair final model performance.

In this paper, we propose a self-calibration enhanced whole slide pathology image analysis framework, termed SEW, which integrates global features and several critical local features for fast and accurate pathology image analysis. SEW first classifies images using the global structural features derived from pathology thumbnails. A focus predictor is then employed to identify suspected lesion areas with high probability. Subsequently, these areas are enlarged to extract local detail features, determining whether they correspond to actual lesions. Finally, a feature consistency constraint between the global and local branches is introduced to enhance the global branch's ability to extract more distinctive features.

The integration of global structural features with local detail features from critical areas effectively mitigates the influence of irrelevant features, thereby improving model accuracy and inference speed. Moreover, unlike most existing methods that analyze image patches, SEW utilizes a superpixel technique to identify initial areas with similar cells and tissues. This approach facilitates feature aggregation and reduces irrelevant feature fusion. Additionally, a pathological prototype vocabulary is constructed using clustered area features, which serves to enforce feature consistency across diverse WSI samples.

With these simplified and discriminative features from the focused regions, the k-means algorithm is employed to uncover distinct clusters of favorable or unfavorable prognosis samples. These unique clusters unveil novel prognostic tumor markers, which are subsequently validated by a pathologist.

Moreover, the spatial distribution of distinct tissues further reveals prognostic tumor markers within the two-dimensional spatial realm of the image.

Our contributions are summarized as follows: We present a self-calibration-enhanced framework for whole-slide pathology image analysis, integrating global structural features with pivotal local features to ensure precision and efficiency. The focus predictor, coupled with feature consistency constraints, enhances the global branch's ability to extract more distinctive, accurate features. Additionally, we introduce a pathological prototype vocabulary to reinforce feature consistency across diverse WSI samples. Extensive experiments demonstrate our method achieves state-of-the-art performance in both inference speed and accuracy. More importantly, the learned simplify and critical features can effectively prompt new tumor mark finding.

2 RelatedWork

Whole Slide Image Analysis. Existing Whole Slide Image (WSI) analysis methods can be divided into three main categories: thumbnail-based, Multi-Instance Learning (MIL), and pyramid feature-based approaches. Initially, researchers [Xiang et al., 2022; Wang et al., 2018] utilized pathological thumbnails for WSI analysis to address the challenges of handling extremely large images. However, these methods often result in suboptimal classification due to their inability to capture fine-grained tissue and cellular details.

To capture local detailed features, researchers have employed MIL approaches for WSI analysis [Chikontwe et al., 2020; Zhao et al., 2020a]. Specifically, Tellez et al. [Tellez et al., 2019] aggregated features from segmented patches to represent the entire WSI. Chen et al. [Chen et al., 2022] utilized hierarchical transformers to integrate features across scales. Li et al. [Li et al., 2024] enhanced structural feature expression by dynamically constructing inter-patch edges. Despite these advancements, MIL methods still struggle to capture global structural features.

To integrate global and local features, many researchers adopted pyramid feature for pathology image analysis. For example, Chen et al. [Chen et al., 2022] aggregated visual tokens at cell, patch, and region levels in a bottom-up manner to construct slide representations. Xiang et al. [Xiang et al., 2022] used a Dual-Stream Network to obtain representations of multi-scale thumbnail images. Yu et al. [Yu et al., 2024] used a self-reform multilayer transformer to address the time-consuming and space-consuming problem in pathological image analysis. Chen et al. [Chen et al., 2021b] adopted a tree-based self-supervision to enhance representation learning and suppress contributions of potentially irrelevant patches. However, these pyramid features elevate the computational time required for processing WSIs, while the extensive mixture of features complicates the learning of critical features for the final task.

Acceleration of Pathology Image Analysis. To enhance the training and inference efficiency for pathology images, several researchers have focused on identifying Regions of Interest (ROIs) within WSI for effective analysis. Lu et al. [Lu et al., 2021] employed attention mechanisms to locate ROIs for ultimate classification, using whole-slide labels as supervisory guidance. Shao et al. [Shao et al., 2021] introduced TransMIL, which explores both morphological and spatial information for weakly supervised WSI classification. Tang et al. [Tang et al., 2022] presented the QuadTree method, which deconstructs histopathology images by identifying clinically relevant regions while disregarding less pertinent areas such as empty spaces or connective tissue. Furthermore, ZoomMIL [Thandiackal et al., 2022] enables the model to identify informative patches, thereby greatly enhancing inference speed.

Graph-based WSI Analysis. Chen et al. [Chen et al., 2021a] constructed a graph using features extracted from equally sized patches of WSI and applied Graph Convolutional Networks (GCNs) to learn structural features for survival prediction. Similarly, Lu et al. [Lu et al., 2022] utilized GCNs to predict HER2 status and breast cancer prognosis. Lee et al. [Lee et al., 2022] harnessed a Graph Attention Network to capture contextual features from a heterogeneous tumor environment. Zhao et al. [Zhao et al., 2020b] utilized GCNs to learn bag-level representations for WSI analysis. To bolster the model's global capabilities, Tang et al. [Tang et al., 2024] introduced TransGNN, merging local structure with global long-range cross-attention for the prognosis prediction of hepatocellular carcinoma. In contrast to the methods described above, we employ superpixels as graph nodes, preserving the original boundaries of distinct tissues. Furthermore, a multi-layer GCN framework is devised to capture features across various scales.

Pathological Tumor Marker Mining. Pathological markers offer invaluable insights into tumor diagnosis, prognosis, treatment response, and personalized care. Traditionally, pathologists identify novel tumor markers by adhering to established principles and guidelines, a process that is both labor-intensive and time-consuming, often proving costly [Sharma, 2009; Das et al., 2023]. Ye et al. [Ye et al., 2023] annotated fine-grained tissue categories and trained a U-Net model to segment diverse tissue types, thereby assisting pathologists in identifying tumor markers. Liang et al. [Liang et al., 2023] introduced a human-centric deep learning framework that utilizes CNNs to classify tissue patches, enabling pathologists to compare the differences between samples with good and poor prognoses. Wagner et al. [Wagner et al., 2023] developed a transformer-based pipeline for end-to-end biomarker prediction from pathology slides, leveraging the transformer's attention mechanism to facilitate biomarker mining. Ahn et al. [Ahn et al., 2024] applied MIL for prognosis prediction, subsequently clustering high-probability patches. Pathologists can then uncover tumor markers from these clustered features. However, the candidate features identified by these methods remain overwhelmingly numerous, making it difficult for pathologists to efficiently and swiftly pinpoint tumor markers from such an extensive pool. Furthermore, some deep learning-based approaches depend heavily on large-scale tissue annotations, which compromises their generalization ability.

3 Method

Pathology images are characterized by their large size and rich microscopic detail. To reduce interference from irrelevant features and enhance model inference speed, we propose a self-calibration enhanced framework for whole slide pathology image analysis, denoted as SEW. As illustrated in Fig. 1, SEW comprises three components: a global branch, a focus predictor, and a detailed branch. The global branch classifies using the pathological thumbnail, while the focus predictor identifies relevant regions based on the final layer features of the global branch. The detailed extraction branch then evaluates whether the magnified regions correspond to the lesion area. Lastly, a feature consistency constraint between the global and detailed branches ensures that the global branch focuses on relevant regions and extracts more discriminative features for final identification.

3.1 Global Superpixel Graph Classification

In the pathological diagnosis process, pathologists begin by identifying potential lesion areas at the thumbnail level. Inspired by this practice, we utilize a whole slide image (WSI) downsampled by a factor of M 2 as input in the global branch. Following superpixel segmentation to establish a graph, features are initially aggregated using graph neural networks. The transformer with global self-attention is then employed to extract global lesion structural features. These global features are directly used for WSI classification, effectively accelerating inference speed.

Superpixel Graph Building. Given a thumbnail I×M2 of a pathology image I, the classic superpixel generation technique SLIC [Achanta et al., 2010] is adopted to obtain the superpixel blocks. Based on the superpixel block, we construct a superpixel graph as follows:

where vn represents the n-th node, en,n′ denotes the edge between node vn and its adjacent node vn′ , N is the total number of nodes. For every node vn, the color histogram of each color channel is extracted as its original color feature zn = [zR n , zG n , zB n ]. This operation is useful for feature aggregation and helps to reduce interference from massive similar features. Additionally, spatial information is incorporated into each node vn by concatenating the average position pn of the pixels in the n-th superpixel block with the color feature zn. Therefore, the feature for each node vn is denoted as a composite feature [zn, pn].

Global Graph Classification. We employ a graph convolutional network f global GCN () to perform convolution operations on the superpixel graph Gglobal(V, E), resulting in the updated graph G ′global (V ′ , E ′ ) = f global GCN (Gglobal(V, E)) with aggregated features of adjacent nodes. It reduces feature diversity and the complexity of subsequent identification tasks.

Then a transformer with a cross-attention module is adopted to model the global relations of the aggregated features {v′n }Nn=1 , without additional positional encoding due to the positional information already imposed by the GCN and the inherent translational invariance of self-attention. The token embeddings [h1, h2, ..., hN] for above transformer are obtained as follows:

where W proj global is a learnable mapping matrix, d is the dimension of the vector v′n , and Norm() denotes normalize the N × d matrix into to [0, 1] according to the d-dimension.

Next, the global cross-attention transformer f global Atten is adopted to model the global relationship among token embeddings [h1, h2, ..., hN] along with an extra global class token CLSglobal that reads out embeddings of all tokens as follows:

where [h′1 , ..., h′N , CLS′global ] denotes the updated embeddings.

Then, an MLP classifier f global MLP () is employed to perform initial classification at the thumbnail level, using the CrossEntropy loss function as supervision:

where y′cls and ygt predicted the probability and ground truth of the WSI thumbnail, respectively.

3.2 Focus Area Prediction

Similar to the diagnostic process, pathologists first examine the overall lesion situation before focusing on critical areas for further scrutiny. In our approach, we propose a focus predictor to identify critical lesion areas for detailed feature extraction and identification.

we utilize a focus predictor f focus MLP () that takes the last layer token features [h′1 , h′2 , ..., h′N ] of the global branch as input and predicts candidate areas. Consequently, we obtain the probability q′n of lesion areas vn as follows:

The training of the focus predictor involves minimizing the Kullback-Leibler divergence between the predicted heatmap Q′focus and the ground truth Qgt as follows:

The focus predictor faces a cold start problem during earlystage training. To address this, the heatmap obtained by the Grad-CAM [Selvaraju et al., 2017] from the global branch is adopted as pseudo label. Once the focus predictor and local branch demonstrate basic identification abilities, we use the prediction result of the local branch as a pseudo label. The local branch is trained using a detailed lesion area mask, enabling the extraction of more intricate tissue and cellular features. Using the local branch's output as a pseudo label for the focus predictor enhances focus on critical features.

Top-K Sub-Graph Selection. Based on the predicted focus heatmap Q′focus = {q′n }Nn=1 , Top-K sub-graphs {G k sub (V k , E k )}Kk=1 , with the highest average probability are selected. Each sub-graph corresponds to a group of nodes {vn′ }N ′ n′=1 , as shown in Fig. 1. The average probability is computed based on the predicted probability q′n′ of all nodes in {vn′ }N ′ n′=1 . It should be noted that the selected sub-graphs do not overlap with each other.

3.3 Local Focus Area Calibration

In this section, the local branch is employed to extract features from the selected lesion areas that correspond to the Top-K sub-graphs. For each sub-graph G k sub (V k , E k ), we isolate the corresponding amplified lesion region from the pathology I×M2, which is downsampled by a factor of M 2 from the original WSI I. The extracted lesion region contains more detailed tissue and cellular features.

Local Superpixel Graph Building. Similar to the global branch, the SLIC technique [Achanta et al., 2010] is adopted to generate superpixel blocks for the corresponding area of each sub-graph G k sub (V k , E k ). Additionally, the adjacent edge building technique is also adopted to construct the corresponding local graph G k local . Using the local graph convolutional network f local GCN(), we obtain the feature aggregation graph G ′klocal = f local GCN(G k local ). It should be noted that the local branch does not require the addition of any position embedding.

The local graph G k local corresponds to the global sub-graph G k sub composed of M nodes. Consequently, the nodes in G k local can be partitioned into T groups, {{utj }Jj=1 }Tt=1 , where J denotes the number of nodes in each group. Remarkably, each group of nodes {utj }Jj=1 corresponds to a node in the global graph Gglobal.

Local Superpixel Graph Classification. For each group of nodes {utj }Jj=1 , a mapping layer combined with a normalization operation is adopted to obtain the new embedding :

where W proj local is a learnable mapping matrix, d is the dimension of the vector utj , and Norm() denotes normalize the J ×d matrix into to [0, 1] according to the d-dimension.

Classification. The selected lesion area may contain multiple lesion types, such as different tumor types. Therefore, intra-group cross-attention is facilitated by adding T class tokens {CLStlocal}Tt=1 . Notably, the node representation rtj in the t-th group is only cross-attentive with representations in the same group and the corresponding CLStlocal . Simultaneously, interactions among the groups are calculated with the class tokens. The overall attention is computed as follows:

where GroupAtten() and Atten() denote intra-group and inter-group cross-attention for class tokens, respectively.

Next, a local MLP classifier is adopted to classify the class token CLStlocal , which is given by:

where yt and ytgt denote the predicted and ground truth categories for the t-th group, respectively. When supplied with a lesion area mask, ytgt indicates whether the corresponding area belongs to the lesion area. On the other hand, if the WSI is labeled with areas of different types, ytgt will be a multidimensional one-hot vector, denoting the type of tumor that corresponds to that area.

Global and Local Consistency Constraint. The group of nodes {utj }Jj=1 corresponds to a node in the global graph Gglobal, as explained previously. Because the local graph G k local corresponds to the global sub-graph G k sub , the k-th group corresponds to the n-th node in the global graph G. Hence, we incorporate global and local consistency constraints to ameliorate the node feature extraction capabilities of vn, which is formulated as follows:

where W proj cls represents a learnable matrix, h′n signifies the feature of node vn following a cross-attention operation.

3.4 Pathological Prototype Vocabulary

To ensure that tissues with the same semantics across diverse WSIs exhibit similar features, the pathological prototypes are derived in the following section. These prototypes can then be utilized to enhance classification performance and visualize the spatial distribution of tissue in WSIs.

Prototype Generation. After aggregating the nodes from all local graphs {G k local}Kk=1 , node representations {ui}K×T×J i=1 were clustered into C clusters using the Kmeans clustering algorithm. These cluster center representations, denoted as {Oc}Cc=1 , form the pathological prototype vocabulary for the current WSI. The pathological prototype vocabulary can be utilized to reconstruct the lesion areas, providing a diagnostic reference for practical application.

3.5 Final Prediction andModel Optimization

To improve the WSI's final classification accuracy, we compute the fused multi-granularity feature vector, denoted as Hall. It includes global features, K local sub-graph features, and cluster center features, combined as follows:

where W mapping global , W mapping local and W mapping proto are mapping matrices. With Hall, the final prediction y′final = f all MLP (Hall) is obtained. The final classifier f all MLP is trained with the CrossEntropy loss function Lall = CE(y′final , ygt).

In summary, we optimize the focus predictor with Lfocus , the local branch with Llocal cls , and the global branch with Lglobal cls , Lcst and Lall combined.

3.6 Application on Tumor Marker Mining

Pathological markers provide objective indicators for early tumor diagnosis and intervention, improving patient outcomes while reducing costs. Our SEW framework achieves accurate pathological classification through discriminative feature learning, while enabling tumor biomarker discovery via dual perspectives: feature cluster uniqueness and tissue spatial distribution patterns.

The focus predictor eliminates irrelevant regions, retaining prognosis-critical areas. Comparative analysis of feature differences between favorable/unfavorable prognosis samples reveals disease-specific biomarkers through K-Means clustering as shown in Fig. 3(a-c), where unique clusters correspond to novel tumor markers. The detail extraction branch with pathological prototypes ensures cross-WSI semantic consistency in tissue classification. This enables identification of prognostic spatial distribution patterns, where differential tissue arrangements provide clinically interpretable biomarkers.

4 Experiments

4.1 Experiment Setting

Datasets. To evaluate the performance of our method, we conducted tests across various types of cancer, assessing both the classification accuracy and speed for tasks such as grading and prognosis. The pathological datasets utilized in our experiments include PANDA [Bulten et al., 2022], CAMELYON16 [Litjens et al., 2018], BRCA [Lingle et al., 2016], and LUAD [Albertina et al., 2016]. In addition to this, we have also collected three additional pathological datasets: HCC: 117 HE-stained pathological sections of Hepatocellular Carcinoma Cancer (HCC), labeled with five grades of lesion severity. GC: 123 HE-stained pathological sections of Gastric Cancer (GC), labeled with five grades of lesion severity. CRC: 343 HE-stained pathological sections of Colorectal Cancer (CRC), labeled with two prognostic categories. Each section is annotated by professional pathologists with the lesion area and grade, as well as actual prognostic feedback.

Implementation Details. We employ SLIC superpixel segmentation (n = 1024), a 3-layer GCN (d = 512), layer normalization, and residual connections. The architecture includes a 12-layer transformer encoder (4 attention heads). With batch size=4 and K = 4 focused regions, the local branch achieves effective batch size=16. Training uses SGD [Ruder, 2016] (momentum=0.9, weight decay=5× 10-4 ) with layer-specific learning rates (0.002/0.01). All experiments run on an RTX3090 GPU.

4.2 Comparison with SOTA

Table 1 shows that SEW achieves SOTA performance across multiple cancer pathology datasets, outperforming graph-based methods (TeaGraph [Lee et al., 2022], TransGNN [Tang et al., 2024], Patch-GCN [Chen et al., 2021a]) and MIL-based approaches (NIC [Tellez et al., 2019], CLAM [Lu et al., 2021], HIPT [Chen et al., 2022], QuadTree [Tang et al., 2022], Zoom-MIL [Thandiackal et al., 2022]). While HIPT's hierarchical architecture and TeaGraph's contextual modeling achieve suboptimal accuracy, their inference latency exceeds 300s per slide due to full-slide graph computation. Though QuadTree and ZoomMIL accelerate inference through region selection (5 ∼ 15s per slide), their preprocessing stages still require 100 ∼ 200s. SEW innovatively combines thumbnail-level graph construction with adaptive region zooming, achieving 5.44 ∼ 10.97s inference per WSI (104 faster than HIPT/TeaGraph) while maintaining 96.2% average accuracy.

4.3 Tumor Maker Mining and Visualization

Distinction feature clusters for tumor maker mining. The superior performance of SEW is primarily attributed to the self-calibrated focus predictor, which accurately identifies key regions of WSIs. In the prognosis task for colorectal cancer (CRC) patients, these key regions contain features strongly correlated with prognosis outcomes. We selected 100 CRC patient cases with complete follow-up information, 50 of which had good prognoses, with no recurrence within five years, and 50 with poor prognoses, resulting in death within two years. Using the SEW model, which was well trained on CRC dataset, we analyzed these patients' tissue slices, collected focused tissue-level features extracted from local subgraphs of all samples, and performed clustering on these features. Fig. 3(a) displays the clustering results, where red and green points represent features derived from poor and good prognosis samples, respectively. Distinct clusters (with only red color points) are observed for features from poor prognoses. Verified by the pathologist, these unique clusters correspond to mucinous lakes (marker 1) and necrosis within glands (marker 2), which are novel tumor markers for colorectal cancer.

Tissue spatial distribution for tumor maker mining. Furthermore, pathological prototypes enable reconstructing entire WSIs. Fig. 3(d, e) illustrates reconstruction results for good and poor prognosis samples. A notable difference lies in cancerous tissue distribution (denoted in red) between the two types. The cancerous tissue invades and spreads into surrounding tissues, revealing the degree of tumor infiltration (marker 3), affirming SEW's feasibility and effectiveness in mining tissue distribution markers. More visualization samples are provided in the supplements.

4.4 Generalization Performance validation

To validate SEW's generalization capability, we pre-trained it on HCC and fine-tuned on BRCA, LUAD, CRC, and GC datasets. Benchmarking against Patch-GCN, TransGNN, and Zoom-MIL with minimal epoch fine-tuning, SEW achieves 3.91% higher average accuracy as shown in Table 2. Notably, the performance gap between full training and fine-tuning remains marginal (0.07 - 0.64%), demonstrating strong crossdomain generalizability in pathological feature extraction.

4.5 Ablation Study

Superpixel vs Patch. We use superpixel blocks generated by the superpixel method as graph nodes. Unlike TeaGraph or PatchGNN which employ patches as nodes, we removed SEW's superpixel segmentation module and directly used 16x16 patches to evaluate performance on the HCC dataset. As shown in Table 3, superpixel nodes exhibit better boundary adhesion, achieving significant improvement.

Focusing areas number K and magnification. SEW achieves high diagnostic accuracy with a limited number of focusing areas (K). We evaluated K values of 1, 2, 4, 8, 12, and 16 on the HCC dataset. The results in Fig.2 show that precision plateaus when K ranges between 4 and 8, suggesting sufficient detail extraction at K=8 for reliable diagnosis. Performance declines at K = 16, likely due to redundant information. Additionally, we assessed the magnification levels in local branches (8, 16, 32). The 16 magnification provided optimal performance, effectively complementing the information conveyed to the global branch.

Different components impact. The global branch can sense the information of the entire slice at the thumbnail level and use the local branch's detailed information for selfenhancement. We compared the performance of the global branch alone and the enhanced one. In the forward process, we compared using only the Grad-CAM score with using the focus predictor for focusing. In the backward process, we removed the Lret cst constraints to determine if the local branch's output enhanced the global branch's information extraction. In Table 4, using the local branch and the dualbranch framework's consistency constraint significantly improved the model's performance. The focusing effect of the focus predictor led to a substantial enhancement compared to using Grad-CAM alone. This improvement results from the local branch enhancing the global branch, which depends on the focus location's accuracy.

5 Conclusion

In this study, we introduced SEW, a method that extracts features closely associated with pathological image analysis, achieving promising classification results based on these features. Moreover, SEW identifies biomarkers at both the tissue and tissue distribution levels. On a colorectal cancer dataset, SEW discovered two novel tumor biomarkers, demonstrating the potential of artificial intelligence in exploring new prognostic tumor biomarkers. In the future, we will focus on developing more user-friendly methods and tools for biomarker mining to facilitate clinical application.

Acknowledgments

This work is supported by National Natural Science Foundation of China (62376248), the Huadong Medicine Joint Fund of the Zhejiang Provincial Natural Science Foundation of China (LHDMZ25H160002), the Zhejiang Province Health Major Science and Technology Program of National Health Commission Scientific Research Fund (No. WKJ-ZJ2426) and Information Technology Center, ZheJiang University.