GroupMIL: Semantic Group Based Multiple Instance Learning for Whole Slide Image Analysing

1 Introduction

The digitalization of pathology through Whole Slide Images(WSIs) has revolutionized microscopic tissue analysis by applying deep learning technologies[Niazi et al., 2019]. Recently, computer-Aided Diagnosis(CAD) has achieved significant success across diverse clinical applications such as cancer screening[Sakaguchi et al., 2025], tumor grading[Campanella et al., 2019], prognosis analysis[Tian et al., 2024], gene mutant prediction[Ning et al., 2025]. Despite these achievements, the gigapixel resolution of WSIs poses a primary challenge, as it renders conventional end-to-end analysis methods infeasible[Pati et al., 2023].

To overcome this, Multiple Instance Learning (MIL) has emerged as the standard solution to WSI analysis[Wang et al., 2016]. The typical MIL pipeline involves extracting patches from the WSI, generating embeddings via a pretrained feature extractor, and utilizing an aggregation module to aggregate these embeddings into a slide-level representation for tasks such as cancer subtyping and survival prediction[Kanavati et al., 2020]. The evolution of WSI analysis is largely defined by the strategy used to model the relationship between patches. Initial research formalized MIL by learning a Bernoulli distribution over patch labels within a WSI using neural networks[Ilse et al., 2018], as shown in Fig.1(a). Subsequent advancements, such as DSMIL[Li et al., 2020], introduced non-local attention mechanisms to compute the similarity between the most discriminative patch and all other instances. To address overfitting, ACMIL[Zhang et al., 2023b] implemented a strategy of randomly masking high-attention patches to redistribute focus to the remaining instances. These models assign the attention score to each patch in isolation and determine its contribution to the final diagnosis. While computationally efficient, these methods represents a significant oversimplification of clinical pathology. Specifically, it ignores the morphological context and the semantic dependencies inherent in tissue structures. As an improvement, recent research has shifted toward modeling global correlations among patches, as shown in Fig.1(b). Frameworks such as TransMIL[Shao et al., 2021] and its variants[Zhang et al., 2023a] utilize Transformer-based architectures to establish a global patch-interaction model. By leveraging self-attention, these methods allow each patch to interact with all other patch in the WSI, effectively capturing long-range dependencies across the slide. However, global correlation models treat each patch as a distinct entity, thereby overlooking the inherent semantic redundancy of tissue where thousands of patches often represent identical biological components (e.g., a tumor region, a stroma region).

∗Corresponding author

Inspired by [Xu et al., 2022], we argue that the correlation between patches should be modeled through coherent groups derived from intrinsic feature similarity. In a WSI, patches belonging to the same tissue phenotype exhibit high feature redundancy. Therefore, an effective MIL framework should automatically aggregate these similar instances into distinct semantic units. By transitioning from global to group-based correlation, the model may explicitly captures the tissue's structural organization, yielding a more robust and semantically meaningful slide-level representation.

Based on this motivation, we proposed GroupMIL, a MIL framework designed to organize the unstructured "bag" of patches into semantic groups.The main difference between GroupMIL and existing paradigms are summarized in Fig.1(c). The core innovation lies in the introduction of learnable Semantic Group Tokens coupled with a dynamic assignment mechanism. Specifically, We introduce Adaptive Multi-stage Grouping module to transform the raw sequence of patch tokens into a structured hierarchy of Semantic Group Tokens. This process operates across multiple stages, progressively transitioning from fine-grained to coarse-grained semantic representations. Further, rather than relying solely on the final layer's output, GroupMIL employs Hierarchical Semantic Aggregation to preserve multi-scale context. By leveraging an attention-based fusion mechanism, HSA dynamically integrates the representative tokens from each stage, ensuring that the final slide-level embedding captures comprehensive information across multiple levels of granularity.

The main contributions of this work are summarized as follows:

1. Adaptive Multi-stage Grouping: By adaptively clustering patches into semantic group tokens based on feature similarity, our model explicitly captures the latent histological structures of WSIs, transforming thousands of unorganized patches into a structured set of tissue phenotypes.
2. Hierarchical Semantic Aggregation: We propose a feature integration strategy that captures the multi-scale information of pathological data. By combining the Adaptive Multi-stage Grouping with Hierarchical Semantic Aggregation, GroupMIL effectively preserves and weights information across different levels of granularity. This allows the model to leverage both high-resolution details and coarsegrained architecture to form a comprehensive WSI-level rep-

resentation.

(3) Superior Performance across Extensive Clinical Tasks: We demonstrate the effectiveness and generalization of GroupMIL through extensive experiments on multiple datasets. Our model consistently outperforms state-of-the-art MIL frameworks across diverse tasks, including cancer subtyping and survival prediction, confirming its robustness and effectiveness in computational pathology analysing.

2 Method

In this section, we first provide a brief review of MIL and then introduce the overall GroupMIL architecture. Subsequently, we provide a detailed elucidation of its two core components: Adaptive Multi-stage Grouping and Hierarchical Semantic Aggregation.

2.1 Preliminaries

MIL servers as the predominant weakly supervised learning method employed to address the computational challenges posed by the gigapixel size of WSIs. Within the MIL paradigm, a WSI is considered as a bag of instances, X = x1, x2, ..., xK. The bag-level label Y can be expressed as:

where yi is the latent label of instance xi. To obtain a holistic representation of a WSI, two predominant methodologies are commonly employed. The first category assigns pseudo-labels to each instance and subsequently aggregates information from the top-k instances. The second, featureembedding-based approach, extracts features for each instance using a pre-trained model and then integrates them, typically via an attention mechanism. As illustrated in Fig.1, these methods can be categorized into two main approaches. The first operates under the Independent and Identically Distributed (i.i.d.) assumption, utilizing attention mechanisms to aggregate features from all patches in isolation. The second approach leverages global correlation, performing exhaustive feature interactions across all patches to generate WSI representations. In contrast, our proposed GroupMIL introduces a hierarchical strategy: it first partitions patches into semantically coherent groups and subsequently calculates intra-group correlations to refine the local feature representations before final aggregation.

2.2 Overview of GroupMIL

The proposed GroupMIL framework, shown in Fig.2, is designed to perform hierarchical progressive grouping of visual concepts of WSI. Similar to many model paradigms[Dosovitskiy et al., 2020][He et al., 2015], GroupMIL follows a hierarchical multi stage design. At each stage, we first initialize learnable group tokens and generate global information with the WSI patch tokens using self-attention. Subsequently, the learned group tokens are employed to cluster the patch tokens, aggregating similar patch tokens together via a grouping layer. Through this hierarchical design, semantic segments within the WSI are progressively grouped. The detailed architecture is described bellow.

2.3 Adaptive Multi-Stage Grouping

WSIs are scanned at multiple magnification levels, and pathological tissues often exhibit a distinct hierarchical structure. Accordingly, GroupMIL adopts a multi-stage grouping operation to aggregate tissue information from fine to coarse granularity.

Formally, a WSI can be patched into N tokens after preprocessing following [Lu et al., 2020], denoted as {pi}N , pi ⊆ R1×1024 . Assuming the model has L stages, the input contains image tokens {sli }Cl-1 i=1 and group tokens {gl i }Cl i=1 at stage l, where C0 = N. We simplify {sli }Cl-1 i=1 to {sli } and similarily {gl i }Cl i=1 to {gl i }. To enbale the information propagation between {sli } and {gl i }, we first concatenate them together and put them into a Transformer layer:

where [; ] denotes the concatenate operator. Following that, the image tokens and group tokens are fed into a grouping block to obtain the new image tokens

By merging multiple image tokens into a single new token, the GroupingBlock facilitates a semantic hierarchical aggregation of the image tokens. Similar to [Jang et al., 2016][Vaswani et al., 2017], we compute the similarity matrix Al between {ˆ sli } and {ˆ gl i } via a Gumble-Softmax, as shown in Fig. 2. The operation is defined as

where Wq and Wk are the learnable parameters of the Linear layer, and γi is an i.i.d sample drawn from the Gumble(0, 1) distribution. We use the straight through and hard assignment trick in [Xu et al., 2022][van den Oord et al., 2017] to make the end to end training and the final output of GroupingBlock is a weighted sum of image tokens assigned

where Wv and Wo are the learnable parameters of the Linear layer. The final output is a set of grouped image tokens {sli }CL i=1 .

2.4 Hierarchical Semantic Aggregation

Based on the multi-stage grouping described above, we perform hierarchical aggregation by integrating the outputs from all stages. At each stage, multiple output tokens are fused into a single token through an attention mechanism. The tokens obtained from all stages are then averaged to derive the final WSI representation.

Formally, the fused feature f l is the weighted sum of {ˆ sli }:

where

where Wt and Wv are the learnable parameters of the Linear layer. In this way, multi-stage features are obtained and the WSI feature is derived by averaging these features

This slide-level representation is subsequently utilized for downstream clinical tasks, specifically cancer subtyping and survival prediction. For the subtyping task, we employ a fully-connected layer optimized via a cross-entropy loss. For survival analysis, we utilize a specialized output head optimized by a Cox proportional hazards loss [Katzman et al., 2016], which accounts for censored survival data.

2.5 Long Patch Sequences Modeling and Position Embedding

Before GroupMIL can be effectively applied, an inevitable challenge must first be addressed: the computational complexity inherent in Transformer-based architecture. The selfattention computation among patch tokens and the similarity matrix calculation between patch tokens and group tokens suffer from O (n2 ) time and memory complexity. This significantly hampers the model's practical applicability. To deal with the long sequence modeling problems in WSIs, the standard row-by-row softmax function is replaced by the Nystrom Method proposed in [Xiong et al., 2021]. The approximated attention form ˆ S can be defined as:

where ˜Q and ˜K are the m selected landmarks from the original n dimensional sequence of Q and K. A+ is a MoorePenrose pseudoinverse of a submatrix obtained by sampling the original attention matrix, with dimensions m×m. When the number of landmarks m is much smaller than the sequence length n, the computational complexity is approximately reduced from O(n2 ) to O(n). By doing this, GroupMIL becomes capable of processing WSI containing thousands of patches.

Following the architecture established by [Shao et al., 2021], we incorporate Pyramid Positional Encoding to model the spatial relationships between patches. To handle the variable number of patches within a WSI, we apply zero-padding to the slide feature map, enabling the application of 2D convolutional neural networks. Specifically, we utilize three convolutional kernels with varying sizes 3, 5 and 7 to encode positional information at different levels of granularity. This multi-scale approach can encode the positional information with different granularity.

3 Experiments

To demonstrate the superior performance of the proposed GroupMIL model, we conducted extensive experiments on two classic tasks: WSI-based cancer subtype and prognosis prediction. For the cancer subtype task, validation was performed on 3 public challenging datasets: BRACS, NSCLC and TCGA-RCC. For the prognosis prediction task, validation was carried out on 5 pubilc datasets, including TCGA-BRCA, TCGA-KIRC, TCGA-KIRP , TCGA-LUAD and TCGA-STAD. All WSIs were preprocessed according to the protocol established in [Lu et al., 2020], and all subsequent analyses were performed at a magnification of 20×. A detailed description of the experiments will be provided in the following sections.

3.1 Datasets and EvaluationMetrics

Cancer subtype

We perform comparative experiments on three public datasets: BRACS, NSCLC and TCGA-RCC. For the BRACS dataset[Brancati et al., 2021], we perform experiments on the official split for fair comparison with existing methods. For the other two datasets, we employ 5-fold Monte Carlo cross-validaton, which partitions the data into training, validation, and testing sets with a ratio of 3:1:1. For the evaluation metrics, we used accuracy (ACC) and the Aera Under Curve (AUC) to evaluate the classification performance.

Prognosis prediction

We compare the performance of various models for tumor prognosis prediction across five public datasets. To ensure a fair comparison, we employed 5-fold cross-validation for each dataset, splitting the data into training and testing sets at a 4:1 ratio in each fold. The average Concordance Index (CI) from the cross-validation was used as the primary evaluation metric for model performance, and we calculated the standard deviation (std) to account for statistical variability.

3.2 Implementation Details

We present the experimental results of the proposed GroupMIL on several datasets, and compare it against the following methods: (1) conventional pooling methods, including Maxpooling and Mean-pooling; (2) i.i.d hyposis based methods, including ABMIL[Ilse et al., 2018], CLAM[Lu et al., 2020], DSMIL[Li et al., 2020] and AttriMIL[Cai et al., 2025]; (3) correlated MIL methods, including TransMIL[Shao et al., 2021] and MambaMIL[Yang et al., 2024]. We following the same data pre-processing as in the CLAM and set a learning rate of 5e-5 for all these model. The features of each patch are embedded in a 1024-dimensional vector by a ResNet50[He et al., 2015] model pretrained on ImageNet[Deng et al., 2009]. We incorporate a projection layer which reduces the dimensionality of the input embeddings from 1024 to 512. During training, we employed an early stopping strategy to prevent overfitting and halt the training procedure when the validation loss ceased to decrease for a consecutive period of 20 epochs. All experiments were done on a server with 8 NVIDIA A6000 GPUs, utilizing the PyTorch framework.

3.3 Comparison Results

Cancer subtype

We will present the results of both binary and multi-class classification. The NSCLC involves a binary classification task distinguishing between LUSC and LUAD subtypes. RCC constitutes a three-class classification task differentiating among KICH, KIRC, and KIRP. BRACS encompasses a more complex classification task covering seven distinct breast lesion subtypes. All the experiment results are provided in Table 1.

As demonstrated in Table 1, our proposed GroupMIL model achieves performance improvements across different tasks compared to the previous state-of-the-art. It attains an AUC of 0.825 on the BRACS dataset, 0.956 on the NSCLC dataset, and 0.983 on the RCC dataset. These results demonstrate the superiority of the proposed model.

Prognosis prediction

In addition to cancer subtyping, we also conducted experiments on tumor prognosis prediction. We compared the proposed GroupMIL with eight other MIL methods across five public datasets, with the results presented in Table 2.

The experimental results in Table 2 demonstrate that our proposed GroupMIL model achieved the best performance across all datasets, confirming its effectiveness. Notably, on the five datasets, the GroupMIL model improved the CI by an average of 2% compared to the second-best performance model.

3.4 Ablation Study

To further analyze the impact of the model architecture on the experimental results, we conducted ablation studies to investigate the effects of the number of group centers and hierarchical aggregation on model performance. Specifically, ablation experiments were performed on the NSCLC dataset, with model performance evaluated using the AUC metric.

Effectiveness of Adaptive Multi-stage Grouping

The primary motivation for Adaptive multi-stage Grouping is to transition from fine-grained histological information to coarse-grained semantic regions. We first established a baseline model by removing the grouping mechanism, which effectively reduces the framework to a standard ABMIL[Ilse et al., 2018] model.We then evaluated single-stage grouping by setting the number of group tokens to K1 = 16 and K1 = 32, respectively. We compared the AUC under different model, with the results summarized in Table 3.

The results demonstrate that while a single stage of grouping improves over the i.i.d. baseline by introducing semantic structure, the Multi-stage approach provides a better performance. This suggests that the hierarchical structure leading to a more robust slide-level representation.

Impact of Semantic Group token Number

We further investigated the impact of group token cardinality at each stage on model performance. Specifically, we varied the number of tokens across the multi-stage architecture to determine the optimal balance between feature effectiveness and semantic representation. The experiment results are detailed in Table 4.

We observed that setting K too large may introduces redundant semantic features, which lead to overfitting on datasets. Conversely, setting K too small may results in a over simplification, where distinct tissue types (e.g., tumor and stroma) are forced into the same group. The combination of 32 fine-grained and 16 coarse-grained tokens provides the optimal granularity for capturing pathological diversity.

3.5 Interpretability and Visualization

By assigning each patch embedding to its corresponding semantic group token, GroupMIL inherently partitions the WSI into distinct clusters. This grouping mechanism allows us for a direct visualization of the semantic distribution across the slide. To interpret these results, we projected the discrete group assignments back onto the original WSI coordinates.

As illustrated in Fig. 3, the model demonstrates a high degree of morphological consistency. Patches with similar histological characteristics are reliably clustered into same group tokens. This alignment suggests that the learned semantic group tokens effectively captures the underlying biological structures of the tissue, enabling it to distinguish between critical phenotypes, such as tumor regions and surrounding stroma.

4 Conclusion

This study proposed GroupMIL, a novel MIL framework that differs from the traditional i.i.d. based MIL or globally correlation MIL to aggregate the WSI information. By leveraging the Adaptive Multi-stage Grouping module, GroupMIL effectively captures the latent histological organization of WSIs, transforming thousands of patches into representative tissue phenotypes. Furthermore, the Hierarchical Semantic Aggregation module ensures that both fine-grained morphological details and coarse-grained structural contexts are preserved and integrated into the final slide-level representation. Extensive experiments across multiple public datasets demonstrate that GroupMIL consistently achieve satisfactory performance in cancer subtyping and survival prediction. The grouping mechanism also provides a clinical interpretability, enabling the automated discovery of diagnostically significant regions without the need for dense pixel-level annotations. Future work will explore the integration of Large Language Models (LLMs) to provide high-level semantic guidance, utilizing medical knowledge to further refine the patch-based groups.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (62501232), the Science and Technology Innovation Program of Hunan Province (2025RC3098), the National Natural Science Foundation of China (62425305), the Research Foundation of Education Bureau of Hunan Province (24B0037) and the Fundamental Research Funds for the Central Universities (531118010979).