Advancing Stain Transfer for Multi-Biomarkers: A Human Annotation-Free Method Based on Auxiliary Task Supervision

1 Introduction

Cancer remains one of the leading causes of death worldwide, posing a significant threat to human health [Chhikara et al., 2023; Li et al., 2025]. Histopathological examination is the gold standard for cancer diagnosis and treatment.

Hematoxylin and eosin (H&E) staining is widely used in clinical practice to enhance the visualization of tissues and cells. Hematoxylin (Hema) stains nuclei blue or dark purple, while eosin stains the cytoplasm and extracellular matrix pink. Though H&E staining reveals tissue structure and cellular morphology, it cannot differentiate cancerous cells from normal ones due to its lack of specific protein expression. This limitation is addressed by molecular staining techniques like immunohistochemical (IHC) staining, which uses antigen-antibody specificity to visualize protein (positive) expression in tissues and cells. IHC staining typically combines Hema with diaminobenzidine (DAB) chromogen, employing different staining configurations to highlight biomarkers such as ER, PR, Ki67 and HER2, which are crucial for breast cancer analysis. However, IHC examination is more expensive, time-consuming and requires more specialized equipments and techniques, limiting its accessibility and hindering progress in pathological diagnostics. Currently, while no research directly establishes a molecular biological connection between H&E and IHC images, progress in related prediction indirectly suggests a relationship between stains worth exploring via artificial intelligence (AI). For example, [Zeng et al., 2022] achieves semi-supervised H&E-to-PR stain transfer, [Liu et al., 2020] predicts Ki67 positive cells, and [Farahmand et al., 2022] infers HER2 status from H&E images.

∗Corresponding author.

We aim to develop a deep learning-based stain transfer method that models the relationship between H&E and IHC staining, as illustrated in Figure 1. It generates virtual IHC

images that align with H&E images in both pathology and structure, enabling immediate diagnosis of multiple biomarkers from the same tissue. Despite significant progress in stain transfer, existing methods still struggle to overcome several key challenges: (a) Absence of pixel-level aligned ground truth (GT). Staining is irreversible in clinical practice, meaning tissue sections stained with one dye cannot be restored to their pre-staining state for re-staining. Pathologists often use consecutive tissue sections (3 to 5µm apart) for different stains. Due to misalignment and staining-induced variability, pixel-level aligned GT is unavailable (Figure 1a). Several studies employ expert annotations to enhance pathological consistency supervision, such as patch-level [Liu et al., 2021; Boyd et al., 2022; Pati et al., 2024] and cell-level [Liu et al., 2020] annotations. However, manual annotations are time-consuming, labor-intensive and impractical for largescale fine-grained labeling, often resulting in coarse, regionlevel annotations. (b) Difficulty in maintaining pathological and structural consistency. Tissue and biomarker diversity (Figure 1b) complicates the preservation of pathological and structural details during style transfer, including consistency in protein (positive) expression regions, tissue structures and specific expression sites (e.g., clustered nuclei, sparse nuclei, or membrane staining). (c) Limited practical applicability. The lack of pixel-level aligned GT for supervision impedes the interpretability of existing methods, limiting their clinical practical applicability.

In this paper, we propose a novel supervised generative stain transfer method that utilizes automatically generated GT masks. Biomarker expression masks, derived from IHC images of consecutive sections, serve as region-level annotations for corresponding H&E images. This approach, achieved through stain unmixing and morphological operations, provides finer-grained annotations than manual labeling in a cost-effective way and allows parameter adjustments to account for varying region differences between consecutive sections. To fully leverage region-level alignment, we introduce multiple auxiliary tasks that exploit the DAB and Hema channels, offering supervisory information on positive expression location, intensity and nucleus number. This improves both model accuracy and interpretability. Furthermore, to enhance the model's ability to extract pathological and structural information across biomarkers, we design a generator architecture incorporating a pretrained encoder trained on large-scale H&E datasets. By robustly capturing stain-invariant feature of various biomarkers from H&E images, this generator boosts model's generalization ability.

The main contributions of this paper are as follows:

1. This paper presents ATST-Net, a generative stain transfer method that converts H&E to IHC images, designed for high interpretability and generalization. It novelly employs non-pixel-aligned IHC-stained consecutive sections as GT, eliminating the need for any manual annotation or prior information, thus markedly reducing annotation costs and errors.

(2) Multiple specialized auxiliary tasks are proposed to fully supervise the generation process, addressing the lack of precise annotations. These tasks include global and local positive expression location matching, intensity matching, and nucleus number matching in positive regions.

1. We propose a robust stain transfer generator architecture that enhances H&E image interpretation and ensures accurate feature extraction of biomarkers. This is the first work to demonstrate the effectiveness of a specialized large-scale pretrained model for stain transfer.

(4) We conduct experiments with the latest methods on diverse datasets, including the public dataset MIST [Li et al., 2023] with multiple biomarkers, and BCI [Liu et al., 2022], which classifies positive expression intensity. Our method, ATST-Net, achieves SOTA accuracy in terms of pathology, structure and style. It also enables bidirectional transfer.

2 RelatedWork

Traditional methods mainly focus on color mapping. [Reinhard et al., 2001] proposed a statistical transfer technique adjusting each channel's values. [Macenko et al., 2009] introduced a histology stain normalization method using stain vector determination and deconvolution. These methods only achieve partial color transfer and lack pathological consistency constraints, failing to capture the full stain relationship.

Deep learning-based methods adopt a generative approach. They better capture pathological semantic relationships between stained images, leading to more accurate and reliable transfer. Methods that directly use pixel-level supervision typically rely on Pix2pix [Isola et al., 2017] as the backbone. For instance, [Liu et al., 2022] introduced a pyramid Pix2pix method that supervises feature at multiple scales during image generation. However, calculating absolute errors between generated images and pixel-level unaligned GT distorts the image structure, resulting in inaccurate stain transfer. More recent models are based on CycleGAN [Zhu et al., 2017; Shaban et al., 2019] and CUT [Park et al., 2020], which apply fully unpaired or non-pixel-aligned H&E and IHC image pairs. For example, UMDST [Lin et al., 2022] enables the simultaneous generation of multiple stains from a single stain using unpaired training data. [Li et al., 2023] proposed an Adaptive Supervised PatchNCE loss to address misalignment between the source and target domains. PSPStain [Chen et al., 2024] focuses on improving pathological semantic mining and spatial misalignment. These specifically designed methods significantly improve the performance of transfer. However, annotation-free methods suffer from limited interpretability, while manual annotation is costly, incomplete and subject to bias. [He et al., 2024] proposed PST-Diff, a pilot study applying diffusion models to this task, though its ability to preserve tissue structure requires further validation.

3 Method

The architecture of ATST-Net is depicted in Figure 2 and 3. This section starts with data preprocessing, which maximizes the utility of annotation information in consecutive sections. For limited annotations, we introduce multiple auxiliary tasks to guide biomarker generation. A specialized generator is designed to deeply mine information, ensuring the preservation of pathological and structural feature during stain transfer. Finally, the whole training and inference process is detailed.

3.1 Data Preprocessing

Though pixel-level alignment is unavailable between H&E and corresponding IHC images from consecutive sections, high-quality dataset preparation ensures region-level alignment. As shown in Figure 2, data preprocessing extracts the positive region mask M ∈ RH×W×C with height H, width W and channel C for each H&E image RH ∈ RH×W×C from its corresponding real IHC image (real IHC) RI ∈ RH×W×C . ATST-Net then adopts the automated annotated masks to capture biomarker information in H&E images, generating virtual IHC images (fake IHC) FI ∈ RH×W×C .

During preprocessing, the color deconvolution stain unmixing method ColorDeconv [Ruifrok et al., 2001] is first applied to extract the DAB channel from the real IHC RI . The extracted channel is converted to grayscale and smoothed by Gaussian filtering Gσ. Otsu thresholding Otsu then separates the foreground from the background. After the regular erosion and dilation operations, morphological closing Closing fills gaps to produce a continuous automated annotated mask M. The entire process is summarized as:

where the filter's standard deviation σ is empirically set to 1. The disk radius in Closing is adjustable based on alignment quality, with larger radii accommodating greater shift.

3.2 Auxiliary Tasks for Supervision

The absence of pixel-level aligned guidance makes it difficult for the model to learn feature of various biomarkers. To address this, we exploit the approximately aligned positive region masks from preprocessing, which are fully utilized via multiple auxiliary tasks. They guide the model in capturing the location and intensity of positive expression, as shown in Figure 3. Positive nucleus number matching is incorporated to improve the precision of pathological details further.

To globally constrain positive regions, the fake IHC FI undergoes stain unmixing and morphological processing to produce the DAB channel D′ , Hema channel H′ and extracted mask M ′ ∈ RH×W×C , as detailed in Section 3.1. The location loss for global positive expression between M ′ and its GT M is:

& Gaussian Filtering ③ Otsu Thresholding & Erosion & Dilation ④ Closing where x and y represent the vertical and horizontal coordinates of M ′ and M respectively. To further enhance feature learning for both positive and negative regions at a more localized level, M is divided into n equal-sized patches, denoted as Pk, with k as the patch index. The positive region area within each patch is S(Pk). The top n4 patches with the highest S(Pk) form set H, and the bottom n4 patches with the lowest S(Pk) form set L. The local positive expression supervision losses L2 and L3 are then computed as:

in which Si denotes the patch set, with S2 = H and S3 = L. Given the relatively small size of focused regions and similar intensity of positive expression within them, average staining intensity of positive regions in masks M ′ and M is used for intensity supervision. The intensity loss L4 is defined as:

where NM′ and NM are the number of foreground pixels, and D ◦ M represents extracting M's foreground region from D.

Finally, a positive nucleus number matching task is introduced to further refine the model's ability to capture finegrained details. The convergence of the total loss, with this term incorporated, offers stronger evidence of pathological consistency and structural preservation. The widely recognized nucleus segmentation method CPP-Net [Chen et al., 2023] is used for segmentation. Loss L5 is expressed as:

where Seg represents segmentation and NCPP is the nucleus number predicted by CPP-Net. | · | denotes the absolute value.

3.3 Deep PathologyMining Generator

In this study, we improve the stain transfer model's generalization ability by leveraging the feature extraction capability of a specialized large-scale pretrained model. While restaining slides after cleaning them is impractical in clinical workflows, AI enables us to simulate this ideal scenario.

ATST-Net employs an encoder-decoder architecture. The encoder, based on Vision Transformer (ViT) [Dosovitskiy, 2020], focuses on extracting essential stain-invariant feature from the source image, which corresponds to the de-staining phase. The decoder, composed of multiple upsampling layers, reconstructs key features to the original image size while integrating the style characteristics of new staining during the re-staining phase. Skip connections are incorporated between encoder and decoder layers to share multi-scale details.

To be specific, the encoder integrates PathoDuet [Hua et al., 2024], a cutting-edge SOTA pretrained model for H&E images based on ViT. PathoDuet is trained within a self-supervised framework built on the contrastive learning method MoCo v3 [Chen et al., 2021] using pretext tasks. In our model, the input image rh ∈ RH×W×C is initially divided into a series of flattened tokens rhp ∈ RL×(P 2 ·C) , where P is the token size and L = HW P 2 is the total number of tokens. These tokens are linearly projected into dimensions L×D, where D is the latent space dimension, producing rhpat. Next, position embedding rhpos is added, along with a learnable class token rhcls and a placeholder parameter ϵ to align with the PathoDuet structure. The process generates the final input v for the transformer encoder, which consists of K transformer blocks. At the network bottleneck, five ResNet blocks are introduced to fully integrate the extracted feature. Upsampling operations then restore feature maps to the original image size. To further preserve structural details, skip connections transmit low-level information to the decoder layers. Following the convolutional process in CellViT [H¨orst et al., 2024], outputs from selected transformer blocks vk, k ∈ N4 , 2N 4 , 3N 4 ,N are concatenated and fused with the corresponding decoder layers after convolutional operations adjust their dimensions.

3.4 Training and Inference

In ATST-Net, CycleGAN [Zhu et al., 2017] serves as the backbone for style transformation. The deep pathology mining generator, along with multiple auxiliary task constraints, ensures the accuracy of pathological consistency and structural preservation during stain transfer. The proposed deep pathology mining architecture is applied for H&E-to-IHC conversion. For IHC-to-H&E conversion, a simpler U-Net structure [Liu et al., 2021] based on convolutional operations is employed. This choice improves computational efficiency, given the lower color richness of H&E images. The discriminator structure follows the design in [Zhu et al., 2017].

During training, two primary loss functions are applied. The first is the adversarial loss Ladv(G1,DIHC) and Ladv(G2,DHE), where G1 is the generator maps images from the real H&E domain HE to the IHC domain IHC , and G2 maps images from the IHC domain to H&E. DIHC and DHE are discriminators that enforce the generated images to conform to the target domain's distribution and staining properties. To constrain the mapping space and avoid infinite possibilities, a cycle consistency loss Lcycle(G1, G2) ensures that the output fake IHC can be mapped back to the H&E domain. Additionally, structural similarity (SSIM) is incorporated into Lcycle(G1, G2) to enhance brightness, contrast and structural fidelity in generated images. Combined with multiple auxiliary task supervision losses applied solely during the H&E-IHC-H&E transformation, the total loss function is:

where n = 5, and α and β control the weight of each term.

4 Experimental Setup

4.1 Datasets

Experiments are conducted on two high-quality public datasets: the Breast Cancer Immunohistochemical (BCI) challenge dataset [Liu et al., 2022] and the Multi-IHC Stain Translation (MIST) dataset [Li et al., 2023]. BCI contains 3896 H&E-HER2 training pairs and 977 test pairs (all 1024×1024) from 51 whole slide images (WSIs), with HER2 expression categorized into four intensity levels: 0, 1+, 2+ and 3+. MIST provides four biomarkers (ER, PR, Ki67, HER2) with 4153, 4139, 4361 and 4642 training pairs, and 1000 test pairs each (all 1024 × 1024). In our experiments, poorly aligned MIST training samples are removed, yielding 4000 training and 1000 test pairs per biomarker. All BCI data is used without filtering. The BCI test set contains 38, 235, 446 and 258 image pairs for HER2 level 0, 1+, 2+ and 3+.

4.2 EvaluationMetrics

We employ six evaluation metrics to comprehensively assess method performance, categorized into paired and unpaired metrics. The paired metrics [Liu et al., 2021] include: (1) Peak Signal-to-Noise Ratio (PSNR). (2) SSIM: Evaluates similarity in brightness, contrast and structure. (3) ContrastStructure Similarity (CSS): A SSIM variant that focuses on contrast and structure rather than intensity. (4) Perceptual Hash Value (PHV): Assesses perceptual similarity between fake and real images via feature maps. The unpaired metrics [Li et al., 2023] are: (1) Fr´echet Inception Distance (FID) and (2) Kernel Inception Distance (KID): Measure feature-space distribution similarity between fake and real IHC image sets.

4.3 Implementation Details

All methods are implemented in PyTorch 1.12.1 on a single NVIDIA GeForce RTX 3090 GPU. During training and testing, images are cropped to 512 × 512 and stitched after processing. To retain pathological details in H&E and IHC images, no normalization is applied. CPP-Net is trained on a systematic IHC nucleus dataset [Xu et al., 2024]. The proposed deep pathology mining generator uses 224×224 inputs to match the pretrained model. Training adopts a batch size of 1, an initial learning rate of 0.0002 with linear decay after half the epochs, and the Adam optimizer. Hyper-parameters in Ltotal are: α = 5, β1 = 60, β2 = β3 = 10, β4 = 20, β5 = 1. The model is trained end-to-end. Parameter settings for comparison methods follow their original papers.

5 Experiments and Analysis

5.1 Comparisons with the SOTAMethods

To assess stain transfer accuracy, ATST-Net is evaluated on four MIST biomarkers and one BCI biomarker with four levels. We compare it with the baseline style transfer method CycleGAN and SOTA stain transfer methods, i.e., PC-StainGAN [Liu et al., 2021], UMDST [Lin et al., 2022], PyramidP2P [Liu et al., 2022], ASP [Li et al., 2023] and PSPStain [Chen et al., 2024]. Following the protocol in Section 4.1, we adopt the same data volume and splits, with image sizes as reported. For PC-StainGAN, H&E labels are generated using the method in Section 3.1 due to unavailable expert annotations. UMDST is trained separately biomarkers.

Quantitative results for MIST (ER, HER2) and BCI are shown in Table 1, with MIST (PR, Ki67) in the supplementary materials. The quality of generated images (fake IHC) is first assessed using PSNR and SSIM, though these metrics are suboptimal due to structural deviations caused by section misalignment and preparation variations. Thus, they serve as reference metrics. This is further reflected in CSS, which mitigates per-pixel intensity impact but still suffers frommisalignment, resulting in lower scores across all methods.

While CycleGAN is effective for style transfer, it lacks pathological consistency constraints, leading to incorrect stain transfer and poor metric performance. UMDST, with its dynamic style adjustment ability, adaptively represents the relationship between stains but struggles with complex IHC images involving multiple biomarkers, where positive region accuracy decreases. PyramidP2P aligns high-dimensional feature maps of fake and real IHC. It maintains positive region consistency through aggregated feature constraints, but loses the original tissue structure totally. PC-StainGAN improves pathological feature differentiation while preserving structure. However, it relies on precise manual annotations, with any decrease in annotation accuracy greatly affects its performance. Notably, ASP and PSPStain excel in PHV, FID and KID by using task-specific networks to preserve pathological consistency and tissue structure in H&E-to-IHC style transfer. PHV shows higher stain transfer accuracy across different layer levels between fake and corresponding real IHC, while FID and KID further confirm consistency in feature distribution across a larger set. Our ATST-Net outperforms existing methods on most key metrics. On MISTHER2, it improves CSS, PHV(avg), FID and KID by 0.019, 0.027, 11.2 and 4.6, showcasing superior accuracy. Quantitative analysis reveals ATST-Net shows reduced improvements on BCI compared to MIST, mainly due to MIST's higher alignment precision, which offers more accurate guidance for model learning.

We further evaluate methods for differentiating positive expression intensity in IHC (0, 1+, 2+ and 3+) when learning pathological information. Results for level 0 and 3+ in the BCI test data are in Table 1, with level 1+ and 2+ in the supplementary materials. PSPStain, which aligns average protein expression intensity during training, outperforms other methods in most metrics. Our proposed ATST-Net constrains positive expression location while focusing on intensity. Its generator excels at identifying varying expression degrees across different locations, surpassing PSPStain on multiple metrics.

To assess ATST-Net's generalization performance, we design cross-dataset validation. The MIST dataset has clearer tissue structures, greater variation in expression intensity and better alignment between H&E and IHC images compared to BCI. We train on MISTHER2 and test on BCI, which exhibits great difference. Results for this MIST→BCI (training dataset→testing dataset) experiment are presented in the MISTHER2 section of Table 1 (values in parentheses). ATSTNet achieves improvements of 0.015, 0.027, 14.4 and 6.3 in CSS, PHV(avg), FID and KID, respectively, demonstrating stronger generalization. However, the metrics are lower than when training on BCI, highlighting how variations in tissue sample, staining and imaging setup, even for the same tissue and biomarker type, can cause significant image variations. In practice, stain transfer models perform optimally only when test data closely matches training data in tissue and biomarker type, sample batch, staining and imaging condition.

In addition, compared to other methods, ATST-Net stands out by enabling bidirectional transfer, including IHC-to-H&E transfer via its backbone. As shown in Figure 3, a simpler network performs the reverse transfer, leveraging the lower color complexity. Figure 5 shows H&E images recovered from IHC inputs. ATST-Net preserves IHC tissue structures while restoring the H&E stain style, enhancing practical value.

5.2 Qualitative Comparisons

We qualitatively compare all methods on the four MIST biomarkers, as shown in Figure 4. PyramidP2P distorts tissue structure in H&E images, while other methods maintain structural integrity. Notably, ATST-Net shows superior pathological consistency, as highlighted by the red boxes in the first two rows. The fake IHC images more closely match real IHC images in both positive expression location and intensity. Moreover, ATST-Net transfers sparse Ki67 expression more accurately (third row). Patches generated by ATST-Net exhibit minimal color discontinuity when stitched (fourth row), ensuring better prediction consistency and practical value.

6 Ablation Study

We conduct ablation experiments on all ER and HER2 test images in MIST. Firstly, we evaluate the proposed generator and multiple loss functions, then analyze the generator's structure. Evaluation metrics are shown in Table 2 and 3, where "Gen", "U", "Skip" and "Res" represent "Generator", "U-Net-based", "Skip connections" and "ResNet blocks".

When solely employing the U-Net-based generator for H&E-to-IHC transfer, identical to that used for IHC-to-H&E transfer, fake IHC images lack pathological consistency, resulting in poor performance, with evaluation metrics comparable to traditional CycleGAN. In contrast, the proposed deep pathology mining generator, based on a pretrained model, effectively learns pathological features of various biomarkers fromH&E images. However, its potential is not fully realized without guidance from biomarker positive expression reference. As shown in the second row of Table 2, the model's performance on CSS, PHV, FID and KID in MISTER shows marginal improvement over the U-Net-based generator.

Applying global positive expression location loss (L1) improves accuracy in tissue regions with clustered positive expression, but the model struggles in sparse expression areas. Local expression supervision losses (L2&L3) mitigate this by guiding the model to focus on both positive and negative fea-

ture, substantially improving all metrics. While L1, L2 and L3 enable the model to identify positive expression locations, generated intensities tend to be uniform and biased toward dominant intensities in the training data. To address this, intensity supervision loss (L4) ensures precise expression intensity. Additionally, nucleus number matching loss (L5) refines expression location while aligning the pathological distribution of all fake IHC images with real ones from consecutive sections. These improvements are reflected in FID and KID, which increase by 3.5 and 0.4 in MISTER. Integrating the proposed generator with auxiliary task supervision losses yields optimal results across all metrics: CSS, PHV, FID and KID in MISTER reach 0.190, 0.468, 46.3 and 5.0, respectively, and in MISTHER2, these values are 0.156, 0.453, 68.1 and 8.6. The last two rows for each biomarker in Table 2 further confirm our generator's higher accuracy and generalization ability.

We demonstrate the role of each module in the proposed generator. Skip connections transmit multi-scale shallow and deep features to the decoder, helping preserve textural and structural details. Multiple ResNet blocks in the network bottleneck integrate and stabilize deep stain-invariant features. As shown in Table 3, the combination of skip connections and ResNet blocks yields superior performance across all metrics.

7 Conclusion

This study proposes a cost-effective H&E-to-IHC stain transfer method with high accuracy, generalization and interpretability. It integrates automatic annotation, auxiliary tasks and a generator architecture. Comparisons on public datasets with diverse biomarkers and expression levels demonstrate its SOTA accuracy and strong clinical applicability.

Acknowledgments

This work is supported by the Fundamental Research Funds for the Central Universities, the National Natural Science Foundation of China (Grant No. 62475072, 62471182), the Science and Technology Commission of Shanghai Municipality (Grant No. 22S31905800, 22DZ2229004), Shanghai Rising-Star Program (Grant No. 24QA2702100) and the Open Research Fund of Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE, ECNU (Grant No. KLATASDS2406).