TSESNet: Temporal-Spatial Enhanced Breast Tumor Segmentation in DCE-MRI Using Feature Perception and Separability

1 Introduction

Breast cancer is the leading cause of cancer death in women worldwide. Early detection of malignancy is crucial for improving the prognosis of breast cancer patients [Zhao et al., 2023b]. Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is a non-invasive imaging technique that can reveals both temporal and spatial characteristics of the physiological tissue. It plays an important role in the diagnosis and staging of breast tumors [Zhou et al., 2022]. The standard DCE-MRI protocol involves acquiring a precontrast 3D image prior to the injection of an intravenous contrast agent, followed by sequential acquisition of postcontrast 3D images. A representative illustration of DCE-MRI sequences can be seen in the first row in Fig. 1. The response to contrast enhancement varies between tumor and non-tumor tissues. Tumor tissues typically exhibit significantly higher enhancement and increased sensitivity in images compared to non-tumor tissues.

As shown in Fig. 1, the second and third rows illustrate two specific cases consisting precontrast MRI slice V0 and postcontrast MRI slice ¶Vi↪ i 2 1 · · · T♦, along with the subtraction image between Vi -V0. From the figure, we can find that the subtraction images can provide better tumor visualization (highlighted in the yellow box). However, temporal changes manifest differently at distinct time points. In Case 1 (second row), V5 -V0 is more appropriate for tumor segmentation, whereas the tumor area in V1 - V0 is incomplete. On the other hand, in Case 2 (third row), we can observe that the enhanced tumors and glands are adhesion in V8 - V0, and V3 - V0 is more suitable choice for tumor segmentation. Consequently, physicians are still unable to accurately differentiate tumors from other enhanced tissues, such as vessels and glands, based solely on intensity changes at a single time point in DCE-MRI.

Despite the extensive focus on general tumor segmentation, the research on breast tumor segmentation in DCE-MR images is relatively limited [Qi et al., 2023; Milletari et al., 2016]. Common methods usually perform the segmentation in a single 3D MRI image based on maximum intensity projection (MIP), which may ignore the temporal characteristics of tumors within the image sequence [Zhang et al., 2019;

Vidal et al., 2022]. Recent methods have begun to explore the segmentation of breast tumors in 3D sequences [Zhang et al., 2023; Lv and Pan, 2021; Lv et al., 2022; Zhao et al., 2023a]. However, the high dimension of the data poses challenges in modeling sequence features, and the tumor segmentation model consumes too much computing resources [Yeung et al., 2022]. Consequently, achieving efficient and effective modeling of tumor features within the DCE-MRI sequence is crucial for accurate segmentation.

To this end, we propose a novel end-to-end TemporalSpatial Enhanced Network (TSESNet) for DCE-MRI breast tumor segmentation. This approach enhances tumor feature representation by jointly considering the spatial and temporal contextual dependency of inter-sequence, enabling accurate segmentation of tumors of 4D DCE-MRI images. Specifically, the TSESNet utilizes a shared-weight encoder architecture and a Temporal Series Feature Fusion (TSFF) module to address the issue of excessive parameterization. The TSFF module efficiently integrates MRI feature maps from different time points while concurrently reducing the parameters. Additionally, we propose a novel temporal-spatial contrastive loss that constrains the feature representations of different classes in both spatial and temporal domains. The aim is to maximize the discrimination between tumor and background samples. Additionally, we also designed a TumorAware (TA) module based on multi-scale-attention technical to improve the perception of tumors with different scales.

In summary, the main contributions of the proposed method are as follows:

• We proposed a novel TSESNet for breast tumor segmentation in DCE-MRI by exploiting spatial and temporal contextual dependencies. Extensive experiments on two breast DCE-MRI datasets demonstrate the effectiveness of TSESNet, achieving the state-of-the-art performance.
• We propose a Temporal-Spatial Contrastive Loss
• (TSCLoss), which captures spatial complexity and effectively models the temporal dynamics in DCE-MRI sequences. By addressing the distinctive challenges of capturing temporal-spatial features, TSCLoss elevates feature representation and significantly strengthens the performance in breast tumor segmentation tasks.
• We proposed a novel Temporal Series Feature Fusion (TSFF) module that effectively integrates temporal MRI features from multiple time points. This module enhances the model's ability to handle temporal sequences and improves overall performance.
• We designed a simple and effective Tumor-Aware (TA) module that enriches the feature representation to handle tumors of various sizes. Additionally, the attention mechanism and the residual-like structure in TA assist in minimizing the number of learnable parameters.

2 RelatedWorks

2.1 Breast Tumor Segmentation

Breast tumor segmentation plays a vital role in early diagnosis of breast cancer. With the advent of deep learning, several CNN-based architectures, such as the popular UNet, are widely used in breast tumor segmentation [Benjelloun et al., 2018; Piantadosi et al., 2018; Qin et al., 2022; Haq et al., 2022]. However, the above methods focus on 2D slices obtained from 3DMRI data, leading to the spatial contextual information missing. To handle this issue, 3D convolutional kernels are performed on 3D MRI volume data.

Vidal et al [Vidal et al., 2022] proposed a 3D U-Net method for breast tumor segmentation in DCE-MRI. Wang et al [Wang et al., 2021] proposed a 3D segmentation method using the comparison of precontrast and postcontrast MRI. Zhou et al [Zhou et al., 2022] proposed a new multi-branch integrated network based on 3D affinity learning for accurate breast tumor segmentation in DCE-MRI. These methods usually rely on Maximum Intensity Projection (MIP) to convert the 4D DCE-MRI sequence into a single MRI. Although this simplification aids in computational efficiency, it neglects the crucial inter-sequence relationships within DCE-MRI.

Recently, some studies have begun to segment breast tumors throughout the DCE-MRI sequence [Zhang et al., 2023; Zhao et al., 2024]. Although these methods enforce global contextual dependencies, they often overlook the spatial and temporal correlations between sequences within DCE-MRI images. Additionally, the high dimensionality of DCE-MRI data aggravates the calculation, and the diversity in tumor shapes and sizes further complicates breast tumor segmentation, making it a challenging task.

2.2 Contrastive Learning

Supervised contrastive learning has emerged as a highly effective technique for representation learning. The key idea is generating positive and negative sample pairs, and then learning discriminative feature representations by minimizing the embedding distance among positive pairs and maximizing that among negative pairs [Zhang et al., 2022;

Lee et al., 2023; You et al., 2022]. By incorporating pixellevel supervision, these models can effectively capture discriminative features and thus improve segmentation accuracy.

Instance discrimination serves as an approach to supervised contrastive learning for medical image segmentation. In this approach, each pixel is treated as an instance, and the task is to learn the discrimination between pixels belonging to the same class and pixels from different classes. [Wang et al., 2023] proposed a pixel-level contrastive branch to pretrain both the encoder and decoder. This offers the advantage of improving the reconstruction of high-resolution features at different scales. [Zeng et al., 2021] propose a novel positional contrastive learning (PCL) framework, which generates contrastive data pairs by leveraging the position information within volumetric medical images. However, previous methods have primarily focused on either single-image segmentation or 3D volume segmentation, overlooking the spatial correlations between slices and the temporal correlations between sequences in DCE-MRI images.

3 Method

3.1 Overview

In Fig. 2 (a), the proposed TSESNet consists of four main components: Weight-share encoder, Temporal-Spatial Contrastive Loss, Temporal Series Feature Fusion, and Decoder. Specifically, for a DCE-MRI represented as V = ¶V0↪ V1↪ ▷ ▷ ▷ ↪ VT ♦, where T represents the number of contrastenhanced samples, we apply a weight-shared encoder to each volume Vi. This encoder extracts diverse hierarchical feature maps F = F l at different downsampling levels l. Then, these temporally sampled feature maps at the same level are merged into temporal-spatial features ¶fi 2 F l ↪ i 2 0 ▷ ▷ ▷ T♦. The entire encoder leverages the Tumor-Aware (TA) module introduced in Sec. 3.2 to enhance the capture capability of tumor features across various shapes and sizes while reducing computational complexity. Simultaneously, the encoder is guided by the Temporal-Spatial Contrastive Loss, ensuring that the represented spatiotemporal feature sequence exhibits higher separability.

Subsequently, by employing temporal attention mechanisms, the model effectively leverages inter-sequence correspondences while preserving the incremental nature of temporal context. As a result, the temporal feature sequence is fused into a spatiotemporal feature map. Finally, this fused feature map is propagated to the decoder to generate predictions for DCE-MRI tumor segmentation.

3.2 Tumor-Aware Module

Multi-scale information can provide rich semantic features for the segmentation of breast tumor with various sizes. Thus, we proposed a TA module that use attention strategy to fuse the channel dependence between different feature maps of varying resolutions and scales, as depicted in Fig. 3.

Specifically, we used three atrous convolutions with different kernel sizes (i.e., 5 ⇥ 5 ⇥ 5↪ 3 ⇥ 3 ⇥ 3, and 1 ⇥ 1 ⇥ 1)

to extract multi-scale feature maps to increase the attention for tumors. Meanwhile, residual mechanisms are used to improve the communication between features, thus promoting the fusion. The detailed calculation is as follows:

where f 2 fi is a temporal-spatial feature map, C[·] is the concatenation operation.

The scale attention is used to weigh the feature channels at different scales, thus making the network focused on the effective features [Chen et al., 2017]. The scale-attention mechanism is calculated as follows. Firstly, the global average pooling is used to embed the global spatial information into the scale vector:

where D↪ H↪ W represent the depth, height and width, respectively. f 0 is the multi-scale feature map.

The scale-attention weight vector is calculated based on the average pooling vector as follows:

where Wf1 and Wf2 are the weights of two linear layers.

The final output feature map is a weighted sum of feature maps of different scales and their attention weights:

3.3 Temporal-Spatial Contrastive Loss

Breast tumor segmentation in DCE-MRI, it is important to consider not only the spatial relationship between individual MRIs, but also the temporal features across sequences. Here, we propose a novel Temporal-Spatial Contrastive Loss (TSCLoss) to enhance feature separability within the encoder. The goal of TSCLoss is to enforce closer feature distances for pixels of the same class in both spatial and temporal domains, while enlarging the feature distances of different classes. This helps enhance the discriminative capabilities of the encoder's feature maps, and thus increase the performance of breast tumor segmentation. The TSCLoss is shown in Fig. 2 (b).

Spatial Domain: In the spatial domain, we utilize the feature maps fi at a given time point to encourage aggregation of features from the same class and dispersion of features from different classes. Consider the feature vectors zp 2 Rd for each pixel p in fi, where d is the dimensionality of the feature vectors. Pixel pairs of zp and zp0 belonging to the same class are regarded as positive pairs (zp↪ zp0) 2 ⌦+. Pixel pairs of zp and zq belong to different classes and are treated as negative pairs (zp↪ zq) 2 ⌦-. We use the cosine simi-

larity sim(zi↪ zj) = zi · zj ♣♣zi♣♣♣♣zj♣♣ to measure the similarity between feature vectors zi and zj within a single feature map fi [Caliskan, 2023].

Temporal Domain: Across the temporal sequence f0 to fT , we treat pixels at the same spatial location as positive pairs, denoted as (zp↪ ztp ) 2 ⌦+. Here, zp and ztp represent feature vectors for pixels at the same spatial location but different time points. Likewise, the cosine similarity sim(zp↪ ztp ) measures the similarity between feature vectors zp, ztp across different time points fi and fj.

Temporal-Spatial Contrastive Loss: The Temporal-Spatial Contrastive Loss (TSCLoss) aims to minimize the feature distance between positive pairs (zp↪ z0p ) 2 ⌦+ and maximize the feature distance between negative pairs (zp↪ zq) 2 ⌦- in the spatial domain, as well as enforcing proximity between positive pairs (zp↪ ztp ) 2 ⌦+ across in the temporal domain. This is achieved by maximizing similarity between similar features and minimizing similarity between dissimilar features:

where ⌦- is the set of indices of negative samples. ⌧ is the temperature used to smooth or sharpen the distribution.

To obtain the global loss of the entire training set, we averaged the single loss value of all pixel areas and applied the loss to different sampling layers (deep supervised):

e where ⌦+ is the set of indices of positive samples, ♣⌦+♣ stands for the size, ♣Z♣ represents the sample count of pixel, l is the encoder layer number.

By minimizing the temporal-spatial contrastive loss function, we effectively constrain the spatial and temporal features to enhance separability, improving the representation of the encoded feature maps.

3.4 Temporal Series Feature Fusion

In order to effectively capture the temporal contextual dependency of MRI sequences and reduce model complexity. We propose the introduction of a Temporal-Spatial Feature Fusion (TSFF) module, which facilitates interactions among the extracted features and calculates their interdependencies, as illustrated in Fig. 2 (c). In clinical practice, radiologists typically confirm the presence of tumors by observing the dynamic changes between post-contrast MRIs and pre-contrast MRI scans. Leveraging this prior knowledge, we model the relationship between the features of post-contrast MRIs (fi) and pre-contrast MRI (f0) for feature fusion.

This module is designed to learn the temporal context and weights to fuse the features effectively. Specially, calculate attention weights based on the temporal dependencies between the sequential feature maps f1↪ · · · ↪ fT with f0 to emphasize the importance of each time step in the sequence. The relationship weight matrix Wi is calculated as follows:

where Wk↪ Wq and Wv are corresponding learned weight matrices, d is feature dimension. fi 2 ¶f1 · · · fT ♦

The final fused feature map ˆ f is obtained by weighting each fi with its corresponding attention weight matrix Wi and summing them up:

where · denotes element-wise multiplication.

3.5 Loss Function

The model is trained end-to-end with dice loss and TSCLoss. The loss function is formulated as:

where LDice is dice loss [Ma et al., 2021], LTSC is TSCLoss, and l is the number of encoder layers. In our experiment, we set ↵ = 0▷7↪ β = 0▷5, and ⌧ = 10-5 .

4 Experiment

4.1 Experimental Setup

Datasets: To assess the effectiveness of our proposed method for breast tumor segmentation, we conduct experiments on two datasets. The first dataset, known as AI-assistantfor-breast-tumor-segmentation (BTS) dataset [Zhang et al., 2023], is a publicly available collection comprising 100 DCEMRI cases from seven different institutions. The second one, named ST177, is a private collection consisting of 177 labeled DCE-MRI cases. In ST177, all patients underwent preoperative DCE-MRI using a 3.0 Tesla scanner (Signa, GE Healthcare, Milwaukee, WI) with a dedicated 8-channel breast coil. All included studies contained a fat-saturated gradient echo T1-weighted pre-contrast sequence and typically five postcontrast T1-weighted sequences acquired after the administration of the contrast agent. Each sequence contained 88 to 108 2D slices acquired with the following parameters: repetition time (TR) ranged [3.9, 4.8] ms, echo time (TE) ranged [1.7, 1.8] ms, flip angle = 5◦ , field of view (FOV) = 340⇥340 mm, matrix size = 320⇥320, and slice thickness = 1.4 or 1.6 mm. Both datasets comprise 3D DCE-MRI volumes obtained from multiple contrasts (V0 - V5). These contrasts were acquired before the intravenous injection (V0) and during the post-injection phase (V1 - V5) using a positive paramagnetic contrast agent at a dosage of 0.1 mmol/kg.

Preprocessing: Before training, we conducted denoising by removing pixels with values falling within the first and last 0.1% of the range. Following denoising, we performed grayscale normalization.

Implementation details: To ensure a fair comparison, all models were implemented using PyTorch and trained on four NVIDIA RTX 3090Ti GPUs. The ADAM optimizer was employed for optimization, with an initial learning rate of 1e-4. Each GPU had a batch size of 1. The training process lasted for 300 epochs, with an early stop criterion set at 30 steps. Following [Zhang et al., 2023], sub-volumes of size 96⇥ 96⇥ 48 were randomly extracted from DCE-MRI scans as the input for the training stage. In the testing stage, we utilized a sliding window-based strategy, with the same window size as in the training stage.

Evaluation metrics: To evaluate the performance of our method, three commonly used metrics are utilized, i.e., Dice Similarity Coefficient (DSC: %), 95% Hausdorff Distance (95% HD: mm), and Sensitivity (Sen: %).

4.2 Comparisons With the State-of-the-Art Methods

We compare our TSESNet with seven existing DCE-MRI breast tumor segmentation methods. These methods include: (1) 2D Slice-based: MA-Net [Peng et al., 2022] and AttU-Node [Ru et al., 2023], (2) 3D U-Net based: [Vidal et al., 2022], (3) Pre- and post-contrast enhancement MRIs as input-based: ALMN [Zhou et al., 2022] and Tumorsen [Wang et al., 2021], (4) Whole DCE-MRI sequence as input-based: ST-Tumor-Seg [Zhang et al., 2023] and SwinHR [Zhao et al., 2024].

Quantitative Comparison

ST177 Dataset: As shown in Tab. 1, the TSESNet achieves the highest DSC of 88.37% on the ST177 dataset, outperforming all comparison methods by a large margin. Additionally, TSESNet demonstrates superior performance in HD (7.154) and Sen. (83.19%), surpassing the existing state-ofthe-art methods. Notably, methods designed for temporal analysis, such as SwinHR and ST-Tumor-Seg, leverage temporal information and achieve competitive results. However, the TSESNet utilizing temporal-spatial contrastive learning, exhibits a remarkable advancement, notably reducing the HD and improving Sensitivity significantly.

BTS Dataset: Similar trends can be observed on the BTS dataset, wherein TSESNet achieves the highest DSC of 78.95%. Further more, our proposed method demonstrates outstanding performance in terms of Hausdorff Distance (HD) with a value of 6.14 and Sensitivity (Sen) at 79.13%, surpassing the comparative approaches.

In summary, the experimental results on both ST177 and BTS datasets validate the efficacy of our proposed TSESNet for breast tumor segmentation in DCE-MRI. The incorporation of temporal-spatial contrastive learning contributes significantly to improved segmentation accuracy and delineation of tumor boundaries when compared to state-of-the-art methods. The consistent outperformance evaluation metrics underline the robustness and effectiveness of TSESNet in handling the challenges posed by breast tumor heterogeneity and complex DCE-MRI data.

Qualitative Comparison

In Fig. 4, our proposed TSESNet demonstrates highly consistent segmentation results that closely align with the ground truth. In contrast, 2D-based models, such as Att-U-Node and LMA-Net, exhibit intermittent slice losses, resulting partial segmentation discrepancies between slices. On the other hand, methods relying solely on single 3D MRI inputs, such as Tumor-sen, ALMN, and 3D U-Net, encounter difficulties in capturing inter-sequence relationships, leading to segmentation errors. In contrast, our TSESNet, leveraging the temporal series feature fusion module and temporal-spatial contrastive loss, achieves more accurate segmentation than other 4D segmentation models.

Parameters and Complexity

The analysis of model size and complexity (Tab. ??) highlights the efficiency trade-offs among different 4D DCEMRI segmentation methods. SwinHR exhibits a larger model size and high computational demand (146.20M parameters, 314.72G FLOPs). ST-Tumor-Seg, with a larger model size than SwinHR, showcases reduced computational complexity (175.54M parameters, 134.37G FLOPs). Our TSESNet achieves a smaller model size (103.58M parameters) and reduced computational complexity (98.66G FLOPs). This efficiency is attributed to the encoder layer with shared weights and the temporal feature fusion module.

4.3 Ablation Study

To investigate the effectiveness of different components in the proposed framework, we conduct a series of ablation studies on BTS dataset. Quantitative and qualitative comparisons are shown in Tab. 3 and Fig. 5.

Effectiveness of the TA Module: In Tab. 3, in the second line, the inclusion of the Tumor-Aware module leads to an increase in segmentation accuracy (DSC) from 63.41% to 70.14% by fusing features from different receptive fields. This module also refines boundary delineation (95% HD) from 21.179 to 13.192 and increase sensitivity from 66.36% to 71.47% by enhancing the perception of small tumors and complex boundaries. These improvements can be attributed to the strong learning ability of the Tumor-Aware module in capturing tumor features. Moreover, by effectively fusing information from multi-receptive fields, this module enhances its capability to discern subtle features of small tumors and intricate boundaries, resulting in a significant improvement in overall segmentation performance.

Effectiveness of the TSCLoss: In Tab. 3, in the third row, the utilization of Temporal-Spatial Contrastive Loss leads to an increase in DSC to 71.51% by enforcing constraints at the feature representation level, thereby enhancing the discriminative features between the tumor and background. Furthermore, while maintaining a low 95% HD (13.914), TSCLoss slightly improves sensitivity to 72.18%, thus enhancing the separability between tumor and non-tumor features.

Combined Impact: In Tab. 3, in the fourth line, the combination of both modules results in a significant performance boost, achieving a DSC of 78.95%, 95% HD of 6.14, and Sensitivity of 79.13%. The Tumor-Aware module enhances feature perception by utilizing varied receptive fields. Simultaneously, the Temporal-Spatial Contrastive Loss improves discriminative feature representation. The combination of these two modules contributes collectively to the improved accuracy and boundary delineation in breast tumor segmentation fromDCE-MRI images.

4.4 Performance Analysis for TSFFModule

To assess the impact of Temporal Series Feature Fusion (TSFF) on the performance of the TSESNet, we experimentally compared TSFF with four different feature fusion methods. 1) MeanPool: Temporal feature averaging in every skip connection to reduce dimensionality. 2) CNN: Adaptive temporal feature fusion. 3) TSFF(Bottleneck): Incorporating TSFF in bottleneck layers, calculating and applying weights to skip connections. 4) STT (Spatial-Temporal Transformer): Temporal feature fusion based on the STTumor-Seg [Zhang et al., 2023]. The detailed experimental results are reported in Tab. 4.

The results demonstrate the effectiveness of our proposed TSESNet with the TSFF module, surpassing alternative fusion strategies in terms of all evaluation metrics. Specifically, in terms of DSC, utilizing the TSFF module only at the bottleneck layer, the TSFF + Bottleneck approach achieved a significant increase in the DSC score (72.43) compared to STT (69.54). Despite obtaining a lower score than TSES- Net, this improvement validates the contribution of the TSFF module in enhancing segmentation precision. In computational efficiency, the TSFF consumes only 98.93M parameters and performs 75.42G FLOPs, which is substantially less than STT at 169.17M parameters and 124.58G FLOPs. Notably, the Bottleneck+TSFF configuration yields competitive results, highlighting the pivotal role of TSFF in not only reducing model complexity, but also improving the model's ability to effectively capture sequential features in DCE-MRI. The comparison revealed a distinct advantage over existing approaches, such as STT, thereby demonstrating the effectiveness of our proposed TSFF module in enhancing the performance of breast tumor segmentation.

5 Conclusion

In this study, we propose a novel Temporal-Spatial Enhanced Network (TSESNet) for Breast Tumor Segmentation in DCEMRI. Our approach leverages temporal-spatial contrastive learning to enhance the model's feature representation. It integrates contrast-enhanced features of breast tumors across different time points and incorporates a tumor-aware module that adapts to diverse tumor shapes and sizes. Additionally, the weight-sharing encoder structure and temporal series feature fusion module effectively capture sequence features in DCE-MRI while reducing the model's parameter count and complexity. Extensive evaluations on two DCE-MRI datasets validate the effectiveness, adaptability, and robustness of our proposed method in achieving accurate breast tumor segmentation.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 62276221, No. 62376232), the Science and Technology Plan Project of Xiamen (No. 3502Z20221025), and the key technologies for intelligent medical equipment (NO.20223160A0621)