ElaD-Net: An Elastic Semantic Decoupling Network for Lesion Segmentation in Breast Ultrasound Images

1 Introduction

Breast cancer is a leading cause of cancer-related deaths in women [Wilcock and Webster, 2021]. Automated breast ultrasound image (BUSI) segmentation is a key tool to assist early diagnosis and treatment planning. Convolutional neural networks (CNNs) [O'Shea and Nash, 2015] are widely used for medical image segmentation, but traditional CNNs struggle with pixel-level classification. To address this, Ronneberger et al. [Ronneberger et al., 2015] proposed U-Net, an encoder-decoder architecture with skip connections that preserves spatial details and integrates multi-level features.

∗ Corresponding Authors

In addition, several U-Net enhancements, such as U2 - Net [Qin et al., 2020], AAU-Net [Chen et al., 2023a], and Swin-Unet [Cao et al., 2022], integrate residual structures, multi-scale analysis, and attention mechanisms to capture intricate image details. However, these methods mainly focus on texture [Geirhos et al., 2019a], making it difficult to capture global shape information in BUSI with blurred boundaries. Thus, strategies like multi-scale fusion and deep supervision, as seen in UNet++ [Zhou et al., 2018], have been proposed. Yet, these approaches still struggle with the unique challenges of BUSI, hindering precise lesion segmentation.

1) Significant similarity fluctuations in pixel intensity distributions of foreground and surroundings. A major challenge in BUSI segmentation is the blurred edges of lesions. To elucidate the nature of this blur, we quantitatively analyze the similarity between the foreground (lesion) and surroundings. Figure 1 presents results from five random samples, where the lesion (in red) and surrounding tissue (in blue) exhibit similar morphological features (semantics). To quantify this, we calculate the Wasserstein Distance (WD) between their pixel intensity distributions, shown in grayscale histograms. Sample 2 demonstrates significantly higher semantic similarity than others (e.g., samples 1, 3, and 4) according to the proportion of overlapping areas. Additionally, even for the same lesion (e.g., sample 1), different edge positions show noticeable variations in similarity. We term this fluctuation in semantic similarity, a core challenge in addressing blurred boundaries often overlooked in existing studies.

2) Inconsistency in spatial details transmission caused by sampling of multi-scale lesions. U-Net, while foundational, relies heavily on downsampling, leading to the loss of key features [Rahman et al., 2024] and difficulty in recovering critical details. As shown in Figure 2, edge details are often captured roughly, with regions frequently missed. This issue arises from the challenge of maintaining spatial semantic consistency during transmission, as varying lesion sizes in BUSI complicate dynamic adaptation. Misalignment between edge and internal structure sampling disrupts semantic consistency, preventing accurate restoration of spatial details.

The core challenge lies in ensuring the elastic and consistent information preservation during semantic decoupling. Our main goal is to answer how to adaptively decouple semantics despite fluctuations in intensity similarity between the lesion and surroundings, while maintaining consistent spatial semantics across multi-scale lesions during sampling.

To this end, we propose an Elastic semantic Decoupled Network (ElaD-Net) for lesion segmentation in BUSI. ElaDNet utilizes EfficientNet-B2 for downsampling BUSI, with mobile inverted bottleneck convolutions to suppress noise and extract key features. Each upsampling step incorporates two modules: Elastic Semantic Decoupling (ESD) and Spatial Semantic Reconstruction (SSR), addressing similarity fluctuations and spatial detail inconsistencies. ESD creates multiscale channels, performs self-calibrated decoupling through multiple branches within each channel, and fuses correlated features to generate the overall map. SSR decomposes this map into edge and body semantic feature maps, which are optimized separately and reconstruct the feature maps for predicting the edge and body. Losses for body, edge, and overall features are balanced during experimentation.

Our main contributions are: 1) An elastic semantic decoupling module resists fluctuations in the pixel intensity distribution similarity between foreground and surroundings, enhancing semantic understanding in complex cases. 2) Spatial semantic reconstruction refines the feature maps generated by ESD, maintaining consistency in spatial detail transmission during sampling. 3) Extensive experiments on three real BUSI datasets demonstrate superior segmentation performance and robustness of the proposed method ElaD-Net.

2 RelatedWork

Feature Enhancement for BUSI Segmentation. U-Netbased methods have achieved notable success in medical image segmentation [Dong et al., 2023; Ning et al., 2022; Lin et al., 2022]. However, in BUSI, the foreground and surroundings have similar distributions, leading to blurred boundaries. To this end, some studies have introduced attention mechanisms and multi-scale feature fusion strategies. For example, Attention U-Net [Oktay et al., 2018] enhances focus on important regions through self-attention mechanisms. ESKnet [Chen et al., 2024a] designs enhanced selective kernel convolutions that integrate multiple feature map regions and adaptively recalibrate their weights from both channel and spatial dimensions, mitigating interference from less relevant areas. Moreover, to effectively suppressing irrelevant features in BUSI, Punn et al. [Punn and Agarwal, 2022] introduce residual initial convolutions (RIC) and cross-spatial attention (CSA) blocks into U-net. Liu et al. [Liu et al., 2024a] propose CMFF-Net, which combines Transformer's global context information with CNN-extracted local spatial information via cross-attention feature fusion. It enhances the feature representation. Lee et al. [Lee et al., 2020] introduce channel attention modules to improve U-Net's performance.

Information Loss Alleviating. Information loss during upsampling and downsampling is common in U-Net-based models for BUSI segmentation. To address this, some studies propose non-stride or dilated convolutions to retain spatial details. For example, DeepLab [Chen et al., 2015; Chen et al., 2018a; Chen et al., 2018b] addresses information loss with dilated convolutions and multi-scale context aggregation. In addition, some studies focus on designing reconstruction modules to recover the edge information lost during downsampling. Chen et al. [Chen et al., 2024b] use adaptive low-pass filter (ALPF) generators and adaptive high-pass filter (AHPF) generators to reduce position shifts caused by upsampling and recover high-frequency edge details lost during downsampling. BATFormer [Lin et al., 2023]ensures accurate edge information learning at different scales by providing supervision signals. Moreover, SegNet [Badrinarayanan et al., 2017] proposes max-pooling indices, where the decoder uses indices computed by the encoder to upsample lowresolution feature maps, achieving precise pixel-level recovery. Also, it effectively mitigates information loss.

However, the issues of fluctuations in the similarity of intensity distribution between the foreground and surroundings, as well as the inconsistencies in multi-scale spatial detail transmission, remain unresolved, making flexible semantic understanding and segmentation of BUSI challenging.

3 Method

3.1 Semantic Encoding

The similar morphological features between lesions and surrounding tissues pose challenges for sampling (semantic encoding). EfficientNet [Tan and Le, 2019] adopts a compound scaling strategy to adjust depth, width, and resolution, enabling efficient multi-scale feature extraction and noise suppression. EfficientNet-B2, using Mobile Inverted Bottleneck Convolution (MBConv) as its core building block, is chosen for our encoder design, as shown in the upper-left of Figure 3.

To address the cold start problem on BUSI and capture general image features, we pre-trained EfficientNet-B2 on ImageNet [Russakovsky et al., 2015]. The encoder details at each downsampling stage are shown in Table 1. Outputs ei from each stage are passed to the corresponding decoder via skip connections and mixed with decoupled semantics from the previous stage to preserve spatial details, as follows:

where [a : b] is concatenation, and Upu (·) denotes upsampling by a factor of u through a transpose convolution.

3.2 Elastic Semantic Decoupling (ESD)

The pixel intensity similarity between lesions and surrounding tissues in BUSI varies significantly, known as the FP problem (Section 1). Since fixed decoupling criteria are impractical for such mixed contexts, we expand branching in multi-scale channels and introduce elasticity into the decoupling process. Inspired by the multi-frequency segmentation strategy under multi-scale attention from [Nam et al., 2024], we further extend this idea by designing the Elastic Semantic Decoupling module, which introduces adaptive semantic disentanglement and dynamic frequency calibration for robust handling of foreground-background similarity fluctuations.

Multi-Scale Semantic Extraction

Breast lesions vary in scale, requiring dynamic adjustment of the proportion of key information extracted. We employ three convolutional layers with different kernel sizes, termed multiscale attention [Nam et al., 2024], for semantic extraction:

where xi ∈ RC×H×W denotes the feature map at the i-th decoder layer and k ∈ {3, 5, 7} the kernel size. Downs(·) indicates downsampling by a factor of s ∈ {1, 2, 4}, corresponding to no/2×/4× downsampling.

Self-Calibrating Elastic Decoupling (SCED)

Multiscale extraction allows the model to dynamically adjust its receptive field. If we treat the domain of similarity fluctuations as image depth, the next step is to endow the model with the ability to adjust this receptive depth. Features of different semantics can be emphasized by their corresponding frequencies [Azad et al., 2021; Geirhos et al., 2019b]. We believe that adjusting the focus of the model's decoupled semantics by utilizing frequency at different depths will help achieve the desired outcome.

We use the Laplacian pyramid mechanism [Lai et al., 2017] to map convolutional feature maps to the frequency domain, dynamically adjusting their weights based on frequency component importance. Low-frequency captures overall structure, while high-frequency captures edges and textures. Specifically, we approximate the Laplacian with the Difference of Gaussian (DoG). Starting from xsi , we extract "L + 1" levels (branches) of Gaussian representations with incrementally increasing variances φ1, φ2, . . . , φL:

Above, G(·) is the Gaussian filter, with * denoting convolution, and j1, j2 as spatial positions in the feature space.

The Laplacian Pyramid LP is constructed by calculating differences between adjacent Gaussian representations:

where LP s i,l is the output of the l-th layer LP from the i-th decoder layer and scale s. For each spectral feature map LP s i,l , we apply average pooling to mitigate noise, max pooling to emphasize locally salient features, and min pooling to capture dark areas or negative outliers. This multi-pooling strategy enhances discriminative power for elastic decoupling:

where Z (s,i) p,l ∈ RCs×1×1 . We then average these pooling results across all pyramid levels to yield combined features:

To dynamically adjust the importance of frequency components, we compute an attention matrix from the pooling results to calibrate the original semantics. Specifically, the pooled features are first passed through two fully connected layers for self-calibration: FC1 reduces the dimension to C/r with reduction rate r, while FC2 restores it to C. Here, δ denotes the ReLU activation function. The attention matrix M s i is then generated by adding and activating with the Sigmoid function σ(), enabling debiasing in the fluctuation (depth)- based elastic semantic decoupling, as follows:

Correlation Feature Fusion (CFF)

To aggregate the correlated foreground details from multiscale semantics while distinguishing the surroundings, we developed an improved multi-scale feature fusion mechanism inspired by [Nam et al., 2024], generating attention maps for both foreground and surroundings. We established two independent attention leaves for ˆxsi :

Squeeze-and-Excitation (SE) layer, illustrated in Figure 4, is designed here to suppress irrelevant features and highlight important ones within each leaf. SE utilizes Global Average Pooling (GAP) to convert the input into a vector, with dimension C representing the leaf's size. Two convolution layers are then applied: one reduces the size from C to r, and the other restores it to C. Afterward, the Sigmoid function generates quantized values, which are multiplied element-wise with original input to suppress or enhance features. The process is defined as follows:

The feature maps are integrated and upsampled to maintain dimensional consistency across different scale channels.

Finally, we fuse the spatially refined feature maps from all scales with learnable weights Ws:

3.3 Spatial Semantic Reconfiguration (SSR)

To address spatial detail inconsistency (Section 1), we propose to enhance the transmission process by decomposing it into three key steps: decomposition, optimization, and reconstruction, thereby recovering the lost spatial details.

Decomposition. Thanks to ESD's semantic decoupling, we can easily separate the body component sbody i and the edge component sedge i from the stage feature map˜xi, as follows:

where AvgPool is a 5×5 average pooling with stride = 1, padding = 2. This facilitates the independent transmission of the two components to reduce spatial detail loss.

Optimization. To mitigate body component attenuation during the cross-scale transmission, we introduce a transmission optimization mechanism to enhance spatial semantics:

Here, SE is employed to amplify key features while suppressing irrelevant ones, and integrate legacy information from the previous level to restore finer details, ultimately achieving a finely tuned body component ˆ s′′body i . The same approach is applied to tune the edge component ˆ s′′edge i :

Reconstruction. We reintegrate the two components with the final semantic feature map ˜xi decoupled from SSR, forming a cohesive whole that effectively recovers the spatial details lost during downsampling, as follows:

Both ˆ s′′edge i and ˆ s′′body i are passed through a 1x1 convolution to obtain the predicted edge probability map pe and the body probability map pb, respectively.

3.4 Loss Function

To enable more comprehensive segmentation, we compute the loss for three segmentation results pe, pb and po:

Above, le, lb, and lo are the ground truth. Considering the class imbalance in edge information, the Dice loss [Milletari et al., 2016] LDice is utilized to ensure the accuracy of edge detection. The weighted binary cross-entropy loss LwBCE and the weighted intersection over union (IoU) loss LwIoU are combined to enhance the model's ability to learn different classes and complex shapes. Finally, we calculate the total loss by assigning weights, i.e., α, β, µ, to the aforementioned components: LF = αLO + βLE + µLB.

4 Experiments

4.1 Experimental Setup

Datasets Experiments are conducted on three publicly BUSI datasets (Table 2): the BUSI-E9 [Al-Dhabyani et al., 2020] dataset from Bahia Hospital, Dataset B [Yap et al., 2020] acquired using the Siemens ACUSON Sequoia C512 system, and the STU [Zhuang et al., 2019] dataset from the Imaging Department of Shantou University Medical Center.

Evaluation Metrics We adopt five common segmentation metrics to evaluate performance: Jaccard Index (IoU), Precision, Recall, Specificity, and Dice Coefficient (DC).

Experimental Details We employ a four-fold crossvalidation approach in experiments. We use the Adamax optimizer with an initial learning rate of 0.002 to train our model. The hyperparameters for loss calculation, α, β, and µ, are set to 0.4, 0.3, and 0.3, respectively. The epoch is set to 300, and the batch size is 6. The experimental environment includes PyTorch 1.13, Python 3.8, and an NVIDIA RTX 4090 GPU.

4.2 Ablation Study

Parameter Ablation. To evaluate the impact of different loss weight configurations on segmentation performance, we designed a series of ablation experiments testing various α : β :

µ. Experimental results shown in Table 3 indicate that when the weights are set to 0.4 : 0.3 : 0.3, the segmentation accuracy reaches the optimal state on BUSI-E9 and Dataset B because the edge loss and body loss are properly balanced.

Evaluation of Module Effectiveness. 1) To systematically verify the impact of different modules on model performance, we conducted ablation studies (Table 4) on the BUSI-E9 dataset using four-fold cross-validation. Replacing traditional U-Net with EfficientNet-B2 led to significant improvements across all metrics, with the Dice coefficient rising from 70.1 to 80.31. As ESD and SSR were progressively introduced, segmentation performance further improved, owing to FP problem resistance from multi-scale feature extraction, self-calibrated elastic semantic decoupling, and spatial semantic consistency from SSR. 2) To assess the impact of edge and body reconstruction within SSR, we test three variants: w/o edge (no edge branch), w/o body (no body branch), and w/ body-edge (both branches). Results in Table 5 show that the w/ body-edge variant yields the best segmentation, with the Jaccard index rising from 72.87 (w/o edge) and 73.49 (w/o body) to 73.86. Adding either branch improves performance, though not as much as using both.

4.3 Segmentation Results

To evaluate the effectiveness of ElaD-Net, we select several state-of-the-art methods as baselines, including U-Net [Ronneberger et al., 2015], Att U-Net [Oktay et al., 2018], UNet++ [Zhou et al., 2018], SegNet [Badrinarayanan et al., 2017], BasNet [Qin et al., 2021], AAU-Net [Chen et al., 2023a], NU-Net [Chen et al., 2023b], ESKnet [Chen et al., 2024a] and Rolling-Unet [Liu et al., 2024b].

Comparison of Segmentation Results. The experimental results are summarized in Table 6. Bold and underline represent the best and second-best performance results, respectively. Asterisks (∗ : p < 0.05) indicates a significant difference via a paired t-test. U-Net and Att U-Net perform the worst, since they fall short in precisely capturing the lesion features and distinguishing interference in BUSI. U-Net++ leverages deep supervision by utilizing intermediate layer outputs for auxiliary loss calculations, yet it struggles to maintain edge information. SegNet mitigates information loss from downsampling via max-pooling indices but still lacks precision in segmenting fine structures in BUSI. Other methods (e.g., BasNet, AAU-Net, NU-Net, and ESKnet) have improved segmentation performance but often rely on specific contextual information or local feature extraction, limiting their adaptability for variably shaped lesions. In contrast, ElaD-Net exhibits significant advantages across multiple aspects. Its IoU value improves by 4.9% on BUSI and 7.2% on Dataset B compared to the second-best method, NU- Net. it achieves elastic semantic decoupling through the ESD module, allowing it to more accurately capture lesion features and differentiate from surrounding environmental interference. Additionally, the SSR model optimizes the transmission of body and edge information, addressing the issue of spatial detail loss caused by downsampling.

Qualitative Comparison. Figure 5 illustrates a visual comparison of different segmentation methods, featuring examples with varying lesion scales and foreground-surrounds intensity distribution similarities. It is clear that our method (last column) produces segmentation results that are closer to the ground truth (second column). Other methods, particularly U-Net, AttU-Net, U-Net++, SegNet, and ESKNet, perform poorly in complex scenarios (rows 2, 3, 5, and 6). This further highlights the robustness of our method.

4.4 Robustness Analysis

Robustness on Benign and Malignant Lesions. Benign lesions have regular shapes and clear boundaries, while malignant lesions are irregular, with fuzzy boundaries and complex intensity distributions, demanding higher segmentation capabilities from the model. As shown in Table 7, our method achieved the best results in both benign and malignant lesion segmentation. The IoU for benign and malignant lesions were 77.22 and 66.94, respectively, demonstrating the robustness of the method in handling the complex morphology of malignant lesions. In contrast, methods like U-Net and Att U-Net struggled with malignant lesions due to their reliance on simpler contextual information, resulting in poor segmentation with missed detections and false positives.

Cross-Dataset Validation. To evaluate the model's generalization, we conducted cross-dataset validation (Table 8). Our method outperformed all others, achieving a Dice coefficient of 74.15, Precision of 89.34 on Dataset B (using a model trained on BUSI-E9). On the STU dataset, it achieved a Dice of 84.57 and Specificity of 97.76, demonstrating strong robustness. In contrast, methods like Att U-Net and U-Net++ showed significant performance degradation, highlighting our method's superior generalization.

5 Conclusion

This study presents ElaD-Net, a novel model for BUSI segmentation that tackles complex backgrounds, similarity fluctuations, and spatial detail loss. Built on a pre-trained EfficientNet-B2 encoder, ElaD-Net integrates two key modules: Elastic Semantic Decoupling (ESD) and Spatial Semantic Reconstruction (SSR). ESD enables multi-scale extraction, self-calibration, and feature fusion to improve robustness against FP issues. SSR decouples body and edge features, then optimizes and reconstructs them to preserve spatial consistency. We balance losses across body, edge, and full outputs to enhance accuracy. Extensive experiments on three real BUSI datasets validate each module's effectiveness, showing superior performance and robustness. Future work will focus on accelerating segmentation for clinical use.

Acknowledgments

This work was supported in part by the National Key R&D Program of China under Grant 2023YFB3107303, in part by the National Natural Science Foundation of China (62402253 and 62301289), in part by the project ZR2024MF050, ZR2024QF109 and ZR2022QF041 supported by Shandong Provincial Natural Science Foundation, in part by the Taishan Scholars Program under Grant tsqn202211210, in part by the Pairing Plan Project of the School of Computer Science and Technology of Qilu University of Technology (Shandong Academy of Sciences) under Grant 2024JDJH12, in part by the "20 New Universities" Project of Jinan City under Grant 202333023 and Grant 202333045, and in part by the Young Innovation Team of Colleges and Universities in Shandong Province 2024KJH044.