DenseSAM: Semantic Enhance SAM for Efficient Dense Object Segmentation

1 Introduction

Dense object segmentation is a critical visual task with broad applications in specialized fields such as cell segmentation in pathology image analysis. This task involves numerous similar and closely packed objects, posing challenges in adhesion and differentiation. Numerous methods have been proposed to tackle these challenges, including CNN-based [Tian et al., 2020; Yoo et al., 2019] and ViT-based [Prangemeier et al., 2020; Lin et al., 2024]. Although these methods have shown remarkable performance, they require extensive training resources and exhibit limited generalization ability.

Recently, the advent of the Segment Anything Model (SAM) [Kirillov et al., 2023] has marked groundbreaking progress in zero-shot image segmentation. Leveraging a vast number of parameters and pre-training on the SA-1B segmentation dataset, SAM excels at accurately segmenting objects across diverse domains. However, despite its powerful capabilities, SAM remains significantly limited in dense object segmentation due to its heavy reliance on numerous accurate manual location prompts to achieve satisfactory results.

As shown in Fig.1 (a), when a single positive point is provided, two adjacent cells are segmented together. Furthermore, using a few positive and negative points still results in large areas of adhesion (b). Consequently, satisfactory results are achieved only when bounding box prompts are provided for each object (c). However, locating all objects in dense scenarios is inherently time-consuming and labor-intensive. This raises two key questions: Why does SAM heavily rely on precise location prompts, and how can we achieve automatic and accurate segmentation of dense objects without relying on location prompts?

To answer the first question, we delved deeply into the mechanics of SAM. First, we hypothesize that SAM's encoder detects all objects in an image but relies on location prompts as query tokens to employ the attention mechanism for locating image tokens corresponding to objects near the specified positions. Based on this hypothesis, we adopted two metrics to analyze the image tokens: (1) a local metric that calculates the variance of image tokens, and (2) a global metric that measures the cosine similarity between image tokens. As shown in Fig. 2(b), the image tokens output by the SAM encoder exhibit higher variance at the object edges and high similarity between the selected object token (green star) and other objects, including those with different semantics (green arrow), which validates the above hypothesis. After crossattention in the SAM decoder (c), the variance of image tokens within the selected object becomes consistent, and the similarity with distant object tokens decreases. Therefore, we infer that SAM accurately identifies object edges but relies on precise location to compute attention between query and image tokens. More specifically, SAM's strong generalization ability stems from its encoder, while its decoder depends on positional cues to locate objects detected by the encoder, rather than on the semantic relationships between objects.

To address the second question, semantic guidance is proposed as a replacement for location prompts to achieve automatic segmentation. Specifically, we observe that in dense object segmentation, these numerous closely packed objects often share similar and homogeneous semantic features. This observation inspired us to leverage SAM's strong generalization capability in edge detection and replace location-based prompts with semantic guidance, enabling automatic and precise segmentation. Thus, an Efficient Semantic Injection (ESI) module is proposed to achieve this goal. Specifically, image tokens from the hidden layers of SAM's image encoder are utilized for local information extraction and global context contrastive learning. By using a local workflow to attenuate background object edges and a global workflow to enhance intra-class feature similarity while increasing contrast with the background, these semantic features are integrated into the decoding process. The metric results for both local and global analysis on the ESI output image tokens are shown in Fig. 2(d). As illustrated, the distribution of image tokens reveals a clear spatial correspondence between foreground and background semantic information. Additionally, a dualhead decoding structure is designed to simultaneously output objects and their boundaries, significantly improving the distinction between adjacent objects. By utilizing semantic guidance, along with the efficient dual-head structure, we enable lightweight automatic semantic segmentation with only a few trainable parameters.

Our contributions are summarized as follows: (1) We propose an automated, cost-efficient segmentation framework, DenseSAM, which is built upon a thorough analysis of SAM, demonstrating that its generalization ability arises from its precise edge detection. (2) A novel Efficient Semantic Injection (ESI) module is proposed that integrates local and global perspectives to enhance foreground-background distinction. (3) Extensive experiments showing DenseSAM's state-ofthe-art performance with minimal parameters and strong generalization to new tasks.

2 RelatedWork

Dense Object Segmentation. Dense object segmentation is a challenging task that deals with identifying and separating individual objects within a dense scenarios [Tu et al., 2008; Li et al., 2018; Lou et al., 2012]. For specialized domain like pathology and remote sensing image analysis, its main challenges include not only segmenting numerous closely packed objects but also addressing the frequent occlusions, partial coverage, and objects adhesion, which significantly complicate the segmentation process. Prevailing approaches often enhance task-oriented segmentation by training models with extensive domain-specific samples to overcome these challenges. Depending on the model structure used, they can be roughly categorized as CNN-based, ViT-based, and a combination of both. CNN-based methods excel at capturing finegrained local and multi-scale features through their hierarchical convolutional structures, making them ideal for distinguishing closely packed objects in dense scenarios [Tian et al., 2020; Yoo et al., 2019]. In contrast, ViT-based methods can handle larger receptive fields and effectively capture global contextual information, making them well-suited for managing complex scenes with significant contextual relationships [Wang et al., 2024; Prangemeier et al., 2020]. Building on the complementary strengths of CNNs and ViTs, various approaches have emerged that integrate both into a single framework to achieve enhanced performance [Li et al., 2023; Hu et al., 2023].

Even though these expert methods have shown promising results in specific fields, training them from scratch demands substantial resources. Moreover, they exhibit poor generalization across different domains, such as a model tailored for cell segmentation is unsuitable for building extraction.

Segment Anything Model and It's Variants. The Segment Anything Model (SAM) [Kirillov et al., 2023] is an advanced universal segmentation model, which consists of an image encoder, a prompt encoder, and a mask decoder. Trained on 1.1 billion masks and 11 million images, SAM demonstrates impressive zero-shot capabilities and has inspired a surge of research into its potential across various downstream tasks. Depending on which components of SAM are tuned, we classify those variants into four types: a). Tuning whole SAM, b). Tuning minimal parameters, c). Training auxiliary network, d). Training auxiliary network without location prompts. For example, MedSAM [Ma et al., 2024] fine-tuning the entire SAM on 1.57 million medical images for promptable medical segmentation. To enable efficient fine-tuning without relying on large datasets, Med-SA [Wu et al., 2023] fine-tunes minimal additional parameters in the SAM image encoder for efficient adaptation to medical imaging, while SAM-PARSER [Peng et al., 2024] and CATSAM [Xiao et al., 2024] use similar techniques for various domains, including medical imaging, remote sensing, and natural images. More advanced approaches [Hu et al., 2024c; Zhang et al., 2024b] use or train auxiliary networks (such as language models or expert models) to generate location prompts automatically, replacing the time-consuming manual intervention. Apart from using location prompts, some methods [Zhang et al., 2024a; Chen et al., 2024] eliminate them altogether by training auxiliary networks to generate masks directly, while either training or freezing the image encoder.

We propose a new type that achieves automatic segmentation by training only a few semantic parameters, offering a cost-effective and efficient solution for adapting SAM to dense object segmentation.

3 Why Dose SAMRely on Location Prompts?

Inspired by studies analyzing the working mechanisms of visual models [Selvaraju et al., 2017; Hu et al., 2024b; Hu et al., 2024a], in this section, we aim to understand the internal mechanism of SAM and investigate its reliance on precise location prompts. Based on SAM's architecture, which consists of a heavyweight image encoder with 640M parameters and a lightweight mask decoder and prompt encoder with a combined 4M parameters, we hypothesize that SAM's encoder detects all objects in an image but requires location prompts as query tokens to identify image tokens corresponding to objects near the specified positions. To validate this hypothesis, two analytical metrics are introduced to examine the distributional changes of image tokens within the encoder and decoder: (1) a local metric that calculates the variance of image tokens and (2) a global metric that measures the cosine similarity between image tokens.

First, the feature extraction mechanism within the SAM image encoder was examined, focusing on its reliance on a series of attention operations applied to flattened onedimensional image patch embeddings. This process can be represented by the following equation:

where El represents the encoder layer's output image tokens with dimensions (h, w, c), h, w and c denote the height, width, and channel, respectively. The total number of encoder layers is denoted by L. The projection function, fproj, maps the flattened image tokens (h ∗ w, c) before it is fed into the attention operation fattn. After computing attention, the image tokens are reshaped back into two-dimensional space and projected by f ′proj , returning El to (h, w, c).

We visualized the layer's output El using heatmaps for analysis. Keeping h and w constant, we calculated the variance along the channel c to generate the local metric heatmap in Fig.2, revealing that image tokens near object edges exhibit greater variability in their channel features. Then, we calculated the cosine similarity between a specific token (the green star in Fig.2) and all other tokens, generating a similarity matrix of shape (h,w), which forms the global metric heatmap displayed in Fig.2. As observed, tokens within the same object exhibit the highest similarity, while tokens from other objects, regardless of distance or semantic class, also display high similarity, as indicated by the green arrows in Fig.2. Thus, it is concluded that the SAM encoder extracts the edges of all objects and preserves object-centric similarity, irrespective of their semantics.

Next, the image token update process during SAM's decoding phase was analyzed using the same statistical approach. In this phase, image tokens are decoded alongside the location prompts, as demonstrated below:

Here, Rk and Qk represent the one-dimensional image tokens and the query tokens containing location prompts in the SAM mask decoder, respectively. The core module of the mask decoder, ftwa, refers to the Two-Way Attention block, which performs k rounds of cross-attention to update the feature values of Rk and Qk. The updated R2 and Q2 are then passed into the decoding function fdecode, where R2 is reshaped into a two-dimensional space and upsampled, while Q2 undergoes processing through a set of MLPs for channel mapping. The final output masks M are generated by combining the upsampled R2 and the mapped Q2.

The same local and global analyses are performed on the updated R2 under one-point location prompts. As shown in Fig.2, by incorporating the location of the green star into the query tokens, the variance of image tokens belonging to the green star's object becomes more consistent after two iterations of cross-attention between the query tokens and image tokens, while the cosine similarity with distant tokens decreases. More typical cases on the updated R2 are provided in Fig.4. The analysis results further confirm our hypothesis that the lightweight SAM decoder relies heavily on precise and unambiguous location prompts to identify corresponding objects, and without such prompts, SAM performs significantly deteriorates with densely packed homogeneous objects.

The above analysis inspired us to explore whether semantic guidance could replace location prompts for dense objects with homogeneous semantics to achieve a lightweight, automatic, and efficient segmentation tool.

4 How To Enhance SAM for Dense Objects?

Since SAM's strong generalization ability stems from its encoder, while its decoder relies on location prompts to segment objects, the presence of numerous semantically similar objects in dense object scenes inspired us to replace location prompts with semantic guidance in the decoder.

To enable semantics-guided automatic segmentation, an Efficient Semantic Injection (ESI) module is proposed, which includes both local and global workflows. As shown in Fig.3, the image tokens from the hidden layers of the SAM encoder are leveraged for local details extraction and global context comparison, given its capability to detect object edges, as analyzed in Section 3. In the local workflow, after reconstructing the image tokens into a two-dimensional (2D) space, a depth-wise convolutional layer is incorporated to capture local information, as demonstrated to be beneficial in previous works [Yuan et al., 2021; Li et al., 2021]. The process is expressed by the following formula:

where flocal denotes the local module to process image tokens Ek. For different layer's output Ek, the parameters of the flocal module are shared, where K is a hyper-parameter.

In the global workflow, supervised contrastive learning [Khosla et al., 2020] is employed to enhance the similarity within the same semantic category and amplify the distinction between different categories. The process is expressed by the following formula:

where fglobal denotes the global module, and the weights are shared, similar to those in flocal.

In summary, the semantic injection process of ESI can be expressed as follows:

Subsequently, the Rnew 2 is fed into the decoder for crossattention with the query tokens. The analysis results on both local and global metrics validate the effectiveness of the proposed ESI module. As shown in Fig.2, the output RESI increases the variance of background object tokens and the similarity between foreground tokens increases, while their similarity with background tokens is significantly reduced.

To refine the distinction between overlapping objects, we have innovatively designed a Dual-Head (DH) structure, enabling the simultaneous output of both object and its boundaries. Specifically, without location prompts, we incorporate semantic embeddings into the query tokens and use two sets of MLPs for mapping the query tokens. The process can be expressed as following:

where Q′2 denotes the query tokens incorporated with newly added semantic embedding, which is processed by f DH decode , the proposed Dual-Head decoding structure that includes two sets of MLPs. The output results, Mb and Mo, represent the masks of objects and their boundaries, respectively. A marker-controlled watershed algorithm is applied to postprocess these two masks, as in Al-Net [Zhao et al., 2021], yielding the final instance segmentation results.

The modules we proposed require only about 3M trainable parameters, making lightweight automatic semantic segmentation feasible. In the following experimental section, we will demonstrate the effectiveness of the proposed DenseSAM on six datasets and conduct an ablation study to substantiate the validity of each component.

5 Experiments

5.1 Experimental Setup

Datasets. To evaluate the performance of the proposed DenseSAM in dense segmentation of pathology images, we used three commonly used pathology datasets. Additionally, to demonstrate the applicability of DenseSAM in other dense segmentation domains, we also used three common remote sensing datasets, showcasing DenseSAM's robust dense segmentation capability. The six datasets are as follows:

Pathology datasets: CPM17 [Vu et al., 2019] includes 32 / 32 pathology images for train / test, with sizes of 500 × 500 or 600 × 600 pixels. We resize each image to 1024 × 1024 pixels, and then crop it into 512 × 512 pixel patches with no overlap. CoNIC [Graham et al., 2021b] consists of 4981 image patches, each sized 256 × 256. Following BoNuS [Lin et al., 2024] we randomly split all images into 7:1:2 ratio, resulting in 3486 / 997 / 498 for train / validation / test. MoNuSeg [Kumar et al., 2017] comprises 44 images with each of size 1000 × 1000 pixels. The dataset has 30 / 14 images for train / test. We randomly split the train subset into 24 / 6 images for train / validation. Each image is resized to 1024 × 1024, and then crop to 256 × 256 without overlap.

Remote sensing datasets: WHU Building [Ji et al., 2018] has a ground resolution of 0.3 meters and an image size of 512 × 512 pixels. It contains 4736 / 1036 / 2416 images for train / validation /test. Inria Building [Maggiori et al., 2017] contains 360 images collected from five cities at a 30cm resolution. We process it consistent with UANet [Li et al., 2024] and crop them in to 512 × 512 pixels, resulting in 9737 / 1942 images for train / validation. Massachusetts Building [Mnih, 2013] owns 151 aerial images with spatial resolution 1 meters and an image size of 1500 × 1500 pixels. we crop the images into 500 × 500 pixels, get 1233 / 36 / 90 images for train / validation / test.

Metrics. For semantic segmentation, we adopt three pixellevel metrics: Intersection over Union (IoU), F1 score (F1), and precision (Pre.). We calculate the mean value of the background and foreground in these metrics, consistent with UANet [Li et al., 2024]. For instance segmentation, three object-level evaluation metrics is utilized: object-level Dice coefficient (Dice), aggregated Jaccard index (AJI), and panoptic quality (PQ), align with BoNuS [Lin et al., 2024].

Implementation Details. By default, we use the ViT-H type for the image encoder, unless otherwise specified. For the loss function, we use a linear combination of Dice loss, BCE loss and contrastive loss. We adopt Adam optimizer and training ranges from 10 to 30 epochs.

5.2 Quantitative Results

In this section, we compare the DenseSAM against several NonSAM-based and SAM-based methods on pathology and remote sensing datasets. To ensure fairness, we reference results from the original papers or reproduced SAM-based methods using open-source code when results were missing.

Pathology Instance Segmentation. To validate the effective of the proposed DenseSAM, we compare three instance metrics on the CoNIC, CPM17 and MoNuSeg datasets. As shown in Table 1, DenseSAM outperforms expert networks including PseudoEdgeNet [Yoo et al., 2019], Scribble2Label [Lee and Jeong, 2020], C2FNet [Tian et al., 2020], SSL-Net [Xie et al., 2020], SC-Net [Lin et al., 2023], and BoNuS [Lin et al., 2024]. These expert networks are specializing in cell segmentation tasks which training parameters range from 10 to 100 million, far beyond of DenseSAM.

For SAM-based methods, Med-SA [Wu et al., 2023] and UN-SAM [Chen et al., 2024] is selected for comparison. Table 1 shows that Med-SA with point prompts performed rather poorly on AJI metric, which is the primary metric for evaluating whether adherent cells are separated. Moreover, when the point prompts are removed, Med-SA's performance significantly deteriorates across all metrics. UN-SAM trains a large number of parameters to eliminate the need for location prompts. However, compared to UN-SAM, DenseSAM achieves more impressive performance across all metrics with fewer trainable parameters.

It is worth mentioning that CoNIC is extracted from the Lizard [Graham et al., 2021a] dataset, which includes around fifty thousand nuclei from 16 different centers, representing diverse cell segmentation domains. The proposed DenseSAM demonstrated superior performance compared to expert models on the CoNIC.

Remote Sensing Semantic Segmentation. Table 2 shows the comparison results on three remote sensing datasets. UNet [Ronneberger et al., 2015] and Uniformer [Li et al., 2023] are classic segmentation networks used in general domains, and FSAU-Net [Hu et al., 2023], UANet [Li et al., 2024], RSBuilding [Wang et al., 2024] and GLGFF-Net [Fu et al., 2024] represent SOTA expert networks for building extraction in recent years. Table 2 shows that DenseSAM, which leverages SAM's powerful capability while training only 0.6% of its parameters, surpasses domain-specific models that require training a large number of parameters.

For SAM-based methods, SAM-PARSER [Peng et al., 2024] and CAT-SAM [Xiao et al., 2024] require training a small number of additional parameters but still relying on manual location prompts. RSAM-Seg [Zhang et al., 2024a] eliminates the need for location prompts but incurs significant training costs. From Table 2, it can be observed that DenseSAM surpasses these SAM-based methods without requiring manual prompts or incurring significant training costs.

5.3 Qualitative Results

We visualized representative results of DenseSAM across six datasets. Fig. 6 shown that DenseSAM can identify most dense objects without location prompts, even for very small areas, such as the small building in the Massachusetts. However, the proposed method has limitations. In Fig. 6, cases of poor performance, such as undifferentiated adhesive cells, background misidentified as cells, and shadowobscured buildings, are highlighted with red circles.

5.4 Ablation Study

In this section, we conduct ablation studies on CPM17. By sequentially removing each of the proposed modules, the experimental results clearly demonstrate their effectiveness. As shown in Table 3, SAM exhibits strong object detection capabilities. However, even with full retraining, its decoder still delivers subpar performance on instance segmentation metrics such as AJI and PQ. DenseSAM leverages SAM's powerful edge detection and object-centric similarity capabilities to achieve SOTA performance with minimal training cost. The ablation visualization in Fig.5 further demonstrates the effectiveness of the proposed modules. Red arrows indicate issues caused by removing specific modules, including missed cell detection and failure to separate adherent cells.

6 Limitations and FutureWork

The proposed DenseSAM, while performing well across multiple datasets for automatic dense object segmentation, has some limitations that warrant further exploration. Firstly, DenseSAM relies on pixel-level annotated masks for supervised training. Given that homogeneous dense objects often follow unified prior patterns, future research could investigate the incorporation of domain-specific knowledge to mitigate this dependency. Secondly, DenseSAM is limited to segmenting semantic masks for specific classes and does not handle multi-class segmentation tasks. Future research could focus on addressing this limitation.

Beyond specialized domains, the pattern analysis and potential applications of SAM in natural images are worth exploring. The analysis results of MSCOCO [Lin et al., 2014] samples are shown in Fig. 7. As shown, SAM demonstrates strong edge detection ability in natural images, but objectcentric similarity patterns appear only in the densely packed fruits in the fourth column. These characteristics highlight the need for further investigation into maximizing SAM's capabilities with minimal training costs.

7 Conclusion

In this paper, we present a novel, cost-effective, and efficient solution for automatic dense object segmentation. Using the introduced local and global metrics, we thoroughly analyzed SAM's mechanism and discovered that its strong generalization ability stems from its precise edge detection capabilities. Moreover, SAM exhibits robust object-centric feature similarity in dense scenes, both of which are independent of object semantics. This inspired us to replace positional cues with semantic guidance for dense object segmentation, particularly in scenarios involving numerous similar and homogeneous objects. Therefore, we propose DenseSAM, which utilizes the ESI module to effectively inject semantic information into the decoding process, replacing the time-consuming and labor-intensive manual location prompts. Through local and global workflows, the ESI module effectively distinguishes between foreground and background objects. Equipped with a dual-head structure, DenseSAM enhances the differentiation of overlapping object edges, enabling lightweight and automatic instance segmentation. Extensive experiments across six datasets for pathology and remote sensing image analysis demonstrate that DenseSAM achieves state-of-the-art performance in both semantic segmentation and instance segmentation metrics, with only ∼3M trainable parameters, making it an economical and practical tool for dense object segmentation. In future research, it is anticipated that the in-depth exploration of the SAM mechanism and the efficient, low-cost semantic injection method in DenseSAM will inspire more innovative ideas.

Acknowledgements

This work is supported by National Natural Science Foundation of China (62376248), the Huadong Medicine Joint Fund of the Zhejiang Provincial Natural Science Foundation of China (LHDMZ25H160002), the Zhejiang Province Health Major Science and Technology Program of National Health Commission Scientific Research Fund (No. WKJ-ZJ-2426) and Information Technology Center, ZheJiang University.