INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation

1 Introduction

Task-generic promptable segmentation leverages a taskgeneric prompt to derive instance-specific prompts for segmenting diverse images under a unified task. Unlike traditional segmentation approaches that require extensive labelled datasets, this method only uses a task-generic prompt applicable across all test samples, e.g. "the polyp" is a taskgeneric prompt for all images in a polyp segmentation task. This approach presents greater challenges due to the lack of labels, but it is more appealing for real-world applications.

A task-generic prompt, being both coarse and potentially ambiguous, can lead to poor segmentation when directly applied. Initial solutions [Hu et al., 2024a; Liu et al., 2023c]

first proposed to use VLMs to mine information from images, deriving instance-specific prompts from a generic task prompt to guide SAM in segmenting task-related objects. However, they often struggle with complex samples where task-relevant objects are hard to distinguish. This can lead to inaccurate segmentation and degrade model performance. ProMaC [Hu et al., 2024b] mitigated this issue by treating hallucinations as a form of prior knowledge. It masks the objects of interest to induce hallucinations, thereby extracting information that appears relevant to the task, aiding in the reasoning of instance-specific prompts. Although these methods have had some success, the lack of labels on samples makes it hard to check if the deduced prompts are accurate. As a result, once an incorrect prompt is generated, it becomes difficult to fix and the errors may increase over time. Yet the question of eliminating this need remains unexplored.

Despite the lack of manual annotation, the output changes of the VLMs show how they link candidate categories to the task. As shown in Fig. 1(a), a frog is hidden among branches that blend with the tree trunk, making it tough to spot. Using the VLM directly, even though the frog triggers a high activation, the tree trunk incorrectly scores higher for the 'snake' class, causing a misclassification of the branch as 'snake'. If we cover the likely area of the camouflaged animal, the identical feature identifying the 'frog' is hidden, but the 'snake' features remain visible on the tree trunk and bark. This leads to a significant decrease in the 'frog' score, while the 'snake' score is barely affected. This drastic alteration in the VLM response when key features are occluded provides valuable insight: we can assess the accuracy of prompts by observing changes in VLM responses when parts of an image are hidden. The element with greater variation is more likely to be the correct instance-specific prompt. Moreover, the difference between the correct category and easily mistaken ones in VLM outputs contrast is often small, leading to incorrect object identification when relying on a single output. In Figure 1(b), under the premise of including the ground truth mask, we randomly drew 2 masks and calculated the VLM changes before and after covering these masks, multiplying them by category to implement hard false class negative mining. Compared to using just one mask for covering, the instance-specific prompt inferred in this way has a higher CLIP semantic similarity to the ground truth class.

In this paper, building on this observation, we propose INT, which aims to gradually eliminate the impact of incorrect categories through progressive negative mining, utilizing changes in the VLM's output. It consists of instance-specific prompt generation and semantic mask generation. For the first component, the model splits the input image into different patches and processes them in parallel. In these patches, objects of interest might be fully or partially visible. This variation induces the model to use its pre-trained knowledge to guess the objects' names and locations within a patch. The names and locations of objects predicted for each patch are then sent to an image inpainting module, which erases the predicted objects and fills the space with the surrounding background. The outputs of the VLMs are compared before and after the image inpainting, and the predictions with the largest output difference are selected as the instance-specific prompts for this iteration. The normalized weights of these differences are multiplied with the VLM's output in the next iteration. This approach helps progressively improve classification of hard-to-distinguish categories and allows for correction of initially misclassified samples as the iterations continue. The semantic mask generation module uses an GroundingDINO [Liu et al., 2023c] to task-related objects in the image. The detected bounding boxes and prompts are then processed by SAM and refined using semantic similarity with CLIP. Masks above the similarity threshold are combined to produce the segmentation for this iteration, and the corresponding soft mask is applied to the original image to enhance segmentation in the next iteration. Out contributions are as follows:

• 1). We introduce INT, a training-free test-time adaptation approach that uses progressive negative mining to identify more accurate instance-specific prompts in the absence of annotations, enabling accurate task-generic promptable segmentation.
• 2). Progressive negative mining identifies hard-todistinguish error categories by cumulatively multiplying the changes in VLM outputs before and after masking across multiple iterations by category, thereby ensuring the accuracy of the generated instance-specific prompts.
• 3). Experiments on six datasets demonstrate the effectiveness of our method.

2 RelatedWorks

Vision Language Models (VLMs) are adept at handling tasks in both vision and vision-language modalities. They encompass visual understanding models [Krizhevsky et al., 2012; Radford et al., 2021; Liu et al., 2023a], visual generation models [Ramesh et al., 2021; Rombach et al., 2022], and general-purpose Interfaces [Zhang et al., 2023a; Alayrac et al., 2022]. Visual understanding models develop robust visual representations that serve as the backbone for various computer vision downstream tasks. Thanks to extensive image-text datasets, visual generation models are equipped to tackle a range of visual tasks, such as segmentation and object detection in images or videos [Kirillov et al., 2023; OpenAI, 2024b], based on multimodal inputs. Although current models have succeeded in addressing many downstream visual tasks, they struggle with specific challenges such as medical image segmentation due to the scarcity of relevant data, making it difficult to achieve high performance in these demanding areas. Our INT introduces negative mining, employing VLMs to iteratively refine the correct prompts to enhance segmentation performance in these challenging tasks.

Promptable Segmentation involves segmenting objects with user-provided inputs, such as points, boxes, or scribbles. Models like SAM [Kirillov et al., 2023], AV-SAM [Mo and Tian, 2023], GroundingSAM [Liu et al., 2023c], and SEEM [Zou et al., 2023b] extend this to multimodal inputs, including video and audio. However, these approaches often depend on manual prompts, which are prone to ambiguity and subjectivity, and are typically effective only for specific tasks. GenSAM [Hu et al., 2024a] introduces a manual-free setting, using a task-generic prompt for instance-specific segmentation across images without additional user inputs. It leverages VLMs to infer object names as prompts for SAM, but its lack of spatial information can result in inaccurate predictions in complex scenes. ProMaC [Hu et al., 2024b] introduced hallucination as a type of prior knowledge to leverage taskrelevant information as much as possible to aid in the generation of more accurate instance-specific prompts. However, since there are no ground truths, how to effectively evaluate the quality of instance-specific prompts and exclude the effects of erroneous prompts becomes an unresolved problem.

Prompt Engineering is a developing field that focuses on creating and refining prompts to improve the effectiveness of large language models (LLMs) for a variety of tasks, encompassing both language and vision modifications. In the language domain, zero-shot prompting [Wei et al., 2021] is employed to leverage LLMs' generalization capabilities for new tasks, although it can lead to inaccurate outcomes. Recent advancements include chain-of-thought prompting [Wei et al., 2022] and graph prompting [Liu et al., 2023d], which enhance complex reasoning skills. Other strategies such as generated knowledge prompting [Liu et al., 2021] and self-consistency [Wang et al., 2022] have also been implemented to boost prediction accuracy. For vision-related tasks, prompt tuning is the primary method, using vision-driven [Jia et al., 2022; Zhou et al., 2023], language-driven [Zhou et al., 2022; Ma et al., 2023], and vision-language driven approaches [Zang et al., 2022; Xing et al., 2022] to create vision prompts that enhance model performance. Despite some successes in multimodal prompt engineering works [Zhang et al., 2023b; Hu et al., 2024a; Hu et al., 2024b] in generating instance-specific prompts, these methods struggle to accurately identify correct prompts without annotations. Our INT approach effectively addresses this challenge by leveraging progressive negative mining.

3 Methodology

We present INT, a training-free cycle-generation method that segments multiple unknown classes of objects with only a single task-generic prompt. This innovative approach leverages negative mining to iteratively reduce the impact of potential erroneous predictions derived from task-generic prompts in an unlabelled setting. Specifically, for an image X ∈ RH×W×3 from a test set, INT utilizes a task-generic prompt Pg to generate a final segmentation mask M ∈ RH×W . This eliminates the need for separate supervision for each image, streamlining the process across datasets in the same task category. The prompt generator identifies multiple candidates for instance-specific prompts, which are evaluated by comparing the VLM outputs before and after image removal. The differences in the outputs are normalized and iteratively weighted, influencing the selection of instance-specific prompts in subsequent iterations. In each iteration, the candidate with the largest difference is chosen as the instance-specific prompt. This selected prompt guides the segmentation process, which further refines the generation of improved instance-specific prompts in the next iteration.

3.1 Instance-Specific Prompt Generation

Prompt Generation Using VLMs. For more accurate segmentation, the prompt utilizes VLMs to transform a generic prompt Pg into instance-specific prompts for each image. Specifically, given an image X and a query P , the VLM with parameters θ generates a response that captures task-relevant information. The image X provides essential visual context, aiding the model in formulating a relevant response y, which is derived auto-regressively from a probability distribution conditioned on P , X, and the previous tokens:

where yt denotes the token at time t and y<t represents the sequence of tokens generated up to time t-1. Despite the advanced capabilities of VLMs, task-relevant objects may blend into their background due to factors like texture, color, or position, making the instance-specific prompts they generate prone to inaccuracies. Inaccurate instance-specific prompts can directly lead to incorrect segmentation. However, without any labels, assessing whether these prompts meet the task requirements becomes challenging. However, VLM predictions are often driven by the unique features of the objects they detect. When such features are obscured, the model's predictions for the corresponding category significantly drop (see Fig. 2). Therefore, if task-related objects can be accurately located and effectively removed, comparing the VLM outputs before and after removal provides a reliable method for evaluating the quality of the instance-specific prompts.

Hallucination-driven Candidates Generation. To generate accurate instance-specific prompts, it is essential to identify as many candidates as possible with a reasonable level of confidence to ensure that the correct prompt is not overlooked. Inspired by ProMaC [Hu et al., 2024b], we divide the input image into patches of varying scales by cutting it horizontally, vertically, both, or leaving it uncut. Each patch is then processed individually by the VLM to generate preliminary instance-specific prompts. The visibility of task-relevant objects differs across patches, encouraging the VLM to leverage its prior knowledge to infer potential objects and their locations. This process helps identify candidate bounding boxes and object names by linking the visual data in each patch with the task context. The VLM processes each patch as follows:

where C k is the caption for the k-th image patch Xk , and Pg is the task-generic prompt. For bounding box predictions, the prompt PB guides the VLM: "This image pertains to the Pg detection task, output the bounding box of the Pg." This instructs the VLM to predict the bounding boxes Bk for objects related to the task within the patch. For object naming, the prompt PA states: "Output the name of the Pg and its environment in one word." This directs the VLM to predict the names of the task-related objects Akfore and their backgrounds Akback from each patch. Object names Akfore and bounding boxes Bk , collected from different patches, are compiled into candidate lists Ai and Bi, where i denotes the iteration.

Prompts Selection with Negative Mining. After generating the candidate lists, we evaluate which candidates are most likely correct. When a prediction is accurate, removing the corresponding object causes a significant change in the VLM's output for that category. Building on the previous section, where we ensured the ground truth was included, we now mask candidate-indicated areas and measure the change in the VLM's output scores. To achieve this, inspired by contrastive decoding, we compare the VLM softmax output of the original unprocessed image patch Xk with that of the masked patch X′k for each patch k. The category with the largest difference in output is selected as the final prediction for the corresponding patch. This approach ensures that the most significant and task-relevant object is accurately identified and used as the instance-specific prompt as follows:

here, X′k is processed by the image inpainting module, which uses the predicted mask mki from the last iteration in Sec. 3.2 as the inpainting mask IMki to guide the modification of the patch X′k . To ensure that X′k remains free of task-related objects Afore i , we employ a negative prompt Pn: "Afore i is not a Pg." For seamless Integration with the existing background Aback i , a positive prompt Pp is used: "Aback i , high quality, detailed, and well-Integrated with the original image." X′k is formalized as:

where Fin represents the inpainting module, implemented using Stable Diffusion. Using this method, we construct a set D(y) = {D(yk i )}Kk=1 , where K represents the number of patch. The accuracy of the predicted category is directly related to the magnitude of change in unique information about the correct category before and after masking. By evaluating these changes, we identify the most reasonable foreground Aui to serve as the instance-specific prompt for the i-th iteration. Specifically, we compare the variations in D(yk i ) across patches and select the candidate with the largest change in output as the final prediction:

This approach leverages the variations in softmax outputs to refine the instance-specific prompt Aui , ensuring they are closely aligned with task-relevant information and enhancing the overall prediction accuracy.

Progressive Negative Mining While this method effectively identifies correct categories when task-related objects are easily distinguishable, it struggles with more ambiguous samples. In such cases, incorrect categories may occasionally exhibit large score differences before and after masking during certain iterations but show little to no change in others. In contrast, the correct category consistently demonstrates stable and significant differences across all iterations. Building on this observation, we design a progressive negative mining technique to iteratively refine Aui . The idea is to accumulate and reinforce consistent patterns by iteratively multiplying scores from each iteration, reducing the influence of unstable changes caused by incorrect categories. To normalize the differences from each iteration for comparability, we use:

where Dnorm(yk i ) represents the normalized difference for category yk at iteration i, and K is the total number of patches. We then iteratively update the differences for the next iteration by applying cumulative multiplication:

where D(yk i+1 ) is the updated difference for category yk in iteration i +1, incorporating the influence of the current iteration's normalized differences. By iteratively applying negative mining, we amplify consistent patterns in the score differences associated with the correct category while suppressing the sporadic and unstable changes caused by incorrect categories, effectively enhancing the model's ability to accurately refine Aui over successive iterations.

3.2 Semantic Mask Generation

After obtaining the instance-specific prompt Aui , we aim to produce a mask that accurately delineates task-related objects without overlooking any targets. To achieve this, we first process Aui through Grounding DINO [Liu et al., 2023c] to gather all potential bounding boxes across various patches:

Subsequently, both Bk i and Aui are input into SAM:

where Spatial CLIP [Hu et al., 2024a] maps the text prompt Aui to regions within the image Xi as the visual prompts. These visual prompts, along with the corresponding bounding box Bk i are fed into SAMduring the i-th iteration to generate the mask mki . These processed masks, along with the generated instance-specific text prompts Aui , are subsequently input into CLIP to evaluate semantic similarity.

where the operation ⊙ retains only those parts of Xi covered by the predicted mask. s(mki ) quantifies the similarity between the masked image and Aui . Similarity scores from various patches are represented as Si = [s(m1i ), s(m2i ), . . . , s(mki )]. After normalizing these elements within Si, a normalized s(mki ) closer to 1 indicates a higher semantic alignment of mki with the instance-specific text prompt Aui . The weighted sum of the normalized s(mki ) and mki is then computed as follows:

where Mi is the resultant mask from the i-th iteration of X. This mask, generated using SAM's capabilities, ensures highly detailed mask production. Concurrently, this mask semantic alignment process guarantees that the output mask is consistent with the task's semantic requirements, overcoming the limitations of SAM's mask prediction.

The mask is then applied to the original image as a weighting factor to generate the next iteration image Xi for segmentation. This helps to exclude irrelevant regions and reduce Interference during segmentation:

where w is a hyperparameter set to 0.3. The mask generated in the last iteration serves to guide the prompt generator in the subsequent iteration, focusing on potential task-related regions, mitigating the impact of irrelevant hallucinations, and yielding more precise instance-specific prompts. These prompts, in return, aid the mask generator in producing improved masks. Through iterative cycles of prompt and mask generation, both elements enhance significantly. Ultimately, masks from different iterations are averaged, and the mask closest to this mean is selected as the final output:

Here, I represents the number of adaptation epochs, and Mi∗ is the definitive mask for image X.

4 Experiments

4.1 Experimental Setup

Baselines. Our study evaluates the ProMaC model's effectiveness in handling complex segmentation challenges such as Camouflaged Object Detection (COD), Medical Image Segmentation (MIS), and Transparent Object Detection (TOD), where traditional SAM models often fall short [Ji et al., 2023]. In our COD assessments, ProMaC is benchmarked against various weakly supervised segmentation methods [Kirillov et al., 2023; Zhang et al., 2020; Yu et al., 2021; Zhang et al., 2020; He et al., 2023c; He et al., 2023c; Hu et al., 2019; Hu et al., 2022; He et al., 2023a]. We employ two levels of supervision for comparison: scribble supervision, where the main structures of foreground and background are delineated during training, and point supervision, where distinct points are annotated for both. For taskgeneric prompt settings, we introduce a demanding scenario by relying solely on a task description as a generic prompt for segmentation. Here,INT incorporates LLaVA1.5 [Liu et al., 2023b] with SAM [Kirillov et al., 2023]. Further experimentation in MIS and PIS tasks aims to validate our method's superiority using task-generic prompts versus traditional techniques. We test combinations like GPT4V+SAM and LLaVA1.5+SAM to highlight the limitations of current VLMmodels in this context. Our INT is also evaluated with leading state-of-the-art promptable segmentation methods to underline its effectiveness. Our results are the average of three trials.

Metric. To assess performance in the first three tasks, we employ the following metrics: Mean Absolute Error (M), adaptive F-measure (Fβ) [Margolin et al., 2014], mean E-measure (Eϕ) [Fan et al., 2021b], and structure measure (Sα) [Fan et al., 2017]. A lower M or higher Fβ, Eϕ and Sα indicate superior performance.

PyTorch Implementation Details. For the VLM models, we employ LLaVA-1.5-13B for evaluation. For image processing, we use the CS-ViT-B/16 model pre-trained with CLIP, and for image inpainting module, we deploy stable-diffusion2-inpainting. The task-generic prompts for the COD task are specified as "camouflaged animal." The MIS task includes two sub-tasks: polyp image segmentation and skin lesion segmentation, each prompted by "polyp" and "skin lesion" respectively. All tasks undergo training-free test-time adaptation, iterating for four epochs, except for the polyp image segmentation task, which extends to six epochs. We utilize the ViT-H/16 model for promptable segmentation methods. Our experiments are conducted on a single NVIDIA A100 GPU, with further details provided in the appendix.

4.2 Results and Analysis

Results on COD Task. The COD tasks are designed to detect animals camouflaged within complex environments. We tested INT on three benchmark datasets: CHAMELEON [Skurowski et al., 2018], CAMO [Le et al., 2019], and COD10K [Fan et al., 2021a]. The CHAMELEON dataset includes 76 images collected from the Internet specifically for testing purposes. The CAMO dataset contains 1,250 images, divided into 1,000 training images and 250 testing images. The COD10K dataset comprises 3,040 training samples and 2,026 testing samples in total. As indicated in Table 1, we compared INT against other methods that apply different supervision levels. Generally, methods with scribble supervision outperformed those with point supervision. Notably, our INT, utilizing only a single generic task prompt, outshines all point-supervised methods and scribble-supervised methods on all three datasets in terms of all the metrics. This highlights

(a) Number of iteration I.

(b) Image preprocess strategy.

Table 3: Ablation study on CHAMELEON dataset.

INT's effectiveness. Additionally, our method consistently surpasses SAM, SAM-P, SAM-S, and CLIP Surgery+SAM, demonstrating that the enhancements provided by our approach are derived from its Intrinsic qualities and not merely from leveraging the advanced segmentation capabilities of SAM.

Results on MIS Tasks. The MIS task involves identifying pathological tissues in medical images. We utilized datasets such as CVC-ColonDB [Tajbakhsh et al., 2015] and Kvasir [Jha et al., 2020] for polyp image segmentation, and ISIC [Codella et al., 2019] for skin lesion segmentation. Comparisons from Table 2 show that we conducted experiments in the task-generic promptable segmentation setting. Since most VLMs have not been specifically trained on medical images, directly applying a VLM to medical tasks results in significantly lower performance compared to natural image tasks. In contrast, our proposed INT, through extensive candidate sample inference and negative mining, effectively explores task-relevant information and progressively filters out incorrect samples. This ensures the accuracy of the generated instance-specific prompts, greatly enhancing INT's segmentation performance on the MIS task.

Parameter Analysis. Tab. 3(a) investigates how various model metrics evolve over a prolonged number of iterations. all the metrics stabilize after just 5 iterations. Although the results continue to change, the variations remain slight. This indicates that our method converges rapidly and remains stable. It also underscores the importance of using iterationbased termination criteria to establish early stopping conditions. Tab. 3(b) examines the effects of different image pro-

(c) module ablation study.

cessing strategies. "Original" refers to the unmodified image, "Halve" splits the image horizontally or vertically into two parts, and "Quarters" divides it into four smaller patches. Testing results indicate that combining "Original", "Halve" and "Quarters" achieves the best balance between global and local information, avoiding excessive fragmentation.

Module Analysis. As shown in Tab. 3(c), we perform an ablation study on the COD and MIS tasks to assess the effects of different modules. "HCG" refers to hallucination-driven candidate generation, "PSNM" stands for prompt selection with negative mining, "PNM" is progressive negative mining, and "SMG" refers to the semantic mask generator. The first row shows that replacing HCG with just a single original image leads to reduced performance, underscoring the importance of using hallucinations to extract task-relevant information. In the second row, replacing prompt selection with negative mining using the VLM inference result yields worse performance than the full model, highlighting the significance of proper prompt selection. Removing progressive negative mining results in a significant drop in performance, indicating that the instance-specific prompts derived from the initial iteration may contain errors, and our approach effectively corrects these mistakes. The comparison between the last two rows emphasizes the importance of aligning the mask with task semantics. The consistently positive results across tasks confirm the robustness and effectiveness of our approach.

Visualization. Fig. 3 visually compares our method, nNamNim, with other approaches across three tasks and also showcases the contrastive images we generated. GenSAM performs well with clear objects but struggles in complex backgrounds. ProMaC produces solid segmentation results across various tasks, but it sometimes misidentifies instance-specific prompts in complex scenes, leading to segmentation results that are unrelated to the task (e.g., the first column in Fig. 3). In contrast, our method introduces a negative mining strategy that not only explores potential candidates to extract taskrelevant information from the image for better segmentation but also corrects misidentified instance-specific prompts from early iterations. This approach effectively improves performance. Additionally, our method can segment multiple taskrelated samples within the same image, something that previous methods could not achieve. It demonstrated the effectiveness of our approach.

5 Conclusion

In this paper, we introduced the Instance-specific Negative Mining for Promptable Segmentation (INT) approach. It leverages the difference in VLM outputs before and after masking as a metric for progressive negative mining. By employing progressive negative mining, INT infers reliable instance-specific prompts from a single task-generic prompt. This allows for effective segmentation of different targets within the same task across various images, even in the absence of annotations. Experiments conducted on six diverse datasets demonstrate the effectiveness of our INT.