P2P: Transforming from Point Supervision to Explicit Visual Prompt for Object Detection and Segmentation

1 Introduction

The accurate detection and segmentation of objects in diverse scenarios stand as important tasks, serving as the foundation for various high-dimensional perception domains, like robotic perception, autonomous driving, etc.

In recent years, weakly supervised methods have gained widespread attention as an approach to reduce the dependence on annotations in fully supervised methods.

∗Corresponding author

Typically, the weak supervision includes image-level [Bilen and Vedaldi, 2016; Wan et al., 2018; Xu et al., 2022], point-level [Chen et al., 2022; Liao et al., 2023] and scribblelevel [Zhang et al., 2020], etc. Among them, object detection and instance segmentation with point-level supervision (shortened to PSOD and PSIS, respectively) have attracted growing attention recently thanks to the low annotation burden and distinctive location information of points.

However, the performance of existing point-supervised methods is still far from satisfactory, about 59% and 55% of that of fully supervised detection and segmentation baselines. This is contrary to the core of weakly supervised methods, i.e., to release the annotation burden while still achieving decent performance. PSOD is used as an example to illustrate the limitations of point-supervised vision tasks. The current PSOD methods generally follow the paradigm of Multiple Instance Learning (MIL) or Cascaded MIL (C-MIL) fashion. They first use proposal generation methods (e.g., Selective Search [Uijlings et al., 2013], MCG [Arbel´aez et al., 2014] or neighbor proposal sampling [Chen et al., 2022]) to construct proposal bags. After that, top-k proposals with high scores are selected from hundreds of independent proposals as the final result by MIL. Due to the lack of object size and edge information, proposal generation is more random and low-quality, and the results are limited by inaccurate proposals. Additionally, multiple independent proposals bring high randomness in the selection process, and it is easy to converge to the sub-optimal solution and focus on the discriminative part rather than the entirety of the object.

Thanks to the substantial progress in the visual foundation models, such as Segment Anything Model (SAM) [Kirillov et al., 2023], many downstream tasks have witnessed significant breakthroughs [Wang et al., 2023]. We hold the perspective that, rather than directly designing large foundation models, it is more meaningful to leverage them for specific tasks in resource-constrained situations. Notably, some efforts have been made to adapt SAM for weakly supervised tasks, e.g., weakly supervised semantic segmentation [Chen et al., 2023b]. However, these studies have employed SAM as a supplementary tool in a simplistic way and have not attempted to explore how to better guide SAM by enhancing the semantic representation capability of prompts. Compared with bounding boxes or masks, point annotations inherently possess limited semantic representation. When points are directly used as prompts for SAM, only 40% of the masks cover more than 70% of foreground pixels, significantly lower than the results (about 80%) obtained when using boxes as prompts. This highlights the crucial importance of semantic-explicit prompts.

In this paper, we propose a novel framework, referred to as Point-to-Prompt (P2P) for point-supervised detection and segmentation, by transforming the point supervision into visual prompt learning. We are the first to attempt to switch point-supervised tasks into visual prompt learning based on foundation model. An overview of the contrast between the existing PSOD framework and our framework is presented in Fig. 1. P2P comprises two integral processes: the SemanticExplicit Prompt Generation (SEPG) stage and the Prompt Guided Spatial Refinement (PGSR) stage. Specifically, the SEPG stage is designed to generate semantic explicit pseudo boxes as prompts under the guidance of semantic confidence. The PGSR stage further refines the target regions covered in the semantic-explicit prompts by leveraging the outstanding generalization ability of the foundation model SAM. It operates through an iterative process, involving multiple iterations between SEPG and PGSR, ultimately resulting in the generation of precise pseudo-labels. Utilizing SAM, our method can output precise pseudo-masks, so it can be applied as both a point-supervised detection and segmentation method that transforms points into accurate pseudo masks and boxes. Experiments on the challenging MS COCO 2017 and PASCAL VOC 2007 datasets are conducted to validate both the detection and the segmentation performance. Given point supervision, P2P further closes the gap with the fully supervised model and achieves 84% and 75% of the performance of fully supervised on the COCO dataset, respectively. Our main contributions are as follows:

• We design a novel Point-to-Prompt (P2P) method for point-supervised object detection and segmentation, which transforms the point supervision into prompting to predict precise pseudo-labels.
• We propose an iterative learning framework using visual foundation model to achieve a semantic-explicit output,
• including a semantic-explicit prompt generation stage and a prompt guided spatial refinement stage.
• Our P2P method achieves state-of-the-art performance on detection and segmentation, significantly narrows the performance gap with fully supervised methods, and provides new insights for point supervision tasks.

2 RelatedWork

Weakly Supervised Detection. For image-supervised detection, the difficulty lies in how to mine the location of each instance with only semantic information. Existing methods [Tang et al., 2018; Wan et al., 2018; Gao et al., 2019] generally build image-level proposal bags containing hundreds of proposals, and then mine instance-level supervision through MIL, unsupervised clustering, and contrastive learning. Due to the lack of location information, the performance is still poor for some complex datasets, such as COCO [Lin et al., 2014]. Point-supervised methods benefit from additional point-level annotations, providing coarse location information. [Ren et al., 2020b] designs a unified network compatible with various supervision forms. P2BNet [Chen et al., 2022] specifies that low-quality proposals limit the performance of this task and proposes to generate proposals through a neighbor sampling policy and designs a cascade MIL framework.

Weakly Supervised Instance Segmentation. Weakly supervised instance segmentation is mainly performed by estimating instance-level pseudo masks and refining the estimated masks by training a segmentation model. To obtain the pseudo mask for each instance with only point-level information, previous approaches have either used off-the-shelf proposal methods [Zhou et al., 2019] or generated instancelevel localization maps [Kim et al., 2022; Liao et al., 2023] by refining the attention maps of CAM [Zhou et al., 2016] or ViT [Dosovitskiy et al., 2020]. The performance of the current methods is significantly constrained by the quality of the attention map. [Liao et al., 2023] analyzes that only about 30% of the ViT attention maps can cover more than 50% of the foreground objects, greatly limiting the performance.

Prompt engineering and Foundation Models. Prompt engineering refers to the process of designing prompts that enable the foundation model to adapt and generalize to different tasks. Segment Anything Model (SAM) [Kirillov et al., 2023], as a representative prompt-based foundation model, is designed for image segmentation and has brought a new trend in solving other downstream tasks. The research community has been actively engaged in exploring and pushing the capability boundaries of SAM and applying it to various tasks, e.g., Remote Sensing [Chen et al., 2023a], medical image analysis [He et al., 2023], and weakly supervised semantic segmentation [Chen et al., 2023b]. Inspired by these approaches, we apply foundation model to point-supervised detection and segmentation and significantly facilitate the performance of point-supervised tasks.

3 Method

3.1 Overview

Problem. A point annotation p can be represented as p = (px, py, c), where (px, py) and c represent point location and object category, respectively. Point-supervised tasks aim to train a point-to-label regressor using point annotations to predict pseudo-labels. Subsequently, a task-related sub-network (e.g., a detector) is retrained in a fully-supervised manner for inference. Thus, the core of this task lies in designing an accurate point-to-label regressor, denoted as Φreg(·), which transforms the point annotation into precise pseudo annotations. To design a well-formed regressor, we introduce the P2P framework, proposing to first transform point supervision into explicit visual prompts and then obtain pseudolabels guided by these prompts.

Framework. We structure the P2P as an iterative refinement process of the Semantic-Explicit Prompt Generation (SEPG) stage and the Prompt Guided Spatial Refinement (PGSR) stage. The two stages respectively serve the roles of "point to prompt" and "prompt to pseudo-mask". In P2P, the point annotation p is first viewed as the prompt of SAM to generate an initial mask for each object. The outer rectangle of the mask is used as the proposal seed S0 . Taking S0 as input, P2P initiates the first round of iteration. While proposal seed may not be entirely accurate, it can still provide valuable prior information on object size. Then, taking the initial proposal seed S0 as input, the refined box is generated by two prompt refiners under semantic supervision. Compared to S0 , the refined box covers the main semantic part of the object and can be used as the subsequent prompt, referred to as P 1 . After, in PGSR, SAM is used to refine the spatial regions guided by the semantic-explicit prompt P 1 and generates the next round of proposal seed S1 . Improved proposal seed leads to better prompt, and better prompt, in turn, contributes to better proposal seed. The two modules iterate T times, ultimately yielding predicted pseudo-masks and pseudo-boxes through PGSR. The overall pipeline of P2P is depicted in Fig. 2.

3.2 Semantic-explicit Prompt Generation

We design a semantic prompt generator that takes semanticagnostic seeds as input and produces semantic-explicit prompts. Our approach adopts a group-then-individual strategy on two refiners, as illustrated in Fig. 2. Initially, feature maps are extracted by ResNet-50 [He et al., 2016] backbone. In Prompt Refiner I, we obtain a semantically accurate proposal group B∗ , followed by Prompt Refiner II, we further refine the proposal group to obtain refined proposal b∗ .

Seeds-based Group Sampling. Previous methods usually use neighbor sampling to build proposal bags that contain hundreds of individual proposals that usually suffer from low quality and lack of good priority. To mitigate that, we introduce a group sampling strategy based on the initial proposal seed S0 . For the first phase, we create the proposal bag B1 by progressively sampling n proposal groups {Bi}ni=1 for each instance based on the proposal seed, i.e., B1 = {Bi}ni=1 . These distinct proposal groups are generated by scaling the proposal seed at various scales. Each proposal group comprises m proposals with strong spatial correlation, denoted as Bi = {bi,j}mj=1 , where bi,j denotes the jth proposal of the ith group and m signifies the number of proposals in Bi. The motivation behind designing proposal groups is to reduce the solution space by selecting proposal group instead of individual proposals. For the second phase, we construct the proposal bag B2 by augmenting the group of proposals produced in the first phase. We adopt a "proposal jittering" strategy [Liao et al., 2022] to generate randomly jittered proposals in four orientations.

Proposal-to-Prompt Semantic Lifting. Different from classical MIL frameworks, we adopt a group-then-individual strategy, i.e., selecting a group of proposals with strong spatial correlation first and then further refining the proposals according to the group in the second phase. We employ a more stable feature prototype for computing group-based semantic distribution. The refiners in P2P comprise a classification head and an embedding head, which calculate classification scores and feature embeddings, respectively.

In Prompt Refiner I, the problem lies in identifying a semantic-accurate proposal group, which determines the direction of model optimization. The basic MIL head or classification head is commonly employed as the refiner, but the inherent instability in its training process easily lead the model towards sub-optimal solutions. To remedy the bias of predicted probabilities, we use a prototype representation to obtain stable group-based semantic distributions of proposals.

A memory buffer is established to keep a set of prototypes V = {Vc}Cc=1 for each category, which preserves the semantic-explicit features. These prototypes are updated using the selected high-quality embedding features from Prompt Refiner II in each iteration , via the Exponential Moving Average (EMA) algorithm. After that, the groupbased instance-level probability distribution Pins(Bi; yc) of each proposal group Bi can be measured by the similarity between the feature embeddings of the proposal groups and their corresponding semantic prototypes.

where Zi indicates the feature embedding of the proposal group Bi. It is calculated by averaging the feature embeddings of all the proposals in the group, as Zi = 1 |Bi | ∑j zi,j, where zi,j indicates jth proposal in Bi, and |Bi| is the number of proposals. sim(·, ·) denotes the cosine similarity metric, utilized to quantify the similarity between the embedding features and the semantic prototypes. Furthermore, we calculate the group-based semantic-level probability Psem(Bi; yc) for each proposal group Bi. It is computed by the score of the proposal group, as

where O(bi,j; yc) denotes the score of jth proposal in Bi. Finally, we define a group-based joint probability distribution that combines the semantic level and the instance level, termed P(Bi; yc) = Pins(Bi; yc) ∗ Psem(Bi; yc), which indicates the semantic probability of proposal group Bi for a given semantic label yc ∈ {y1, y2, ..., yC}.

In the learning procedure, based on the above definition, P(Bi; yc) is applied to the refinement process, the corresponding loss function of the first phase (termed L1) is defined as:

The proposal group that contains multiple semanticexplicit proposals with the highest semantic confidence is selected, termed B∗ .

Prompt Refiner II performs further proposal refinement as well as prototype update with a similar structure as the first phase, i.e., comprising a classification head and an embedding head. For further refinement, based on the proposal group B∗ , the proposal bag B2 is constructed as the input of this phase. The proposals in B2 maintain strong spatial correlation and have identified the main semantic regions. So in this phase, we only employ the general classification head with Sigmoid function to compute proposal score s(b; yc) and adopt the focal loss for further refinement. (More details are in the supplementary material.) The proposals with the top-k highest scores are weighted to obtain the final refined box b∗ .

For prototype update, we treat the proposal score of this phase as an indicator. For example, for the proposal b ∈ B2, the corresponding embedding feature and score are denoted as z and s, respectively. During each training iteration, the embedded features whose corresponding score s exceeds a certain threshold τ are selected, as

where vc denotes the local prototype of the current iteration for category c. And then the global semantic prototypes are updated with local prototypes via the EMA algorithm, as Vc = α ∗ Vc + (1 - α) ∗ vc, where Vc denotes the semantic prototype of category c, and α is momentum parameter and empirically set to be 0.99. Consequently, we obtain a set of prototypes V = {V1, V2, ..., VC} for all categories, which are continuously updated during the training process.

Discussion. The design of prototypes allows the two refiners to mutually reinforce each other. We use the scores from Prompt Refiner II as an indicator to update the semantic prototypes with high-quality embeddings. High-quality prototypes are then utilized in Prompt Refiner I to compute instance probabilities, yielding high-quality proposals. When these proposals are inputed into Prompt Refiner II, the quality of the semantic prototypes is further enhanced, leading to a mutually improving process.

3.3 Prompt Guided Spatial Refinement

In this stage, we leverage SAM to further refine the spatial regions of objects. We use the pre-trained weights of SAM, without modifying the original structures and fine-tuning its parameters. Since SAM does not have the ability of semantic understanding, the ability to get the desired output relies on the accuracy of the semantic prompts. A single point is semantic-ambiguous because there are semantic biases between part and the global of an object (e.g., clothes and person). In contrast, refined boxes generated in the SEPG stage cover the main semantic regions of objects and can be used as semantic explicit prompts to guide SAM in generating spatially refined regions.

SAM consists of three main components, a heavy image encoder (Φimg), a light mask decoder (Φmask), and a prompt encoder (Φpmt). In our approach, we input the predicted boxes and the original point annotations as prompts into the prompt encoder to extract prompt embeddings. We extract the image embeddings in advance with the image encoder, and then during model training, the prompt embeddings and the offthe-shelf image embeddings are fed into the lightweight mask decoder to get the refined maskM. The pre-extraction operation eliminates the computation of the heavyweight image encoder and greatly reduces the time and computation consumption. The overall process can be illustrated as follows:

where I indicates the original image, Fimg represents the image features encoded by Φimg, Fpmt denotes prompt features encoded by Φpmt, and {Pbox, Ppoint} indicates the box and point prompts, respectively. With semantic-explicit prompts, SAM can further refine the target region to cover more complete objects and generate accurate masks.

3.4 Training and Evaluation

We follow the standard pipeline [Chen et al., 2022] of point supervised tasks. During training, (i) a point-to-label regressor (P2P) is trained to obtain pseudo-labels (boxes or masks). (ii) After that, we retrain a task-related sub-network (e.g., a detector or segmentation network) with the full supervision of pseudo-labels. For evaluation, we only use the retrained sub-network to obtain detection or segmentation results and evaluate the detection or segmentation performance.

4 Experiment

4.1 Experiment Settings

Datasets. We evaluate the proposed method on two benchmarks: MS COCO 2017 [Lin et al., 2014] and Pascal VOC 2007 [Everingham et al., 2015]. MS COCO 2017 is a widely used large-scale dataset that contains 115K images in the train set and 5K images in val set, with 80 object categories collected in natural scenes. In Pascal VOC 2007, there are 2501 and 2510 images in training and validation sets, respectively, with 20 categories under common scenarios.

Evaluation Metrics. We use AP for MS COCO and VOC to measure the performance of detection and segmentation. And we report AP, AP50, AP75 for MS COCO and AP50 for VOC. The mIoU and Correct Localization (CL) are also calculated to directly measure the quality of the pseudo boxes. Specifically, mIoU is calculated by the mean IoU between predicted pseudo boxes and their corresponding ground-truth bounding boxes of all objects in the training set. CL, denoting the correct localization rate, is computed as the ratio of IoU between the prediction and the ground truth exceeding a certain threshold. We report CL at thresholds of 0.5, 0.7, and 0.9 (termed CL@0.5, CL@0.7, and CL@0.9, respectively) to assess the quality of pseudo boxes.

Implementation Details. For SEPG, it is trained with SGD [Robbins and Monro, 1951] optimizer with batch size 16. The learning rate is initialized as 0.02. SEPG is trained for 12 epochs at the first round of iteration then 6 epochs in each subsequent round of iteration. For PGSR, we use the pre-trained ViT-H version of SAM [Kirillov et al., 2023] and freeze its weights during the training stage. Before training starts, we pre-extract the image embeddings offline. So only the lightweight prompt encoder and mask decoder are involved in the PGSR, which greatly reduces the time and computational consumption of the training process.

4.2 Performance and Comparison

P2P serves as a pseudo-boxes and -masks generator. By retraining Faster RCNN (FR) and Mask RCNN (MR) with pseudo-boxes and -masks in a fully supervised manner, we report the detection and segmentation performance. Furthermore, we conduct comparative analyses by benchmarking our method against fully supervised methods and with various forms of weakly supervised methods.

Detection Performance. We conduct comparisons between our method and fully, image-, and point-supervised detection methods using COCO and VOC datasets, as presented in Tab. 1 and 2. On the COCO dataset, our method outperforms the SOTA P2BNet [Chen et al., 2022] by 9.5% (31.6% vs 22.1%) and 6.5% (53.8% vs 47.3%) in terms of AP and AP50, respectively, and achieves 84% of the fully supervised Faster RCNN (FR). On the VOC dataset, our method achieves 86% of the performance of fully supervised FR. From Tab. 1, we find that the image-supervised methods perform poorly on the challenging COCO dataset, achieving only about 36% of the fully-supervised baseline. This is notably lower than the performance of the point-supervised method, highlighting the favorable trade-off between labeling burden and performance offered by the point-supervised approach. With the SOTA DINO [Zhang et al., 2022] equipped with Swin-L [Liu et al., 2021] backbone as the retrained detector, P2P-DINO even exceeds the fully supervised baseline methods (e.g., Sparse RCNN), further demonstrating the potential of our method.

Segmentation Performance. Tab. 3 gives the performance of the segmentation methods on the COCO dataset with different forms of supervision. Our P2P-MR reports a significant performance improvement of 5.2% AP over the AttnShift [Liao et al., 2023] approach and outperforms the image-supervised BESTIE [Kim et al., 2022] (with HRNet48 [Wang et al., 2020] as the backbone) by a large margin. Additionally, it also outperforms SAM-MR (where SAM is used as a pseudo-masks generator) by 2.1% AP and 4.8% AP50. Furthermore, our method with only point supervision achieves nearly 80% of the performance exhibited by both baseline (Mask RCNN) and SOTA (Mask2Former [Cheng et al., 2022]) fully supervised methods.

4.3 Ablation Study

We conduct analyses of the impact of key components of our method on the COCO dataset.

Effect of each component in P2P. As shown in Tab. 4, P2BNet is used as the baseline, and the key components include (i) SEPG: The group sampling reduces the solution space, and more accurate pseudo-labels are obtained with the two semantic confidence-guided refiners. It contributes to a mIoU improvement by more than 3 points. (ii) PGSR: SAM leverages semantic-explicit prompts for spatial refinement, leading to a substantial enhancement in the quality of pseudo-labels, with an improvement of about 8 points. (iii) Employing an Iterative strategy where the two models mutually enhance each other, we observe a notable 12% improvement in performance compared to the baseline model.

Effect of different techniques in SEPG. We validate different techniques in SEPG and the results are presented in Tab. 5. The baseline model is designed by adopting the seedsbased sampling strategy followed by cascaded MIL refinement, referred to "SEPG-base", achieving 21.3 AP and 44.9 AP50. Then, we adopt group refinement, i.e., selecting groups first and then individual proposals. When using the MIL refinement head, referred to as "SEPG-G", AP reaches 21.5, as shown in the second row of Tab. 5. Further improvement is achieved with our designed semantic confidence refinement head, denoted as "SEPG-G-SC", resulting in a mIoU of 60.3 and an AP of 23.1, as shown in the fourth row of Table 5.

Effect of iterative training. We examine the impact of training iterations T. Tab. 6 reports the performance of P2P with different iteration numbers. We observe that the value of mIoU is consistently improved as T increases, reaching a peak when T = 3, followed by a stabilization around the peak. Notably, the highest performance surpasses that of T = 1 by nearly 2% across all metrics, showing the effectiveness of iterative training in elevating the quality of predictions. Some representative visualizations of different iterations are presented in Fig. 3.

Comparison of the quality of pseudo labels. As shown in Tab. 7, we conducted a more detailed comparison of the quality of pseudo-labels generated by P2BNet and singlepoint prompted SAM. Our approach achieved significant improvements, outperforming SAM by 11.45 and 18.73 points on mIoU and CL@0.5, respectively. This indicates that our method effectively enhances the quality of pseudo labels.

Comparison of memory cost. Tab. 8 illustrates the comparison of memory consumption between P2BNet and our P2P in the training stage. Thanks to our pre-extraction operation, even with the large foundation model, the number of parameters in the actual training process is still comparable, just about 10M higher than P2BNet.

5 Conclusion

In this paper, we introduce a point-supervised object detection and segmentation framework, called P2P, which transforms weak point-level annotations into explicit visual prompts, and guides the foundation model to produce the desired output by improving the semantic representation of the prompts. P2P performs as an iterative procedure that includes two stages: SEPG and PGSR. The SEPG performs semantic confidence lifting of the proposals through group sampling and semantic prototypes. The PGSR stage leverages SAM to output refined masks, which in turn are transformed into new proposal seeds. Finally, accurate pseudo masks and boxes are obtained in the iteration of SEPG and PGSR. Extensive experiments validate the effectiveness of our method.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 62372382 and Grant 62306239.