VQCounter: Designing Visual Prompt Queue for Accurate Open-World Counting

1 Introduction

Object counting is an essential task with broad applications across agriculture, biomedicine, industry and so on. Historically, research has concentrated on class-specific counting, targeting categories such as human [Babu Sam et al., 2022; Abousamra et al., 2021; Li et al., 2018; Liang et al., 2022], animals [Arteta et al., 2016; Zhu et al., 2021; Sun et al., 2023; Jia et al., 2023], cells [Xie et al., 2018; Guo et al., 2019; Zheng et al., 2024], and vehicles [Mundhenk et al., 2016; Amato et al., 2019]. Recently, attention has shifted towards class-agnostic counting (CAC) [Lu et al., 2019], which empowers a model to count objects of arbitrary classes not encountered during training. Within the CAC paradigm, lowshot counting methods have gained considerable prominence, encompassing few-shot counting using visual exemplars [Liu et al., 2022; Shi et al., 2022; You et al., 2023], zero-shot counting based on textual descriptions [Xu et al., 2023; Ranjan and Nguyen, 2022; Amini-Naieni et al., 2023; Kang et al., 2024], and hybrid approaches that fuse both modalities [Amini-Naieni et al., 2024; Dai et al., 2024; Pelhan et al., 2024b; Pelhan et al., 2024a; Mondal et al., 2024]. Advancements in large-scale, pre-trained vision-language models (VLMs) have further propelled CAC's state-of-the-art performance. CountGD [Amini-Naieni et al., 2024], a leading method in both few-shot and zero-shot benchmarks, leverages the pre-trained vision-language model GroundingDINO [Liu et al., 2025], an open-world text-specific object detector. By integrating visual exemplar features with textual tokens, CountGD transforms GroundingDINO into an open-world counter, allowing users to specify objects for counting via text, visual exemplars, or a combination of both.

* Corresponding authors.

However, despite its exceptional performance, CountGD does not fully exploit GroundingDINO's potential, revealing avenues for enhancement. In particular, we identify four major limitations in CountGD. (1) Insufficient Diversity of Visual Prompts. CountGD heavily relies on the dataset for visual exemplars. For example, FSC-147 [Ranjan et al., 2021] provides only three visual exemplars per image, which inadequately captures intra-class variations such as posture, color, and size. (2) Simplified Label Matching. Most counting datasets offer only point annotations, preventing the calculation of IoUCost for Hungarian matching [Kuhn, 1955] between ground truth and predictions. As a result, CountGD omits this critical component, leading to diminished convergence efficiency and overall performance. (3) Monolithic Training Strategy. CountGD exclusively focuses on the branch that combines text and visual exemplars during training, neglecting both purely textual and purely visual prompts. This approach limits performance when uni-modal prompts are used and undermines the full potential of multi-modal fusion. (4) Inflated Counting Error Metrics. Metrics like Mean Absolute Error (MAE) fail to accurately reflect localization errors. This issue is amplified in CountGD, where adaptive cropping reduces counting errors by balancing the numerous false positives and false negatives.

To overcome these limitations, in this paper we present VQCounter (Visual prompt Queue-based Counter), a refined yet highly effective open-world counting framework. As depicted in Figure 2, VQCounter addresses the aforementioned challenges and integrates novel localization-aware metrics to rigorously evaluate localization errors. (1) Enhanced Visual Prompt Diversity. To harness GroundingDINO's capacity to generate precise bounding boxes from point annotations, we introduce a dynamic queue that stores detection boxes throughout training, thereby augmenting the sparse visual exemplars within the dataset. Drawing inspiration from the Least Frequently Used (LFU) algorithm [Lee et al., 2001], we implement a Most Frequently Used (MFU) queue management strategy. This technique prioritizes frequently used visual prompts for dequeuing while retaining less-utilized prompts for longer periods, thus bolstering the model's ability to accommodate intra-class variations. (2) Improved Label Matching with VoronoiCost. We introduce VoronoiCost, grounded in the Voronoi diagram [Aurenhammer and Klein, 2000], as a superior alternative to IoUCost for Hungarian matching. Unlike IoUCost, VoronoiCost is adept at handling point annotations, thereby enhancing both training efficiency and overall performance. (3) Modality Switching Training Strategy. To remedy the single-branch limitation, VQCounter adopts a tripartite training approach, incorporating three distinct prompt branches: text-only, visual exemplars-only, and a combined text and visual exemplars branch. By strengthening each modality independently, we enhance the efficacy of multi-modal prompt fusion. (4) Localization-aware Metrics: We devise novel metrics inspired by the Voronoi diagram, to assess localization errors in counting predictions. These metrics rely exclusively on ground truth data, ensuring a higher degree of objectivity compared to traditional methods.

We validate VQCounter through comprehensive experiments on two well-established datasets, namely FSC147 [Ranjan et al., 2021] and CARPK [Hsieh et al., 2017]. The results clearly illustrate that VQCounter substantially surpasses existing methods in both zero-shot and few-shot benchmarks, as depicted in Figure 1.

2 RelatedWork

2.1 Existing Counting Tasks

Object counting has been extensively explored, primarily focusing on class-specific counting across various scenarios, including human [Abousamra et al., 2021; Idrees et al., 2018; Lian et al., 2019; Song et al., 2021], cells [Xie et al., 2018; Guo et al., 2019; Zheng et al., 2024], animals [Arteta et al., 2016], and polyps [Zavrtanik et al., 2020]. While these methods achieve high accuracy for predefined categories, they often fail to count unseen classes during testing. In contrast, class-agnostic object counting approaches [Lu et al., 2019; Zhu et al., 2025; Liu et al., 2022; Xu et al., 2023] offer flexibility by accommodating arbitrary entities with or without prompts. Since the introduction of the FSC-147 benchmark [Ranjan et al., 2021], significant progress has been made in zero-shot [Xu et al., 2023; Ranjan and Nguyen, 2022; Amini-Naieni et al., 2023; Jiang et al., 2023] and fewshot [Lu et al., 2019; Pelhan et al., 2024b; Pelhan et al., 2024a] counting, enhancing performance to unprecedented levels. Additionally, the class-agnostic paradigm has spurred novel tasks such as Referring Expression Counting [Dai et al., 2024], Training-free Counting [Mondal et al., 2024], and Unified Textual-Visual Prompts Counting [Amini-Naieni et al., 2024], broadening the versatility of counting methodologies. In this paper, we adopt the class-agnostic framework, focusing on a textual-visual unified benchmark akin to CountGD [Amini-Naieni et al., 2024], and improve performance by addressing previously unreported key issues.

2.2 Counting with VLMs

The advent of pre-trained vision-language models has significantly enhanced counting task performance. Zero-shot counting was first introduced by ZSC [Xu et al., 2023], utilizing class names and CLIP's [Radford et al., 2021] text encoder to generate prototypes for interaction with image feature maps. Subsequent models like CLIP-Counter [Jiang et al., 2023] and VLCount [Kang et al., 2024] refine the integration between text and image embeddings. OminiCount [Mondal et al., 2024] leverages the Segment Anything Model (SAM) [Kirillov et al., 2023] alongside semantic and geometric priors for improved accuracy, while PseCo [Huang et al., 2024] extends CLIP and SAM through a multi-task framework for precise instance segmentation. Recent advancements using GroundingDINO [Liu et al., 2025], a state-of-the-art open-world detector, have set new benchmarks. VA-Count [Zhu et al., 2025] enhances exemplar selection with exemplar enhancement and noise suppression, and GroundingREC [Dai et al., 2024] introduces advanced feature fusion techniques and a fine-grained counting benchmark. CountGD [Amini-Naieni et al., 2024] further improves these methods by incorporating an image-text fusion module, enabling the model to process both textual and visual prompts effectively. Inspired by CountGD, our approach leverages GroundingDINO more efficiently, significantly increasing visual prompt diversity and enhancing class-agnostic counting performance.

2.3 CountingMetrics

Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) are prevalent in counting tasks due to their simplicity and computational efficiency. However, these metrics do not account for localization errors, as false positives and false negatives can cancel each other out, potentially misrepresenting model performance. To mitigate this issue, REC [Dai et al., 2024] employs a manually defined threshold combined with Hungarian matching to compute true positives, false positives, and false negatives. Additionally, Song et al. [Song et al., 2021] introduced the nAP metric, which utilizes a sophisticated sorting and filtering mechanism, while Ciampi et al. [Ciampi et al., 2024] proposed the Mosaic Test to estimate coarse localization errors by splicing images. In this work, we introduce novel localization-aware metrics based on the Voronoi diagram [Aurenhammer and Klein, 2000], which accurately quantifies localization errors with enhanced interpretability and independence from predictions.

3 Method

In this section, we first briefly review the general practice in turning the pre-trained VLMs, e.g., GroundingDINO [Liu et al., 2025], into an open-word counter. Then, we elaborate on the proposed VQCounter framework and its key components. After that, novel localization-aware metrics are presented.

3.1 Preliminaries

Review of Counting with VLMs

We briefly review counting with pre-trained vision-language models, focusing on CountGD [Amini-Naieni et al., 2024]

as a representative of recent state-of-the-art approaches. In these methods, the target object is specified by visual exemplars (e.g., bounding boxes B = {b1, . . . , bn}) or textual descriptions t. CAC aims to count entities of arbitrary classes, ˆ y = f(X,B, t), where X is the input image and ˆ y is the predicted count. CountGD consists of an Image Encoder (fθSwinT ), a Text Encoder (fθTT ), a Query Selection Module (Select), a Feature Enhancement Module (fφ), and a Cross-Modality Decoder (fψ). After extracting visual prompts Pvisual = RoIAlign(fθSwinT (X,B)) and textual prompts Ptextual = fθTT (t), the counting process is then formulated as follows:

where zv,t are the fused visual-textual features, zI are the image features, k is the number of selected image tokens, ˆY are the final similarity scores, thresholded by the confidence threshold σ and counted to estimate the object count ˆ y during inference.

Discovery

During the replication of CountGD, we discover a surprising phenomenon that proves highly beneficial for low-shot CAC tasks, which has not been previously reported. We find that even when trained solely with point annotations, the model retains the ability to generate precise bounding boxes. As shown in the upper right corner of Figure 2, the predicted bounding boxes are accurate, and can be utilized as supplementary visual prompts. Their effectiveness is fully demonstrated in the experiments. Consequently, visual exemplars will be expanded to B ∪ ˆB, where ˆB = {ˆb1, . . . ,ˆbm}, ˆbi denotes a pseudo bounding box.

Algorithm 1: MFU-Queue Algorithm

Data: capacity E, insert batch size n, access batch size m, incoming vectors Vin = {v1, . . . , vn} Result: MFU Queue Qi with frequencies fi(v) 1 Initialize Qi ← ∅ and fi(v) ← 0 for all v; 2 while training do 3 Insertion Phase; 4 foreach vnew ∈ Vin do 5 if |Qi| = E then 6 vremove ← argmaxv∈Qi fi(v); 7 Qi ← Qi \ {vremove}; 8 Qi ← Qi ∪ {vnew}; 9 fi(vnew) ← 0; 10 Access Phase; 11 Select m vectors Vaccess fromQi with probability P(v) = 1/(fi(v)+ϵ) ∑v′∈Qi 1/(fi(v′)+ϵ) ; 12 foreach v ∈ Vaccess do 13 fi(v) ← fi(v) + 1;

3.2 VQCounter

In this part, we start to introduce the proposed counting framework VQCounter, a Visual prompt Queue-based Counter. In VQCounter we propose four improvements, namely MFU-Queue, VoronoiCost, Modality Switching Training, and Localization-aware Metrics, to improve the algorithm from a global perspective. Each part will be elaborated on in detail as follows.

MFU-Queue

Based on the description in Section 3.1, the pseudo yet accurate bounding boxes produced by the model could significantly enrich the diversity of visual exemplars. As shown in the upper right corner of Figure 2, even in the first training epoch, the model can detect several precise bounding boxes, and as training progresses, the recall of the detected boxes will gradually increase (more visualizations in the Appendix). These boxes can serve as a supplement to visual examples to enrich the diversity of visual prompts. To store these additional visual prompts, we first construct FIFO (First in, First Out) queues Q = {Q1, . . . ,QK} ,Qi ∈ RE×d , where K denotes the number of classes in the training dataset and the capacity of each queue is set as a constant value E. In each training iteration, the visual prompts are drawn randomly from the ground truth boxes in the dataset and the prompt queues Q.

Although the baseline queue helps to improve the diversity and effectiveness of visual prompts, the strategy for prompt entry, exit and sampling can be further optimized. Inspired by the Least Frequently Used (LFU) algorithm [Lee et al., 2001], a famous cache algorithm for memory management within a computer, we propose MFU-Queue, a Most Frequently Used priority dequeue Queue. As its name suggests, MFU-Queue tracks how often each prompt is accessed. When the queue is full, the most frequently accessed prompt is removed first. Simultaneously, within the queue, we employ a probabilistic accessing strategy that prioritizes prompts with fewer prior selections. During training, the enqueue and dequeue operations and access to visual prompts are performed alternately. Therefore, MFU-Queue significantly enhances the richness of visual prompts and demonstrates greater robustness to variations in object size and background within the same category. The MFU-Queue training procedure is summarized in Algorithm 1.

VoronoiCost

MFU-Queue optimizes the algorithm's input, while VoronoiCost, discussed in this section, enhances the algorithm's key components. One of the key components in DETR-like detectors is Hungarian matching between N object queries and M ground truth (GT) boxes. Its goal is to seek a matching function σ that assigns each GT box to a predicted box, that is σ : {1, 2, . . . ,M} → {1, 2, . . . ,N}. Here, σ (i) outputs the index of the prediction matched to the i-th GT box. The Hungarian algorithm is utilized to find the optimal matching σ∗ that minimizes the total cost:

C ∈ RM×N is a cost matrix, where each element Ci,j represents the cost of matching the i-th ground truth box with the j-th predicted box. In DETR-like detectors, this cost typically combines classification cost, bounding box coordinates cost, and GIoU cost (α, β, and γ are coefficients):

Object counting datasets typically provide point annotations, where only the center point of each object is given, and the width and height are unknown. Consequently, in existing methods, Ccoord only considers point coordinates, and the γCGIoU marked with underline is discarded, leading to suboptimal performance and efficiency. In this paper, a novel VoronoiCost (denoted as CV oron) is proposed to make up for the lack of CGIoU when only point annotations are available in counting datasets.

VoronoiCost is constructed upon the Voronoi diagram [Aurenhammer and Klein, 2000], which is a partition of a plane into regions close to each of a given set of points. Each region corresponds to a point in the given point set, and the distance from any point within a region to its corresponding point is less than the distance to any other points.

Formally, given a set of GT points G = {g1, . . . , gM}, we first construct their Voronoi Polygons as:

where d (·) denotes euclidean distance. Then, given a set of predictions P = {p1, . . . , pN}, we can calculate CV oron as follows:

where I (·) is the indicator function, and csml and clag are constant values representing the matching cost when the predicted point p is within the Voronoi polygon area of the GT point g. Eq. (4) now is turned as:

TP and FN Compensate (MAE=0, LaMAE=6)

Figure 3: Comparison of MAE and LaMAE. (a) Predictions perfectly match the ground truth, resulting in both MAE and LaMAE being 0. (b) False positives and false negatives cancel each other out, yielding a MAE of 0 but a LaMAE of 6, indicating the presence of localization errors.

Experimental results show the CV oron boosts the performance and matching efficiency remarkably.

Modality Switching Training

In contrast to previous approaches that consider either unimodal or multi-modal prompts in isolation, we propose a more generalized framework known as Modality Switching Training (MST). The key insight behind this design is that improving the performance of any individual modal prompts can enhance the effectiveness of multi-modal fused prompts. Specifically, VQCounter employs a gating controller that regulates the modality of prompts during training. Formally, the equation in Eq. (1) is rewritten as follows:

where ˜P ∈ {Pvisual, Ptextual, [Pvisual, Ptextual]}, and ˜z ∈ {zv, zt, zv,t}. Meanwhile, the zv,t in the Eq.(2) will be replaced by ˜z too. Building upon the MST strategy, the model's performance is reliably ensured for any uni-modal prompts, which significantly contributes to enhancing its performance in multi-modal prompts.

Localization-aware Metrics

To evaluate the algorithm's capabilities with greater precision and objectivity, this part introduces the novel localizationaware metrics. The motivation aligns with previous studies [Dai et al., 2024; Song et al., 2021; Ciampi et al., 2024]. Common error metrics, such as Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Normalized Absolute Error (NAE) fail to capture localization errors, potentially yielding accurate counts but imprecise localization. Specifically, false positives (FPs) and false negatives (FNs) may cancel each other out, leading to inflated performance metrics.

Unlike previous methods, the proposed localization-aware metrics are also built on the Voronoi diagram and offer greater objectivity and accuracy without relying on the information of predicted results. Before computing the true positives (TPs), false positives (FPs), and false negatives (FNs), we first define the number of predicted points within each Voronoi Polygon sj as follows:

where V (gj), P, and range of i and j is the same as Eq. (5). Each Voronoi polygon containing at least one predicted point is counted as one TP, and any predicted points beyond the first are considered FPs. Each Voronoi polygon with no predicted points is counted as one FN, indicating a missed target.

Assume that the dataset size is B, and there are Mb GT points in the b-th image. Based on the counted TPs, FPs and FNs, we define a set of Localization-aware (La) metrics, including LaMAE, LaRMSE and LaNAE, as follows:

These localization-aware metrics not only quantify counting errors but also assess localization inaccuracies. Taking MAE and LaMAE as an example, Figure 3 illustrates two typical scenarios. Both Figure 3a and Figure 3b exhibit MAE of 0. In Figure 3a, the absence of false positives and false negatives results in a LaMAE of 0. Conversely, Figure 3b includes FPs and FNs, leading to a LaMAE of 6. LaMAE is clearly more reasonable and accurate than MAE.

4 Experiments

In this section, we perform comprehensive experiments on two widely used datasets, i.e. FSC-147 [Ranjan et al., 2021] and CARPK [Hsieh et al., 2017], to validate the efficacy of our proposed VQCounter. To ensure a fair comparison, we adopt the same experimental setup as previous studies: VQCounter is trained using the FSC-147 training set and subsequently evaluated on the FSC-147 test set as well as the CARPK dataset without any fine-tuning.

4.1 Datasets &Metrics

FSC-147 consists of 6,135 images spanning 89 training, 29 validation, and 29 test classes, with each set containing mutually exclusive classes. Every image is annotated with at least three visual exemplars. CARPK includes 989 training and 459 test images of parking lots captured by overhead drones. Each image is annotated with at least two bounding boxes. Adopting the approach from [Liu et al., 2022], we utilize these bounding boxes as visual exemplars and use the class name "car" for textual descriptions. More details about the two datasets and preprocessing are provided in the Appendix. For evaluation metrics, we employ both the standard metrics commonly used in counting tasks, namely MAE, RMSE, and NAE, as well as their localization-aware counterparts: LaMAE, LaRMSE, and LaNAE. These localizationaware metrics integrate counting and localization errors, offering a more comprehensive evaluation framework.

4.2 Implementation Details

For training, we use strategies similar to CountGD with the Adam optimizer and train for 30 epochs. Unlike CountGD, we fix fθSwinT and fθTT during training, and also stabilize the neck module, which adjusts feature dimensions, to ensure consistency in the MFU-Queue. Additionally, we adopt a non-parametric feature aggregation method from [Xu et al., 2024; Ren, 2015], replacing a single 1 × 1 convolution layer with multi-level feature fusion. During inference, we use simple cropping instead of adaptive cropping, which introduces a significant number of false positives and false negatives. All other settings are consistent with CountGD. See the Appendix for more details.

4.3 Comparison with Other SOTAMethods

In this section, we evaluate the proposed VQCounter against other state-of-the-art methods using the widely recognized CAC datasets, FSC-147 and CARPK. Following CountGD, VQCounter is assessed with various modal prompts, i.e. textual-only, visual-only, and textual-visual combinations. For visual prompt, previous works focused on the interactive (Visual-I) setting, in which the visual prompts come from the current test image. Inspired by [Jiang et al., 2025], we introduce an additional generic (Visual-G) setting, in which a fixed set of prompts are shared across the test set. As presented in Table 1 and 2, VQCounter significantly outperforms existing methods in nearly all metrics. On the FSC-147 benchmark, VQCounter outperforms the previous state-of-the-art method, CountGD, by reducing MAE by approximately 8% (from 7.1 to 6.47) on the validation set and by 28% (from 6.75 to 4.86) on the test set. To the best of our knowledge, this is the first method to achieve MAE values below 7 and 5 on the validation and test sets, respectively. Remarkably, in the zeroshot setting utilizing textual-only prompts, VQCounter surpasses CountGD by up to 50% (from 14.76 to 6.84), despite being trained exclusively on textual-based prompts, whereas our model employs a unified architecture. Furthermore, VQCounter demonstrates superior performance under the more stringent localization-aware metrics.

On the CARPK benchmark, we present results evaluated using both standard and localization-aware metrics (detailed in brackets). Consistently, VQCounter achieves state-of-theart performance across all three prompt categories, significantly outperforming existing approaches.

4.4 Comparison to CountGD inMore Aspects

For a detailed evaluation, we create a subset by selecting samples containing fewer than 900 targets for both validation and test sets. This subset fits well within the detection limit of the model, allowing for a single inference without the necessity of multi-cropping inference. Additionally, we investigate the impact of the two post-processing strategies, namely Adaptive Cropping and SAM TT-Norm proposed by [AminiNaieni et al., 2024]. Table 3 presents a comprehensive comparison between VQCounter and CountGD with (filled with gray) and without these post-processing strategies on both of the complete dataset and the newly formed subset.

Of the 156 metrics evaluated, VQCounter achieves superior results in 148 cases, with many metrics showing substantial improvements over CountGD. Similarly, in terms of the localization-aware metrics, VQCounter demonstrates overall superior performance compared to CountGD. In a few cases the performance decreases in the Visual-I setting. We hypothesize that MFU-Queue, while improving classes with significant intra-class variation, may slightly hurt classes with minor intra-class variation. However, this impact is negligible. Results in the Visual-G setting show that when visual prompts are sourced from different images, our method significantly outperforms CountGD, supporting our hypothesis.

4.5 Ablation Studies

We conduct some ablation studies to explore the proposed method systematically. For simplicity, we report results without the Adaptive Cropping and SAM TT-Norm [AminiNaieni et al., 2024] post-processing. As illustrated in Table 4, both VoronoiCost and the Baseline Queue independently improve the performance relative to the baseline method. Their combination yields additional performance gains. Furthermore, integrating the MFU strategy to optimize the Baseline Queue continues to augment the overall performance of the proposed method.

4.6 Visualization Analysis

To better understand the superiority of the localization-aware metrics and VoronoiCost, we visualize them in Figure 4. The difference between LaMAE and MAE is shown in Figure 4a and Figure 4b. LaMAE is stricter and can measure FPs and FNs. The Hungarian matching results between the predictions and the ground truth points in the first epoch are visualized in Figure 4c and Figure 4d. It can be seen that the label assignment converges faster after adding VoronoiCost. See more visualizations in the Appendix.

(c) with VoronoiCost

(d) without VoronoiCost

Figure 4: Visualizations of the localization-aware metrics (a-b) and Hungarian matching results (c-d). Each cyan dashed line indicate a pair of matched prediction and GT point.

5 Conclusion

In summary, we present VQCounter, a superior open-world counting framework that enhances diversity of visual prompts with dynamic queues, utilizes VoronoiCost for better matching between predictions and GT points, and adopts the modality switching training approach for better collaborative learning of different modalities. Additionally, we introduce the Voronoi diagram-based metrics to accurately measure the localization errors. By exploiting vision-language models more effectively, we achieve state-of-the-art results on FSC-147 and CARPK in both zero-shot and few-shot settings, setting a new benchmark in class-agnostic counting.