Learning Spatial Similarity Distribution for Few-shot Object Counting

1 Introduction

Visual object counting aims at counting how many times a certain object occurs in the query image, which has received growing attention in the past years. Existing methods often focus on specific domains, such as crowd counting [Shu et al., 2022; Wang et al., 2020; Abousamra et al., 2021], animal counting [Arteta et al., 2016], or car counting [Hsieh et al., 2017]. These methods typically rely on large amounts of data to train accurate counting models. Furthermore, they are limited to counting objects of specific categories and cannot generalize well to novel categories.

∗Corresponding author.

To overcome these limitations, a recent approach called Few-shot Object Counting (FSC) has been introduced and gained great attention with the emergence of a dataset [Ranjan et al., 2021]. FSC addresses the challenge of counting objects from arbitrary categories using only a few exemplars. This enables the model to generalize to unseen categories, offering potential for application in various scene categories beyond those encountered during training. By leveraging few exemplars, FSC provides a more flexible and adaptable solution for object counting tasks.

As shown in Fig.1, existing few-shot object counting methods typically follow a general workflow. They first calculate the similarity between query and exemplar features, and then directly regress the similarity matrix or enhance the query features using the similarity matrix and exemplar features before regression. In terms of similarity computation, some methods, as demonstrated in [Ranjan et al., 2021; Yang et al., 2021; You et al., 2023; Ðuki´c et al., 2023], employ exemplar features as fixed kernels to perform convolution with query feature. However, in this approach, the distribution of kernel features remains fixed during convolution matching, limiting its adaptability to different sizes and shapes of object features in the query. Another approach, such as the one used in [Shi et al., 2022; Lin et al., 2022; Liu et al., 2022], involves pooling exemplar features to obtain 1 × 1 feature prototypes, followed by cosine similarity computation between feature vector of each position in the query and these prototypes. This method disregards the distribution information of query and exemplar features, and counting performance becomes dependent on the performance of previous feature extraction and self-attention mechanisms.

To accurately locate the center of an object and generate an appropriate density distribution, we leverage distinct similarity distribution characteristics between each parts, such as object centers, edges and background, when compared to exemplars. Explicitly, as shown in Fig.2, the similarity distribution of the object center in exemplars gradually diminishes from the central position towards the surrounding regions, while the similarity distribution at the edges exhibits variations across different locations. On the other hand, the background demonstrates generally lower similarity values across all positions except the background area. Taking advantage of these patterns, we propose a novel method that tries to preserve the spatial structure of exemplars during similarity computation, and name it as learning Spatial Similarity Distribution (SSD). Concretely, this method yields a 4D similarity tensor, which allows for flexible extraction of point-to-point similarity distribution information between query and exemplar features using convolution operations in the 4D space. The features obtained through convolution enable precise calculation of density values for each position in the query during regression. In addition, we introduce a Feature Cross Enhancement (FCE) module for query and exemplar features. This employs the similarity matrices as weights to mutually enhance the features, aiming to achieve higher matching accuracy for objects belonging to the given category.

We conduct comprehensive experiments on two renowned public benchmark datasets, i.e., FSC-147 [Ranjan et al.,

2021] and CARPK [Hsieh et al., 2017]. The results clearly illustrate that our approach surpasses the performance of current state-of-the-art methods. Our contributions can be summarized as follows:

• We design a model based on learning the 4D spatial similarity distribution between query and exemplar features in Similarity Learning Module (SLM). This model is capable of obtaining accurate counting results after comprehensive integration of similarity distribution information among point pairs and their surroundings.
• Before calculating the similarity between query and exemplar features, we introduce a Feature Cross Enhancement (FCE) module, which enhances the interaction between them, reducing the distance between the target objects and exemplar features to achieve better matching performance.
• Extensive experiments on large-scale counting benchmarks, such as FSC-147 and CARPK, are conducted and the results demonstrate that our method outperforms the state-of-the-art approaches.

2 RelatedWork

2.1 Class-Specific Object Counting

Class-specific object counting focuses on counting a specific class of objects, such as crowd [Stewart et al., 2016; Liang et al., 2023; Lin and Chan, 2023; Du et al., 2023], animals [Arteta et al., 2016], or cars [Hsieh et al., 2017]. In related methods, the class information can be incorporated into the feature extraction process without additional classification steps. Existing methods can be broadly categorized into detection-based and regression-based approaches.

Detection-based methods detect the positions of objects in an image to perform counting. However, counting accuracy in these methods relies heavily on the performance of the detection process, which introduces errors. This limits the effectiveness of counting tasks in scenarios with densely packed objects. To address this issue, regression-based methods have been proposed to generate a density map, where the sum of the density values represents the predicted object count.

Classic detection-based methods, for example, [Stewart et al., 2016] propose a model that decodes an image into a set of people detections, generating distinct detection hypotheses directly from the input image. On the other hand, recent research in regression-based methods, such as [Cheng et al., 2022], utilizes locally connected multivariate Gaussian kernels as replacements for convolution filters. Moreover, a recent work [Liang et al., 2023] proposes knowledge transfer from a vision-language pre-trained model (CLIP) to unsupervised crowd counting tasks, eliminating the need for density map annotation.

2.2 Few-shot Object Counting

In recent years, few-shot object counting (FSC) has gained significant attention and witnessed a surge of interest. FSC aims to accurately count objects in an image by leveraging only a few exemplars as references. This ability to adapt to unseen categories during the testing phase is a key advantage of FSC.

Several noteworthy methods have been proposed for FSC. GMN [Lu et al., 2019] concatenates support features and query features together, and regresses a predicted density map based on this concatenation. In contrast, FamNet [Ranjan et al., 2021] convolves the query image with exemplars used as convolutional kernels, generating multiple similarity maps that provide insights into the comparison results between the query and exemplars. A predicted density map is then regressed from these similarity maps. Another approach, BMNet [Shi et al., 2022], employs global pooling to transform exemplars into prototypes, and replaces fixed inner product operations with a learnable bilinear similarity metric for comparing exemplar prototypes with query image features. CounTR [Liu et al., 2022] introduced a transformerbased architecture for extracting image features and utilized cross-attention modules for effective feature matching. Recently, LOCA[Ðuki´c et al., 2023] is proposed and considers the exemplar shape and appearance properties separately and iteratively adapts these into object prototypes by a new object prototype extraction (OPE) module considering the imagewide features.

2.3 Generalized Loss

Generalized loss function [Wan et al., 2021] is proposed for learning density maps for crowd counting and localization, which is based on unbalanced optimal transport. And [Wan et al., 2021] prove that both L2 loss and Bayesian loss [Ma et al., 2019] are special cases of the generalized loss. The approach proposed in [Lin et al., 2022] also utilizes this loss function and introduces a scale-sensitive generalized loss that applies different loss computation methods to object categories of different scales.

3 Methodology

3.1 Problem Setting

In few-shot object counting, the dataset is split into base classes Cbase and novel classes Cnovel, where Cbase and Cnovel do not overlap. The remarkable generalization capability of Few-shot Object Counting (FSC) lies in its ability to achieve high performance on the val set and test set, even for categories Cnovel that have not been encountered during training on Cbase. FSC addresses the task of counting the number of objects of interest present in a query image X ∈ R3×H×W , with the assistance of K exemplars Z.

3.2 Overall Architecture

As shown in Fig.3, our entire framework follows the following steps: (1) Feature extraction, (2) Feature Cross Enhancement (FCE), (3) Similarity pyramid calculation, (4) Similarity learning and (5) Regression decoder. Initially, the ResNet-50 [He et al., 2016] feature extractor is used to extract features from the image X ∈ R3×H×W , with the option of weight freezing (no fine-tuning is performed), and generated pyramid features {Fql }Ll=1 , where each level feature Fql ∈ RCLp ×HLp ×WLp (l ∈ Lp). Among all levels, several adjacent levels have features with the same spatial dimensions. All these levels together form a large layer Lp (p = 1, 2, · · ·, P). For each level in the feature pyramid, K exemplar features Fsl ∈ RK×CLp ×h×w are extracted using the RoIAlign method [He et al., 2017]. CLp represents the feature channel dimension of the respective layer, while HLp ×WLp and h ×w denote the spatial dimensions of the query feature and exemplar features at that layer. Here we maintain all exemplar features at a uniform size h × w.

The K exemplar features are input into the FCE module, along with the query feature at the same level, resulting in enhanced features Eql and Esl with the same dimensions as Fql and Fsl . We partition the K exemplars, with each exemplar corresponding to a set of feature pyramid combinations {Eql ,Esl,k }L l=1 , where Esl,k ∈ RCLp ×h×w . For pyramid feature set of each exemplar, we perform cosine multiplication on each feature pair to generate the similarity matrix:

Here, xq and xs denote 2-dimensional spatial positions of query feature map Eql and exemplar feature map Esl,k , respectively. '·' denotes vector dot product. For each similarity tensor, Sl,k ∈ RHLp ×WLp ×h×w .

As shown in Fig. 4, we concatenate the similarity matrices of the same large layer Lp and partition {Sl,k}Ll=1 accordingly, transforming it into {{Sl,k}l∈Lp }P p=1 and set it as {SLp,k }Pp=1 . And each tensor in it is SLp,k ∈ R|Lp|×HLp ×WLp ×h×w , where |Lp| is the number of pyramid levels. They are then fed into the Similarity Learning Module (SLM) to produce a learned and fused feature map Mk. Finally, Mk is input into the regression decoder module to obtain the density map Pk ∈ R1×H×W . The K sets of feature pyramids correspond to the generation of K density maps. The final predicted density map, denoted as P, is obtained by taking the mean of these K maps:

3.3 Feature Cross Enhancement

The distribution of object features within the query features of the same category is often uneven. Directly matching and counting using the original features can result in varying density values for each object. To address this issue, we propose a Feature Cross Enhancement (FCE) module that aims to bring the object features within the query closer to the exemplar features while also facilitate the exemplar features to be closer to the center position of all object features. By enhancing the proximity of the object features specific to a certain category, the model is able to generate more accurate density values.

In the FCE module, the input features Fql and Fsl are jointly transformed into Vq ∈ RCe p ×HLp ×WLp and Vs ∈ RCe p ×K×h×w through a convolutional layer. They are then individually passed through other two convolutional layers, with Fql being transformed into Q and Fsl into K, which are the same dimensions as Vq and Vs . Multiplying the transpose of Q and K matrices results in the attention matrix Al:

Then we utilize Al to separately enhance Fql and Fsl :

Here, MLP (·) is a multi-layer perceptron consisting of fully connected layers and activation functions, and used to map the channel vector into a channel-wise feature space of similarity relation. Trans (·) represents the convolutional layer that transforms channel C e p into the original channel CLp , and ⊙ denotes element-wise multiplication.

3.4 Similarity LearningModule

4D convolution. Several existing works [Rocco et al., 2018; Yang and Ramanan, 2019; Min et al., 2021] have proposed various implementations of 4D convolutions. In our framework, we employ the center-pivot 4D convolution from [Min et al., 2021] which sparsifies a significant portion of unimportant weights and computations. This method focuses solely on the information associated with the convolution center, reducing computational overhead while maintaining effectiveness. With 4D convolutions, tensors are fused for each 4D position based on convolution kernel weights, integrating information from the vicinity in 4D space and transforming the vector at that position into the corresponding output dimension.

For the input set of similarity tensors {SLp }Pp=1 (here we omit the exemplar subscript k), each tensor is fed into its corresponding 4D convolutional module:

where f eLp (·) is an encoding module composed of multiple 4D convolutional layers, group normalization [Wu and He, 2018], and ReLU activation function. The large strides of the 4D convolution compresses the spatial dimensions h × w of the exemplar spatial structure to h′ × w′ , while embedding the dimensions of all similarity tensors from |Lp| into Cout.

Next, starting from the apex of pyramid {S′Lp }P p=1 , we proceed to fuse each subsequent layer downwards. For instance, the tensor S′LP ∈ RCout×HLP ×WLP ×h′×w′ is upsampled on its dimensions HLP ×WLP to match the corresponding dimensions HLP-1 ×WLP-1 of the layer below. It is then added to the respective tensor S′LP-1 of the layer below and passed through a fusion module based on 4D convolution:

The structure of function f Mix Lp-1 is identical to that of function f eLp , with the difference being that the stride of f Mix Lp-1 is set to 1, which does not alter the spatial dimensions of the tensor. And the input and output dimensions in f Mix Lp-1 are all set to Cout .

SMix LP-1 is fused with the tensor S′LP-2 in a similar manner, iteratively continuing the fusion process with each subsequent layer until reaching the bottom layer of the pyramid S′L1 . Consequently, we obtain the final fused tensor SMix L1 ∈ RCout×HL1 ×WL1 ×h′×w′ . By calculating the mean along the last two dimensions, we derive the fused feature M ∈ RCout×HL1 ×WL1 .

3.5 Regression Decoder

The decoder module used for regression consists of multiple component modules composed of convolutional layers, ReLU activation layers, and upsampling layers. With each component module, the size of feature M is increased to twice until reaching the size of the input image H×W. Subsequently, it passes through a 1 × 1 convolutional layer and a ReLU activation layer. The output is the predicted density map.

3.6 Generalized Loss

In previous object counting tasks, ground truth density maps are generated by convolving dot labels with fixed Gaussian kernels. The MSE loss function is then employed for supervised training of the predicted density map. In a recent study [Wan et al., 2021], a generalized loss function was proposed that directly measures the distance between the predicted density map and the dot labels. This loss function is based on entropic-regularized unbalanced optimal transport cost.

We represent the predicted results as A = {(ai, xi)}ni=1 , where ai denotes the predicted density value at pixel xi ∈ R2 . Here, n represents the total number of pixels. Then we denote the predicted density map as a = [ai]i . On the other hand, the ground truth dot label is denoted as B = {(bj, yj)}mj=1 , where yj indicates the location of the j-th annotation and bj represents the number of objects represented by that annotation. In general, it is assumed that b = [bj]j = 1m. The whole loss function can be defined as:

where C is the transport cost matrix, with Cij representing the cost of moving the predicted density at xi to the ground truth dot annotation yj. D is the transport matrix that assigns fractional weights to associate each location xi in A with its corresponding yj in B for cost calculation. The optimal transport cost is obtained by minimizing the loss over D. H (D) = - ∑ij DijlogDij is the entropic regularization. The intermediate density map representation ˆa = D1m is constructed from the ground truth annotations, while ˆb = DT 1n is the reconstruction of the ground truth dot annotations.

3.7 Dynamic Image Scale

Certain sample images may contain objects that are small sizes or densely distributed, leading to challenges in effectively distinguishing between individual objects. This results in overlapping density within the predicted density map, thereby impacting model performance. To address this issue, we dynamically resize the input images based on the size of exemplar boxes before entering the backbone. This resizing is performed proportionally to the dimensions of the exemplar boxes, allowing the model to better recognize samples containing smaller objects. For an input image X, we compute the average length and width of K exemplar boxes B ∈ RK×2 :

If min ( ¯B) is below a threshold γ , we calculate the scale of image expansion:

where both γ and η are hyperparameters to be tuned. Finally, the image size and exemplar boxes B are simultaneously expanded by the determined scale value before being input into the model.

4 Experiments

4.1 Datasets andMetrics

Datasets. FSC-147 is a comprehensive multi-class few-shot object counting dataset. It comprises a total of 6,135 images covering 89 distinct object categories. The images in the dataset exhibit significant variations in terms of object counts, ranging from as low as 7 to as high as 3,731 objects, with an average count of 56 per image. Notably, each image in the dataset is accompanied by three or four exemplar images that are annotated with bounding boxes. To facilitate experimentation, the dataset is further divided into training, validation, and testing subsets, with each subset containing 29 non-overlapping object categories.

CARPK is a class-specific car counting dataset, which consists of 1448 images of parking lots from a bird's view. These images are captured from four different parking lots, encompassing various scenes. The training set comprises three scenes, while a separate scene is designated for test.

Metrics. We employ Mean Average Error (MAE) and Root Mean Squared Error (RMSE) as performance metrics for evaluating the SSD method, as these metrics are widely utilized in counting tasks.

where N is the number of all the query images, C i and C ipred are the ground truth and the predicted number of objects for i-th image respectively.

4.2 Implementation Details

Architecture Details. Our approach involves resizing the input image initially to 384 × 576. Then the image is dynamically resized to the suitable scale, followed by the application of the pre-trained ResNet-50 backbone, utilizing features extracted from the final three layers. The number of features at each layer, denoted as |Lp|, is 4, 6, and 3, with corresponding feature channel dimensions of 512, 1024, and 2048, respectively. The size of exemplar features extracted from each layer is uniformly resized to 16 × 16. In the FCE module, the embedded channel dimension is set to half of dimension of the input feature. In the Similarity Learning module, the

4D convolution module consists of three component modules with output dimensions of 32, 128, and 256, respectively. To fuse the three layers of similarity tensors, two fusion modules are required, each containing three component modules, and all the output dimensions are 256. γ and η in DIS method are set to 32 and 12.

Training Details. We apply AdamW [Loshchilov and Hutter, 2017] as the optimizer with a learning rate of 1×10-4 and the learning rate decays with a rate of 0.95 after each epoch. The batch size is 4 and the model is trained for 100 epochs.

4.3 Comparison with State of the Art

We evaluate the proposed SSD on the FSC-147 dataset with several state-of-the-art methods. the results are summarized in Tab.1. We conduct both 3-shot and 1-shot experiments on the dataset. SSD consistently outperforms existing methods in terms of performance.

In the 3-shot scenario, even compared to the recent stateof-the-art method LOCA [Ðuki´c et al., 2023], SSD demonstrates superior performance on the val set with a 5.0% improvement in MAE and an 8.7% improvement in RMSE. Notably, SSD also exhibits better performance on the test set with a 11.2% improvement in MAE.

In the 1-shot scenario, SSD surpasses all previous state-ofthe-art methods. This underscores the minimal dependence of SSD on reference samples, showcasing its robust adaptability to scenarios with limited available data.

Qualitative Results. In Fig.5, we visualize and compare the predicted density maps of BMNet, LOCA, and SSD. The results demonstrate that SSD has higher accuracy compared to the other two methods.

4.4 Cross-dataset Generalization

Following [Ranjan et al., 2021], we validate the generalizability of SSD on the CARPK dataset. The model is trained on the FSC-147 dataset and then tested on the CARPK dataset, with the car category samples excluded during training. During testing, we randomly select twelve annotations from the CARPK dataset as exemplars to count cars in images. The experimental results is presented in Tab.2. SSD

outperforms three other methods, achieving an improvement of 3.9% in MAE and 2.9% in RMSE compared to the most recent state-of-the-art method LOCA.

4.5 Ablation Study

We design a series of experiments to validate the individual contributions of the FCE module, generalized loss, and dynamic image scale on the performance improvement. In the absence of the FCE module, the model directly computes the similatity between Fql and Fsl . When excluding the generalized loss, we replace it with the more commonly used MSE loss. Each component undergoes four sets of comparative experiments with and without that component.

FCEmodule. Analysis of the four sets of experiments involving the FCE module reveals a consistent improvement in model performance. The addition of the FCE module results in a performance boost ranging from 2% to 19% in MAE and up to 14% in RMSE. This indicates that FCE module significantly enhances the ability of model to recognize objects within a given category by minimizing the distance between individual object features and exemplar features, leading to improved accuracy and uniformity in similarity and final density predictions across object positions.

Generalization loss. The contribution of generalization loss is pronounced to performance improvement. The four sets of comparative experiments show performance gains ranging from 10% to 27% in MAE and 1% to 30% in RMSE. The substantial improvement attributed to the generalization loss is due to the more precise recognition capabilities compared to MSELoss. By measuring point-to-point distance loss between predicted values and ground true labels, the generalized loss effectively guides the model to accurately locate object center positions.

(a) Channel ratio in FCE mod-

ule

Figure 6: Ablation study of channel ratio in FCE module and number of component modules in fusion module. The vertical coordinates are the values of MAE on the val set.

1 1 Dynamic image scale. The utilization of dynamic image scale also significantly improves model performance, particularly for dense samples. Expanding image scales proves effective in distinguishing between individual objects and counting them separately. Application of this method results in performance improvements ranging from 15% to 35% in MAE and 14% to 43% in RMSE.

Channel ratio in the FCE module. The query features and example features are embedded into another channel before they enhance each other. We conduct a series of experiments to determine the optimal ratio of the embedded channel length to the original channel length, setting various ratios at 18 , 14 , 12 , and 1. We then train the model to achieve the best performance and present the experimental results in Fig.6 (a). It is observed that as the ratio increases, the model performance peaks at a ratio of 12 and then begins to deteriorate.

Number of component modules in fusion module. The fusion module used to integrate tensors from different levels of the similarity pyramid is composed of several component modules. We set the number of component modules to range from 1 to 5 and conduct experiments on the FSC-147 dataset, with the results displayed in Fig.6 (b). The line graph in the figure indicates that as the number of component modules increases from 1 to 3, the performance of the model gradually improves, peaking at 3, and then begins to decline. This decline could be attributed to an increase in the number of model parameters due to more component modules, leading to overfitting and negatively affecting model performance.

5 Conclusion

We propose a novel few-shot object counting method, SSD, which leverages a point-to-point 4D space to learn the spatial similarity distribution between query and exemplar features. In contrast to existing methods, we exploit the distribution information of similarity, enabling accurate identification of the position and precise prediction of the count for objects of arbitrary classes. Additionally, we introduce a Feature Cross Enhancement (FCE) module that enhances the interaction between query and exemplar features, reducing the feature distance within the same class for improved matching. Experimental results on datasets such as FSC-147 and CARPK demonstrate that SSD outperforms state-of-theart methods.

Acknowledgments

This work was supported in part by National Natural Science Foundation of China under the Grants 62371235 and 62072246, and in part by Key Research and Development Plan of Jiangsu Province (Industry Foresight and Key Core Technology Project) under the Grant BE2023008-2.