CrowdFormer: An Overlap Patching Vision Transformer for Top-Down Crowd Counting

1 Introduction

Crowd counting is an essential topic in a variety of applications such as public safety, activity recognition, and transportation management, aiming to estimate the number of people in a crowd image. Current methods[Wan et al., 2020; Jia et al., 2021; Boyu et al., 2020] obtained excellent progress by utilizing the CNNs to regress the corresponding density maps of the input images, where the summed value in a density map indicates the total counting of people in the crowd image. The density map is an intermediate representation of the crowd, which groundtruth is typically generated from the original image by placing a density kernel on each person's dot annotation.

Due to the perspective phenomenon in real crowd scenes, there is a large-scale variation on different targets, which necessitates a crowd counting method that can deal with various scales. However, the current density map based crowd counting methods remain the following drawbacks: 1) Due to the limited and fixed receptive fields of pure CNNs, which cannot fit a wider range of continuous scale variation of targets, their density map estimators typically only deal with discrete scales. 2) For an efficient counting context information extraction, the dense or far-focus crowd scenarios are better suitable to a small density kernel, whereas sparse or closefocus crowd scenarios adapt to a large-scaled kernel. However, the groundtruth of density maps are typically generated by placing a hand-crafted density kernel on the corresponding dot annotations, which cannot fit to the scale change of the targets and scenarios. Based on the preceding observations, it is preferable to exploit a comprehensive solution for coping the continuous scale variation of crowds in both density map estimation and generation.

∗Corresponding author

A human can estimate the approximate number of people in a crowd image quickly using a global to local visual perception mechanism, without the scale and receptive field problems. People using the Top-Down visual perception mechanism typically scan a crowd image with a global receptive filed, and then estimate the degree of congestion of crowd regions based on the prior knowledge from the global scanning. Furthermore, Transformer architectures can provide a global to local receptive field[Zheng et al., 2021]. Inspired by the human's Top-Down visual perception mechanism and recent research progress about the Transformer, we propose a Crowd Counting Transformer network, namely, CrowdFormer, which models the human's Top-Down visual perception mechanism based on a series of Overlap Patching Transformer (OPT) blocks. An OPT block first learns global prior knowledge about a crowd image from an overlap visual patch sequence before focusing on analysing local crowd regions. In the OPT block, we can obtain a crowd counting from global to local, and encode the relative spatial position of visual tokens rather than absolute position, which is adopted in the standard vision transformer[Alexey et al., 2021] and may result in the predictive model losing the translation and rotation invariance.

Moreover, to generate more accurate groundtruth of density maps considering various target scales, we propose a multiple density kernels fusion-based density map generator (KFMG), which can fit targets of various scales by fusing multiple density kernels. Finally, we incorporate the density map generator and the estimator into an adaptive learning system, which takes both the dot annotation map and original image as input, and jointly learns the density map estimator and generator within an end-to-end framework.

In a nutshell, the main contributions of this paper are as follows:

• We propose to model the human's Top-Down visual perception mechanism in crowd counting by using the Transformer architecture. To the best of our knowledge, this is the first work to utilize Transformer architecture network for regressing density maps in crowd counting.
• A learnable density map generator, which takes various target scales into account, is proposed to generate more accurate density map groundtruths by fusing multiple density kernels.
• Extensive experiments are carried out to validate the proposed method, and demonstrate that our method is thestate-of-the-art frommultiple perspectives.

2 RelatedWork

Crowd counting is a fundamental task in computing vision. In the early phase, people usually treat the crowd counting as target localization tasks. For example, Min Li et al.[Min et al., 2008] used heads and shoulders localization to construct crowd counting. However, in severely occluded scenarios, such paradigms will be invalid. To cope with the densely crowded scene, two types of regression-based methods were proposed: global regression and density map regression. The global regression-based methods[Chan et al., 2008; Chattopadhyay et al., 2017; Yifan et al., 2020] typically estimate the final count of people directly from images, while the density map regression-based approaches[Yuhong et al., 2018; Shuai et al., 2020; Jia et al., 2020] first predicts a density map, which is then summed to obtain the final count. Because density maps contain more spatial context information, the performance is usually superior to that of global regression-based methods. Our work falls under the category of density map regression-based methods. As a result, we primarily examine two types of density map regression-based methods in recent literature. Moreover, since we introduce the Transformer architecture into the crowd counting task, existing Transformer approaches are also discussed.

2.1 DensityMap Ground-truth Generation

The use of density maps as the supervised information is a common choice among most of recent crowd counting methods. Traditional work[Lempitsky et al., 2010] typically convolve the crowd image by placing a fixed bandwidth Gaussian kernel on the dot annotation map to obtain a density map groundtruth. However, due to target scale variation, the Gaussian kernel with fixed bandwidth and parameters cannot contribute to all variation in existing images.

To cope with the scale variation on targets, some work was proposed to fit the scale variation by adjusting the bandwidth or parameters of the Gaussian kernel manually according to the scene perspective or crowdedness. For example, Idrees et al.[Haroon et al., 2018] fuse multiple density maps which were generated by multiple Gaussian kernels with different bandwidths. Wan et al.[Wan and Chan, 2019; Wan et al., 2020] propose using a CNN-based network as the density map generator. The parameters of the Gaussian kernel are changed in this way to make them learnable. Although many compelling methods for improving the quality of density map generation have been proposed, the problem of scale variation on targets remains unsolved due to the hand-crafted settings lacking generalization. In contrast to these previous work that requires manually setting the parameters or bandwidth of Gaussian kernel, we propose to adaptively fuse multiple density kernels, and jointly learn the density map generator with the estimator in an end-to-end manner.

2.2 Multiscale DensityMap Estimation

To deal with a large-scale variation on targets, some literature proposed using multiscale features or multicontext information during the feature extraction phase. As a result, using the feature pyramid network (FPN)[Tsung-Yi et al., 2017] to fuse multi-scale information is a common choice among recent crowd counting methods. For example, Chen et al.[Chen et al., 2021] utilized FPN to extract multiscale features with different receptive fields, whereas a pyramid region awareness loss[Qingyu et al., 2021] is utilized to recursively searches the most over-estimated sub-regions.

Although multiscale features and multicontext information about targets have been focused on in recent literature, these multiscale learning-based methods still fail to cope with the continuous scale variation on targets. In contrast to previous work that only consider a few scales, we present CrowdFormer, a novel Transformer network, that can fit targets with continuous scales by modeling the human Top-Down visual perception mechanism.

2.3 Transformers in Crowd Counting

CNNs are regarded as a hierarchical ensemble of local features with different reception fields. Unfortunately, most CNNs excel at extracting local features but struggle to capture global cues. Transformer[Ashish et al., 2017], which was proposed in the field of natural language processing to capture the long-term dependencies between input and output, was recently introduced to vision tasks. Due to fusing local and global features, Transformers promote the performance of many vision tasks significantly. For example, Dosovitskiy et al.[Alexey et al., 2021], constructed a Transformer network for image classification, and achieved excellent results compared to CNN-based models. While, Nicolas et al.[Nicolas et al., 2020], utilized Transformer structures to augment a standard CNN network for improving the performance on object detection. Moreover, Liang et al.[Dingkang et al., 2021] proposed TransCrowd, which involves a Transformer network into the weakly-supervised crowd counting task, and achieves good performance. However, because the final counts are directly regressed the TransCrowd fails to learn the spatial and contextual information of scenes. In contrast to TransCrowd, we propose a novel cutting-edge Transformer framework for density map estimation that takes efficiency, accuracy, and robustness into consideration.

3 CrowdFormer

In the real scene, there is a large-scale variation on different targets due to the perspective phenomenon. Recent work typically only can fit a few scales. However, Humans, on the other hand, are usually not affected by such scale variation, since we can take the global context information of the crowd scene into account with a global receptive field. In light of this, we propose CrowdFormer, which is capable of modeling the human's Top-Down visual perception mechanism in crowd counting task.

The CrowdFormer cascades CNN and visual Transformer structures for local crowd region enhancement and global context information capturing. An overview of CrowdFormer is depicted in Figure 1, which is made up of three major modules, i.e., a local feature embedding module, a Transformer structure-based encoder, and a lightweight decoder. To begin, the local feature embedding module extracts local features about crowd counting, and embeds them into the given low-resolution representations using a CNN structure inherited from the YOLOv5 backbone. Then, the encoder, which is stacked by our proposed Transformer blocks to generate feature maps of different resolution, is used to fuse the local crowd features and the global information of crowd scene. Finally, the lightweight decoder combines the multi-scale feature maps to predict a density map.

3.1 Local Feature Embedding

Considering that self-attention-based Transformer when performed across n entities requires O(n2 ) memory and computation, which is memory and computation consuming. While CNN can extract local but compact feature representation from the inputs efficiently. We first implement a CNN feature extractor to obtain a compact feature representation from the inputs for computing resource saving and local information refinement.

Specifically, the structure of local feature extractor we adopted in the CrowdFormer is inherited from the backbone of YOLOv5, and used for 1/4 downsampling feature extraction. In this way, we can obtain a good fitting ability with a small computing cost, as well as alleviate the effective information loss in the downsampling process. In practice, we directly use the part of pretrained backbone of YOLOv5 as the local feature extractor, because the tinny object detection and crowd counting have similarities in terms of task properties and their domain knowledge may can be transferred to each other.

3.2 Overlap Patching Transformer Block

The human could focus on the region of interest in the image, based on the goal of current task and global prior knowledge. Similarly, we boost the model attention on crowd regions by mining global context information. As a result, we propose a Transformer-based encoder to model the human Top-Down visual perception mechanism.

The encoder is stacked by Overlap Patching Transformer (OPT) blocks, which can mining the global prior knowledge of the crowd scene to reinforce the local crowd region feature from coarse to fine-grained. As shown in Figure.1, given the feature maps with a resolution of Hm × Wm, our encoder performs i-th OPT block to obtain feature maps Fi ∈ R Hm 2i × Wm 2i ×Ci , where i ∈ {1, 2, 3}. An OPT composed of two sequential parts: an overlap patching layer and a self-attention based feature encoder.

The overlap patching layer is to resolve the inputted 3D feature maps into a sequence of 2D patches xi ∈ RN×D like the stander vision transformer[Alexey et al., 2021], where N = Hi ri × Wi ri , D = ki ×ki ×Ci, ki ×ki is the resolution of the patches and ri is the stride of the sliding window. Instead of splitting inputs into non-overlapping patches in manner of ViT[Alexey et al., 2021], we use a ki × ki slide window with stride ri (ri < ki and ri = 2) on inputted feature maps to obtain overlapped 2D feature patch sequence, which patches maintain the relative position relation, while traditional transformers patch an input without overlapping, and may lost the relative position information between patches.

Transformer encoder utilizes a multi-head self-attention module (MSA) to learn the global prior knowledge about crowd sense from the patches xi in multiple feature subspace. Then, a learnable projector is used to project patches into their relative spatial positions, which is shown as in the overlap patch transformer block of Figure 1. Formally, the feature encoder can be denoted as:

where the MSA first splits the query, key, and value matrices of the standard self-attention into h ploids and performs attention functions of these ploids in parallel:

where Qi,p,Ki,p, Vi,p ∈ RN× Dh are the query, key, value, and input matrices of the p-th ploid self-attention, and WQi,p ,WKi,p ,WVi,p ∈ R Dh × Dh are the corresponding mapping matrices. The cat(∗) concatenates the outputs of h selfattentions together.

Specifically, fold(∗ , ∗) is a compound operator to keep the relative position of patches unchanged, and fold the patch sequence x′i into feature maps Fi ∈ R Hi ri × Wi ri ×Ci . It first reshapes the learnable project tensor Wi ∈ RCi×Ci-1×ki×ki to be the ˆWi ∈ RCi×D , and then performs a dot product between x′i and ˆW T i . Finally, the result of dot product is fold into Fi. With the weights sharing and overlap patching, similar to convolution network, the relative spatial position information of patches can be encoded into their feature representation. Comparing with adding the absolute position embeddings with feature vectors of patch sequence used in the standard vision Transformer[Alexey et al., 2021], proposed method may result in improved generalization performance in the terms of translation and rotation. It should be noted that, recent work[Ze et al., 2021] shows that encoding the absolute position of vision tokens may led to losing the translation and rotation invariance, which are vital to the generalization performance of visual models.

3.3 DensityMap Estimation

To predict the final density map, we propose a lightweight decoder. As shown in the decoder phase of Figure.1, each feature Fi outputted by the corresponding OPT block is sent into a feature collection unit (CFU), which utilizes a 1 × 1 convolution layer to fuse the Fi to be the ˆ Fi ∈ R Hm ri × Wm ri ×C , and then upsamples each ˆ Fi with bilinear interpolation into the given resolution Hp ×Wp. Then, all features ˆ Fi are concatenated to be the feature F ∈ RHp×Wp×3C , where C = 256 is a hyper-parameter, and a 1 × 1 convolution layer following a novel activation function, i.e., P-Sigmoid, is used to fuse the different level features and generate the density map prediction.

Specifically, P-Sigmoid can be defined as:

where a is a learnable parameter. It should note that a point on density map may express more than one person. Usually used activation functions in deep learning, e.g., tanh, Sigmoid and PReLU, are unsuitable for this task due to their ranges. Therefore, most of existing methods adopt the absolute function to be the activation function for generating the final densisy map. In comparison to the absolute function, the scale of P-Sigmoid values are close to the absolute function due to the learnable parameter a, as well as the gradients are smoother due to having the attributes of Sigmoid. The proposed P-Sigmoid is better suited to crowd counting.

4 Density Kernels Learning and Fusion

Previous related work mainly focuses on the density map estimation but ignores the generation of density map label. They usually generate a density map label by using a hand-crafted Gaussian kernel to convolve the dot annotation map, where the spatial location of each annotated person takes the value 1, (and 0 otherwise). In this way, these work failed to fit the scale change of the targets and scenarios. As a result, we propose a multiple density kernels fusion based density map generator, i.e.,KFMG, to generate density map label taking multiple scales content information into consideration. The pipeline is shown in Figure 2. We first generate multiple adaptive density kernels with different sizes from the original image, and then fuse those density kernels to be one fused kernel. Finally, the fused kernel is used to convolve the dot annotation map and generate the density map label.

Given an input image I, we construct a CNN to generate multiple adaptive density kernels K1,K2, ..,Km, and fuse them to be one adaptive density kernel ˆK. Formally, the generating process can be denoted as:

where Ω(∗) denotes a CNN, which first downsamples I to be the feature maps K ∈ RHp×Wp×CK with the given resolution of Hp × Wp. CK = ∑mi=1 k2 i is the number of channels in K, and ki is the size of the i-th adaptive density kernel. Then, split(K; [k2 1 , k2 2 , .., k2m]) is to divide K into m groups on the channel dimension, and each group Ki can be treated as an adaptive density kernel with the given size of ki × ki. Finally, these adaptive density kernels are aligned along with their center points, and fused to be one adaptive density kernel ˆK with the size of ˆ k × ˆ k, where ˆ k = max(k1, k2, ..., km).

Let D = {pj}Nj=1 be the set of N annotated 2D coordinates of the persons in dot annotation map, pj = (xj, yj) in the image I. For each annotation position pj, we retrieve the corresponding kernel map ˆKpj ∈ Rˆk׈k from ˆK, and then normalize it to sum to 1, resulting in the location-specific kernel ˆ kpj = ˆKpj /sum( ˆKpj ). Finally, the density map is generated by placing the location-specific kernel maps on the corresponding positions in the density map,

where the indexing of ˆ kpj is on p ∈ (-r, ..., r) × (-r, ..., r), and r = (ˆ k - 1)/2. It is important to note that the equation (5) is analogous to placing a Gaussians kernel on each dot annotation to generate a traditional density map, except that the multiple kernels are learned, and could be different for each position and each image. The learned density map M is then used as the groundtruth to train our density map estimator.

5 Experiments

In this section, we conduct experiments evaluating our proposed CrowdFormer and density map generation framework. First, we go over the datasets, training, and evaluation metrics in detail. Second, we compare our method with other approaches. Finally, we ablate the key design elements of our framework.

5.1 Experimental Setup

In this experiment, three real-world datasets, i.e., NWPUCrowd[Qi et al., 2021], UCF-QNRF[Haroon et al., 2018], and ShanghaiTech[Yingying et al., 2016], are used for evaluation in our experiments.

In model training, the CNN based local feature embedding are initialized from the first three blocks of pre-trained YOLOv5, and the rest parts of network are randomly initialized by a Gaussian distribution with the mean of 0 and the standard deviation of 0.01. The scales of adaptive density kernels we adopted for generating the density map groundtruth are 3×3 and 5×5, respectively. For parameters training, an Adam optimizer are employed 900 epochs with a cosine decay learning rate scheduler and 10 epochs of linear warm-up. The initial learning rate and weight decay are set to 1e-5 and 1e-4, respectively. The training batch size is set to 12, and data augmentations such as random-cropping of raw input with size 512×512, random horizontal flipping and color jittering are adopted.

During the training, we incorporated the proposed density map estimator and generator into an adaptive system, which takes the both annotation dot map and original image as input, and jointly learns the estimator and generator by pixel-level L2 loss between the estimation and the generation of density maps within an end-to-end framework.

5.2 Evaluation and Analysis

On four real-world datasets, we compare our proposed density map prediction framework to previous state-of-the-art methods to assess overall counting performance. Following the previous work, we adopt Mean Absolute Error (MAE) and Mean Squared Error (MSE) as the criterion to evaluate the counting performance. According to the results shown in Table1, comprehensively compared with previous methods, the performance of our method is the state-of-the-art, despite being challenged by other approaches in the highly congested scene, i.e., ShTech A.

Specifically, the TransCrowd[Dingkang et al., 2021], which also use the Transformer architecture for crowd counting task, appears to be inadequate to other methods. However, due to CrowdFormer, which encodes relative spatial position of visual tokens rather than absolute position, our approach outperforms TransCrowd on a variety of benchmarks. Furthermore, on the largest dataset, i.e., NWPU, our method achieves the best performance, which improves the state-of-the-arts by 13.3% on MAE and 12.8% on MSE. This result validates the effectiveness of the proposed method in real-world application. Finally, our method outperforms the dot counting regression methods, such as NoiseCC[Jia et al., 2020], DM-count[Boyu et al., 2020], BM-count[Liu et al., 2021], GLoss[Jia et al., 2021] and P2PNet[Song et al., 2021], on most of the datasets, because our density map generation method can utilize more context information about crowd counting during both the training and testing phases, and allowing us to deal with a wide range of complex crowd scenes. Moreover, comparing with the recent density map and multiscale methods, such as KDMG[Wan et al., 2020], DKPNet[Chen et al., 2021], SASNet[Qingyu et al., 2021] and S3[Lin et al., 2021], our method also can obtain significant advantage on most of the datasets, because the CrowdFormer can obtain a crowd counting in manner of global to local.

5.3 Ablation Studies

Ablation on the structure of CrowdFormer: We examine the effectiveness of the CrowdFormer structure. As the results shown in Table 2, when we use 3 × 3 convolution layers to replace all OPT blocks in CrowdFormer, the performance of crowd counting, i.e., Non OPT, degrades significantly when compared to the performance of CrowdFormer. Moreover, when we use an OPT block to replace the local feature embedding module of CrowdFormer, the performance, i.e., Pure OPT, can approach the performance of CrowdFormer, but a huge computation cost. These results demonstrate that the effectiveness of the CrowdFormer structure on context information extraction about crowd regions.

Ablation on activation function: We substitute a common used absolute function (Abs) for the P-Sigmoid to determine the effect of the proposed P-Sigmoid activation function. In detail, we use model training and testing to demonstrate the performance gain. As shown in Figure 4(a), in the training stage, the convergence performance of the model using PSigmoid is better than using Abs, because the gradients of P-Sigmoid are smoother. Furthermore, in the testing stage, the P-Sigmoid outperforms the Abs in the terms of MSE and MAE as shown in Figure 4(b). The preceding demonstrates that the performance gain of our model in terms of activation function is due to the proposed P-Sigmoid.

Ablation on density kernel fusion: We perform a series of experiments to determine the effect of density kernel size on the proposed density map generator. As the results shown in Figure 3, the small single kernel, e.g., k = 5, tends to yield better performance on the ShTech A and UCF-QNRF because their crowd scenarios are congested, while the large size, e.g., k = 7, can obtain better performance in relative sparse crowd scenarios, i.e., ShTech B. We thus choose to fuse 3 × 3 and 5 × 5 kernels in our density map generation framework, and achieve the best performance on all the datasets. This demonstrates that the suitable kernel fusion can improve the density map generation, because the fused density kernel has a better scene generalization.

6 Conclusion

In this paper, we primarily propose a novel vision transformer for modeling the humans' Top-Down visual perception mechanism in the crowd counting task, as well as a density kernels fusion framework for obtaining more accurate ground-truth of density map from the dot annotation maps. Moreover, we also incorporate these two innovations into an adaptive crowd counting model, which can jointly learn density map estimator and generator within an end-to-end framework. Finally, we conduct extensive experiments, and prove our proposed approach can achieve superior performance in the terms of MAE and MSE on widely used datasets.

Acknowledgments

This work is jointly supported by the National Natural Science Foundation of China(62106290) and the Program for Innovation Research in Central University of Finance and Economics.