ThinFormer: Channel Sparse Transformer for Efficient HRW Object Detection

1 Introduction

Object detection is a fundamental yet challenging topic in computer vision. Close-up settings such as MS COCO [Lin et al., 2014] have witnessed impressive performance with successful real-world applications. Nowadays, object detection in high-resolution wide (HRW) shots is attracting more and more attention [Wang et al., 2020; Li et al., 2024a; Li et al., 2024b; Liu et al., 2024]. As trivially adopting vanilla close-up detectors results in unsatisfying performance, difficulties unique to such setup come into view. The extreme sparsity of objects and the variability of the sparsity ratio among shots embarrass conventional close-up detectors.

† Corresponding author.

As found in PANDA [Wang et al., 2020] and DOTA [Xia et al., 2018], the most significant feature of HRW shots lies in the sparsity of information, as no more than 5% area of the image contains some objects. Such characteristic imposes challenges on backbones, which are supposed to effectively extract key features out of inundating background noise. On the other hand, the sparsity ratio varies violently among images. Conventional backbones tend to suffer from deteriorated performance and high computational expense. They lack the ability to distinguish between foreground and background, not to mention the adaptivity to variable sparsity ratio.

To address the difficulties above, a dynamic sparsification mechanism is necessary. Previous works mainly focus on taking advantage of the spatial sparsity of HRW shots by local techniques [Li et al., 2024a]. However, there is still a large gap from optimality. Inspired by the spectral sparsity of images and videos [Hassanieh et al., 2012], we notice the sparsity in channel dimension has long been ignored. This motivates us to generalize existing sparsification method on spatial positions to channels (Fig. 1).

We propose a dynamic framework named ThinFormer to take full advantage of the inherent sparsity of HRW shots, both in the spatial and the channel domain. In the spatial domain, Thinformer inherits existing local attention method; in the channel domain, it adopts a novel approach called local FFN, which is designed according to the delicate duality between spatial domain and channel domain. Furthermore, the weights of local FFN are dynamic since they are calculated from the inputs, distinct from the trivial generalization of the static weights of FFN. Owing to the sparsity of both domains, ThinFormer obtains significantly economic computational expense, while maintaining comparable performance with state-of-the-arts. Global attention and FFN are still preserved but in a coarse form to facilitate understanding of global features. Additionally, we implement a dynamic sparsity ratio estimator to address the variability of semantic complexity in images.

In summary, our main contributions are as follows:

• We propose a dynamic architecture for object detection in HRW shots, called ThinFormer. Our methodology takes full advantage of sparsity inherent in HRW shots in both spatial positions and channels. In pursuit of sparsity of channels, we propose a novel dynamic sparse feedforward block, named local FFN.
• We propose a dynamic sparsity ratio generator to flexibly deal with shots of different levels of complexity. The generator is trained in a self-supervised way for the sake of self-consistency. A series of accessory score nets is incorporated. The effectiveness of Thinformer is evaluated under a unified theoretical framework for sparsity in models.
• We extensively validate our method on a large-scale HRW-shot benchmark PANDA. Our model gains significant improvement in efficiency and maintains comparable performance compared with state-of-the-arts.

2 RelatedWorks

Close-up shot detection models. Close-up object detection primarily focuses on megapixel-level datasets like PASCAL VOC [Everingham et al., 2010] and MS COCO [Lin et al., 2014], leading to the development of two main detection architectures: single-stage [Redmon et al., 2016; Redmon and Farhadi, 2018; Ge et al., 2021; Liu et al., 2016] and two-stage [Girshick et al., 2014; Girshick, 2015; Ren et al., 2015; He et al., 2017; Cai and Vasconcelos, 2018]. In recent years, there has been a growing interest in anchorfree approaches [Zhou et al., 2019; Tian et al., 2022], especially with the introduction of the Transformer-based detection architecture, DETR [Carion et al., 2020]. However, these methods are generally designed for megapixel-level detection. Even those claiming real-time capabilities fail to meet the real-time requirements in gigapixel scenarios.

High-resolution wide shot detection models. The benchmark PANDA [Wang et al., 2020] is the first gigapixel-level dataset for HRW shots detection and has recently attracted significant attention. Some prior works focus on reducing latency through techniques like patch selection or arrangement [Najibi et al., 2019; Fan et al., 2022; Chen et al., 2022; Li et al., 2024b; Li et al., 2024c]. GigaHumanDet [Liu et al., 2024] leverages corner modeling for gigapixel-level full-body detection, especially in crowded scenes. While these methods address speed issues in relatively straightforward ways without deeply considering the architectural aspects, we found that the architecture of the detection model itself is the key factor influencing inference speed.

Efficient computation in vision transformers. Some prior works have explored sparse execution strategies, including dynamic selection of patches [Rao et al., 2021], self-attention heads [Meng et al., 2022], dimensions [Li et al., 2021; Li et al., 2022], and Transformer blocks [Meng et al., 2022], primarily for image classification tasks. PnP-DETR [Wang et al., 2021] proposed a poll-and-pool sampler to abstract image features from the backbone and feed sparse tokens to the attention encoder, demonstrating effectiveness across object detection, panoptic segmentation, and image recognition. However, sparse sampling strategies applied directly on the backbone have not been thoroughly investigated. DGE [Song et al., 2021] is a general-purpose plugin for Vision Transformers, but it is not compatible with ConvNet-based models and cannot handle arbitrary input sizes. While the challenge of designing a flexible and model-agnostic architecture for object detection in high-resolution wide (HRW) shots remains largely unexplored, SparseFormer [Li et al., 2024a] proposed a spatial sparsification strategy. However, it fails to account for the redundancy in the channel dimension.

3 Methodology

3.1 Overall Architecture

The overall architecture of our model is shown in Fig. 2. ThinFormer follows the conventional paradigm of attentionbased backbones like Swin Transformer. After position embedding phase, the input image is split into non-overlapping patches. Such division exists in both spatial and channel domains: neighboring positions (in spatial domain) are grouped into windows and channels are grouped into bands. In this paper, a window consists of 7x7 neighboring spatial positions and a band consists of 8 contiguous channels.

The feature-extracting procedure mainly consists of 4 stages, each of which begins with a window-merging layer that concatenate the features of 2x2 neighboring windows. The concatenated features are then projected to half of the previous dimension using a fully connective layer. Then the feature map is sent to the core components: global blocks and local blocks. A global block consists of a global multi-head self-attention (MSA) layer and a feedforward network (FFN). Similarly, a local block consists of a local MSA layer and a local FFN. The global blocks are aimed at extracting global features coarsely, while the local blocks focus on fine-grained analysis within a certain window and band. In this way, we strike the balance between performance and efficiency.

Taking into account the potential variability of sparsity of different pictures, we implement a dynamic sparsity ratio generator and a corresponding rating network for each stage. On entering a stage, each window or band is rated according to the richness of information it carries. Meanwhile, the generator performs a comprehensive evaluation on the feature map and decides on a reasonable sparsity ratio accordingly. The sparsity ratio and the scores altogether decide whether a certain window or bands should be sent to local blocks for further fine-grained feature extraction.

3.2 Sparsity in both Spatial and Channel Domain

The main concern of the backbone is how to extract meaningful features as efficiently as possible. In this section, we shall introduce in detail our key solution: a sequence consists of global blocks and local blocks.

Global attention on aggregated features. Some semantics of an image can only be interpreted by taking global information into account. That is why the ability to understand highlevel global features is an essential competence of a backbone. We adopt the global information extracting pipeline proposed in (Li et al. 2024). We take the input feature map z and divide it into windows of size M. The left-top location of a window is denoted (x, y), and other locations is determined by its relative deviation (∆x,∆y). The aggregated feature can be derived as follows:

where x′ = xM , y′ = yM . In this paper, we simply let α∆x,∆y ≡ 1, which indicates that the aggregated feature is the algorithmic average of information within the window. This aggregation dramatically cuts computational expense, as it is actually equivalent to M× downsampling. These aggregated feature map proceeds to be sent to trivial multihead self-attention blocks, which completes the extraction of global features. We shall rewrite it in a formula form:

where zl represents the feature map after the l-th global block. Finally the features are decompressed to the shape of the input of current stage, using the following inverse formula:

Local attention within selected windows and bands. Global blocks offer only coarse semantics. To extract finegrained features while avoiding expensive computational cost, prior works exploit spatial sparsity but neglect channel sparsity. We therefore propose local blocks that leverage sparsity in both domains. Inspired by signal processing, where filtering is applied to retain informative frequency bands, we treat channels as a transform domain and introduce a local block equipped with a joint selection mechanism over windows and bands.

We partition the feature map z of size H ×W × C into NS = HM × WM windows and NT = CD bands, i.e. a window consists of M × M spatial position and a band contains D channels. Be aware that such a partition is parallel, i.e. a window is of size M ×M × C and a band is of size H × W × D. Given an input feature map z, we allocate each window and each band a score based on their information richness and generate a sparse ratio k based on holistic statistics (Details in the following section). Taking into account that the sparsity ratio may be different between windows and bands, we introduce a hyperparameter µ to indicate such difference. In other words, KS = ⌊kNS⌋ windows and KT = ⌊µkNT⌋ bands will undergo further feature extraction. Windows or bands with higher scores are preserved for subsequent processes. Technically, we maintain two binary decision masks,AS ∈ {0, 1}NS and AT ∈ {0, 1}NT , subject to 1T AS = KS and 1T AT = KT , to indicate whether a window or a band should be preserved. Suppose the patched feature map is Z ∈ RNS×NT×N , then the sparse form of it can be derived as follows:

where ⊗ denotes Kronecker product and ⊙ denotes Hadamard product, and N = M ×M × D is the amount of entries in the intersection of a window and a band.

Inspired by spatial-channel duality [Han et al., 2022], we generalize local MSA (a sparsified global MSA) to its channel counterpart, local FFN (a sparsified FFN). Local MSA performs MSA within each window with shared Q, K, V matrices, yet produces distinct window-wise weights via dynamic self-attention. Analogously, performing FFN within each band with shared weights gives a form of local FFN, but the static nature of FFN yields identical weights across bands, which we term static local FFN. The name local FFN is reserved for our proposed dynamic form, referred to as dynamic local FFN when necessary.

It is widely known that one-layer FFN can be viewed as point-wise convolution:

To allow the weight to vary according to the input, we calculate the weight by the formula

Hence local FFN can be presented as

where x is the intersection of a window and a band, i.e. x ∈ RM×M×D .

In summary, the sparse feature map ˆ ZS undergoes the following operations:

where Z lS denotes sparse feature map after the l-th local block. The output of local blocks is merged back to the output of global blocks, in the following manner:

where Z is the output of global blocks, ˆ ZS is the output of local blocks, and ˆ Z is the output of an entire stage.

3.3 Dynamic Sparsity Ratio and Rating Network

The sparsity ratio may vary sharply among shots. Using a pre-defined sparsity ratio regardless of the specific case of individual shot inevitably leads to inefficiency. Therefore, we implement a sparsity ratio generator and a rating network for each stage.

When a feature map z is proceeded to the next stage, each band or window of it is rated according to the variance of information it carries:

where ˜ z = MaxPool(z)-AvgPool(z) somehow indicates the variance.

In analogous to Nyquist Sampling Theorem [Nyquist, 1928], having estimated the importance of each window and band, we are able to come up with a reasonable sparsity ratio, by

Given that vanilla Sigmoid function tends to push input towards 0 or 1, which is not desired, we insert an affine transformation before Sigmoid, i.e.

where α and β are learnable parameters.

To make end-to-end optimization possible, we introduce a self-supervised style loss term Lk for training sparsity ratio generator:

where k denotes the predicted sparsity ratio, Sunion denotes the area of the union of all predicted bounding boxes and S denotes the area of the entire shot. Heuristically, k and Sunion/S should be as close as possible, which means the model is able to estimate the sparsity ratio reasonably and consistently. Hence, the total loss is as follows:

To train the rating network, we adopts the Gumbel-Softmax trick [Maddison et al., 2017] to bridge the gap of gradient back-propagation between soft values and binary values through re-parameterization. In other words, before the output of global blocks Z are sent to local blocks, the estimated scores of windows sS and scores of bands sT are merged into it, by

where sS and sT denotes the scores of windows and bands, respectively.

4 Theoretical Analysis

In this section, we establish a theoretical foundation for ThinFormer, focusing on the relationship between dynamic sparsity (as used in ThinFormer) and classical static sparsity. We show that while dynamic sparse networks can be approximated by static ones, there exists a fundamental gap in representation efficiency that explains the empirical advantages of our approach.

4.1 Unified Framework for Sparsity

We begin by formulating a unified framework that encompasses both static and dynamic sparsity, which are also referred to as model sparsity and ephemeral sparsity in the literature [Hoefler et al., 2021]. Let f(x; w,m) denote a neural network with parameters w and binary mask m indicating active parameters. The most general sparse learning objective can be written as:

where γ measures computational complexity (e.g., FLOPs) and C is a budget. Both w and m can depend on the input x through parameters θ. This formulation captures:

• Model (Static) Sparsity: w and m are inputindependent.
• Ephemeral (Dynamic) Sparsity: w is inputindependent but m depends on x.
• ThinFormer's Dynamic Sparsity: Both w and m depend on x.

One should be aware that the loss function (15) is exactly with the form of the Lagrangian dual of (17).

4.2 Dynamic-to-Static Approximation

We now analyze the relationship between dynamic and static sparse layers in ThinFormer. Consider a simplified model where channels are partitioned into NT bands (blocks), each of dimension D. Let xi ∈ RD denote the i-th band of input x, i.e. x = [x1, · · · , xNT ]⊤.

Definition 1 (Dynamic Sparse Layer). A dynamic sparse layer is defined as fdyn(x) := [ydyn 1 , · · · , ydyn NT ]⊤, where

mi(x) ∈ {0, 1} is a dynamic mask and W ′ i (x) ∈ RD×D are dynamic weights. σ is the activating function.

Definition 2 (Static Sparse Layer). A static sparse layer is defined as fstatic(x) := [ystatic 1 , · · · , ystatic NT ]⊤, where

mi ∈ {0, 1} is a fixed mask and Wi ∈ RD×D are fixed weights. σ is the activating function.

Assumption 1 (Boundedness). There exist constants R,B,B′ > 0, s.t. ∀x ∼ D, i = 1, · · · ,NT : (1) ∥x∥2 ≤ R and ∥xi∥2 ≤ R/√NT ; (2) The expected weights ¯Wi = E[mi(x)W ′ i (x)]/E[mi(x)] satisfy ∥ ¯Wi∥F ≤ B; (3) ∥W ′ i (x)∥F ≤ B′ when mi(x) = 1.

Assumption 2 (Concentration). ∀i = 1, · · · ,NT , mi ∼ Ber(pi) and W ′ i (x) follows a sub-Gaussian distribution.

Assumption 3 (Activating Function). σ is Lσ-Lipschitz with σ(0) = 0.

Theorem 1. Under Assumption 1,2,3, given a dynamic sparse layer fdyn, ∃ϵ0 > 0, ∀ϵ > 0, there exists a static sparse layer fstatic such that with probability at least 1 - δ(ϵ):

where δ(ϵ) = O(exp(-c · ϵ2 )) for some constant c > 0.

Proof Sketch. The construction chooses a threshold τ and activates bands with pi ≥ τ . The error comes from two sources: (1) approximation error in active bands, bounded by concentration of W ′ i (x) around ¯Wi; (2) residual activation in inactive bands (pi < τ), bounded by τ . We can further show that τ = O(ϵ) for this construction. The full proof is provided in the supplementary material.

Theorem 1 shows that dynamic sparse layers can be approximated by static ones, but with crucial caveats.

To achieve error ϵ, the static model must activate all bands with pi ≥ τ(ϵ) = O(ϵ). For small ϵ, τ(ϵ) becomes small, requiring activation of most bands. This means: (i) A static model with fixed sparsity ratio cannot achieve arbitrarily small error; and (ii) To achieve a given error bound, a static model needs lower sparsity (more computation) than the dynamic model.

The inability to drive ϵ arbitrarily small without losing sparsity reveals a fundamental representation gap between dynamic and static sparsity. Dynamic sparsity implements a form of conditional computation that cannot be efficiently simulated by static sparsity with comparable computational budget.

Our theoretical analysis validates key design choices in ThinFormer:

• Dynamic FFN: The input-dependent weight generation enables richer representations than static FFN, justifying the additional complexity.
• Channel Sparsity: The band structure allows exploiting sparsity in the channel domain, complementing spatial sparsity.
• Adaptive Sparsity Ratio: The dynamic estimation of k adapts to input complexity, achieving better sparsityerror trade-offs than fixed ratios.

5 Experiments

5.1 Experimental Setup

Dataset and Evaluation. We conduct our experiments on the PANDA dataset, a large-scale benchmark for object detection in gigapixel-level images, featuring 18 scenes and over 15.9 million annotated bounding boxes. Following the official protocol, 13 scenes are used for training and 5 for testing. We adopt the standard COCO evaluation protocol, reporting Average Precision (AP) as the main metric, along with AP for small (APS), medium (APM), and large (APL) objects. To assess computational efficiency, we report the GFLOPs and parameter count of the backbone when processing a 1344 × 672 image patch.

ImplementationDetails. We implement all detectors using MMDetection toolbox [Chen et al., 2019]. To ensure a fair comparison, all models are configured with identical architectural hyperparameters when paired with the same detection head. Thinformer is trained from scratch for 60 epochs in an end-to-end style. Unless otherwise specified, some of the crucial hyperparameter are as follows: we adopt the dynamic local FFN and dynamic sparsity ratio generator; a band consists of 8 channels and a window consists of 7x7 spatial positions; µ = 2, λk = .1. The code is available in the supplementary materials.

5.2 Main Results

We benchmark Thinformer against state-of-the-art methods on the PANDA dataset, with results detailed in Table 1. Conventional close-up detectors like FasterRCNN and RetinaNet show limited performance, underscoring the unique challenges of gigapixel imagery.

To ensure a fair comparison of backbone architectures, we integrate Thinformer into the DINO detector framework. Our method achieves the performance of 80.6 AP.50, surpassing all other backbones. Notably, compared to the highly efficient SparseFormer, Thinformer not only obtains performance improvement but also reduces computational cost from 33.84 GFLOPs to 7.24 GFLOPs - a 78.6% reduction. This demonstrates that Thinformer establishes a novel and significantly better trade-off between accuracy and efficiency for object detection in high-resolution wide imagery.

5.3 Ablation Study

Effectiveness of Dynamic Mechanisms. In Table 2, we analyze the core dynamic components of our model. We compare our full model (with dynamic local FFN and dynamic sparsity ratio k) against variants with fixed sparsity ratio or static local FFN. The results confirm that our dynamic mechanisms are crucial, achieving the best overall performance while maintaining low computational cost. Furthermore, even with static local FFN Thinformer still achieves non-trivial performance, which validates our conjecture about the sparsity in the channel domain and our group-and-select method on channels.

Sparsity Ratio Distribution. Table 3 investigates the impact of the hyperparameter µ, which tunes the relationship between the sparsity ratios of the spatial and channel domains. Performance peaks at µ = 2, with degradation for both higher and lower values. Notably, simply preserving all the bands (i.e., µ = ∞) does not lead to improvement. This suggests that an optimal balance between spatial and frequency sparsification is key to success. Performance remains robust across a range of µ values, indicating our method is not overly sensitive to this hyperparameter.

Impact of BandSize. We explore the influence of band size on performance in Table 4. The results show that ThinFormer shows little sensitivity to this parameter, as performance remains stable across different configurations. A band size of 8 yields the best result and is adopted for all other experiments.

5.4 Visulization

As shown in Fig. 4, we visualize the learned scores that guide spatial sparsification at each stage of the backbone. The model effectively learns to assign higher scores (highlighted regions) to areas containing objects, thereby preserving salient information while discarding background.

Fig. 3 offers a glimpse into the transform-domain sparsification. We visualize the amplitude spectra of the average feature map within different bands and highlight the bands selected by our model (red bounding boxes). The selected bands clearly correspond to spectra with richer structural information, whereas bands with dominating low-frequency coefficients or much redundant information are pruned. This confirms that Thinformer learns to identify and process the most informative frequency components to some extent, which is critical for its efficiency and effectiveness.

6 Conclusion

In this paper, we present ThinFormer, a dynamic and efficient object detection framework tailored for HRW shots. By jointly exploring sparsity in both spatial and channel domains, ThinFormer addresses the challenges of extreme object sparsity and variable scene complexity. The proposed local FFN leverages channel-domain sparsity through dynamic activation, while a self-supervised sparsity ratio estimator adapts to diverse image conditions. Extensive experiments demonstrate that ThinFormer achieves competitive accuracy with significantly reduced computational cost.

Acknowledgments

This work is supported by National Science and Technology Major Project (No. 2022ZD0119402), National Natural Science Foundation of China (No. 62572268, No. 82441013), "Pioneer" and "Leading Goose" R&D Program of Zhejiang (No. 2024C01142), Tsinghua University - Migu Xinkong Culture Technology (Xiamen) Co., Ltd. Joint Research Center for Intelligent Light Field and Interaction Technology, Tsinghua University - Joint R&D Project (R2511DKW) - Phase 2, and Ministry of Science and Technology of China (No. 2024YFB2809103). We also acknowledge the KLATASDSMOE.

Contribution Statement

Wenxi Li and Kunpeng Liu made equal contribution to this work.