Towards Region-Adaptive Feature Disentanglement and Enhancement for Small Object Detection

1 Introduction

Object detection using Unmanned Aerial Vehicles (UAVs) plays a critical role in various applications, such as remote sensing [He et al., 2024] and autonomous driving [He et al., 2023]. However, the prevalence of small objects in UAV imagery poses a substantial challenge for models in extracting effective features, leading to a significant performance disparity between small and regular objects [Khanam R, 2024].

To address this challenge, multi-scale feature fusion methods leverage the distinct receptive field properties at each level by combining feature maps from different scales [Lin et al., 2017; Yang et al., 2024]. However, due to the limited features of small objects, these methods often result in weak activations, which increases the likelihood of losing critical information with existing downsampling strategies [Hou et al., 2020]. Furthermore, directly fusing multi-scale features from adjacent levels can cause strong activations from medium to large objects to overwhelm those of small objects at the current level [Huang et al., 2024b]. This exacerbates the loss of small object features and further diminishes the availability of learnable features for small objects (Fig. 1(a)).

∗ Corresponding author: Yang Ning and Xiushan Nie.

To address the challenge that existing downsampling and feature fusion strategies lead to significant loss of small object details, thereby resulting in suboptimal performance for small objects, we have investigated the activation intensity of objects at different scales across adjacent levels. Based on these insights, we propose the Region-Adaptive Feature Disentanglement and Enhancement (RAFDE) strategy, which effectively reduces the loss of small object detail features. Specifically, through visualization of the feature maps generated by the backbone, we observed that small objects tend to activate only the central region due to their limited feature representation. This increases the risk of losing small objects during the backbone's downsampling process (see the orange box in F3). To mitigate this, we introduce the Boundary Transitional Region-enhanced Downsampling (BTRD) module. This module disentangles and enhances the boundary transitional regions, which contain both strongly and weakly activated features, effectively reducing the loss of critical boundary features for small objects during the downsampling process (see the BTRD in Fig. 1(b)).

Through the methods outlined above, we obtained multiscale feature maps that effectively activate a larger number of small objects. However, there remains a need for a suitable strategy to fuse these features effectively. While extensive research on multi-scale feature fusion exists, most methods rely on addition or concatenation to combine feature maps, and still struggle with the loss of small object details (see yellow boxes in Fig. 1(a)). In multi-scale feature fusion, addition can enhance and filter critical features but may result in the loss of fine details. In contrast, concatenation preserves more details but may overwhelm small object features and introduce redundant information [Xiao et al., 2024]. To address these challenges, we propose the Region-Adaptive Feature Fusion (RAFF) module, which disentangles co- and uni-activated regions across adjacent levels and applies addition and concatenation strategies for fusion, respectively. This approach effectively mitigates the risk of small object features being overwhelmed. Finally, we integrate this module to replace the Feature Pyramid Network in fusing feature maps from three adjacent levels, significantly reducing the loss of small object features after fusion (see yellow boxes in Fig. 1(b)).

The RAFDE is a simple yet effective method that can be seamlessly integrated with various convolution-based detectors. It significantly enhances model performance while maintaining a minimal increase in computational cost and substantially reducing the number of parameters. In summary, the key contributions of our work are reflected in three main aspects:

• We introduce the Boundary Transitional RegionEnhanced Downsampling (BTRD) module, which enhances boundary transitional regions that contain both strongly and weakly activated features, thereby reducing the loss of critical boundary details for small objects.
• We propose the Region-Adaptive Feature Fusion (RAFF) module, which disentangles co-activated and uni-activated regions across adjacent levels, selectively enhancing and fusing key features to mitigate the risk of small objects being overwhelmed.
• We evaluated the RAFDE on the VisDrone and Dronevs-Bird datasets, achieving mAP improvements of 3.4% and 2.3%, respectively. The results demonstrate that RAFDE significantly enhances small object detection across different scenarios, underscoring its effectiveness in UAV detection.

2 RelateWork

2.1 Unmanned Aerial Vehicle Object Detection

In computer vision, detecting objects in UAV images is a challenging yet crucial task, as these objects typically exhibit a long-tailed distribution and are predominantly small. Due to the limited and densely packed nature of object features, existing models often struggle to extract effective representations, resulting in suboptimal performance.

In recent years, deep learning methods have made significant progress in addressing the challenges associated with small object detection. For instance, data augmentation techniques, such as copy-pasting, scaling, and cropping, increase the diversity and quantity of small object samples in UAV scenes, thereby enhancing dataset size [Meethal et al., 2023; Wang et al., 2023]. However, while these methods are effective for specific datasets, they lack generalizability due to their heavy dependence on particular data sources. Contextual learning strategies [Du et al., 2023; Sun et al., 2022; Huang et al., 2024b] improve object and scene classification and localization by leveraging environmental and inter-object relationships. Nonetheless, not all objects have clear contextual information, and the use of incorrect context can negatively impact performance.Additionally, alternative methods [Dai et al., 2024; Lei et al., 2023; Moser et al., 2024], including improvements in loss functions and attention mechanisms, have also boosted small object detection performance.

The widely used multi-scale feature fusion module effectively combines high- and low-level features, thereby improving the representation of small objects. However, traditional fusion strategies may overwhelm small object features by failing to account for the impact of large object activations in high-level features on small object activations in lowlevel features. To overcome these limitations, we propose the RAFDE strategy, a plug-and-play solution designed to reduce the risk of small object features being overwhelmed and lost, thereby improving small object detection performance. Additionally, we introduce a boundary transitional regionenhanced downsampling technique for the backbone, which generates multi-scale feature maps that activate more small object features. This approach significantly reduces model parameters while improving overall performance.

2.2 Multi-scale Object Detection Strategy

Multi-scale detection strategies address object scale variations by integrating high-level features, which capture richer semantics and larger receptive fields, with low-level features that preserve finer details, thereby enhancing small object detection.

Traditional computer vision methods typically extract features at a single scale, limiting their ability to detect objects of varying sizes or handle scenes with different proportions. To overcome this limitation, researchers developed the Feature Pyramid Network (FPN) [Lin et al., 2017; Li et al., 2024], which combines feature maps from multiple scales. By merging these maps, FPN leverages both detailed and semantic information, improving small object detection and addressing the constraints of single-scale features. However, this approach lacks effective integration between high-level and low-level features. To address this, the PAFPN [Liu et al., 2018b] extended FPN by introducing a bottom-up path, allowing high-level features to retrieve finer details from low-level ones. Despite this, a significant semantic gap remains between high-level and low-level features, complicating the selection of a fusion method that minimizes the loss of small object features. For instance, AFPN [Yang et al., 2024] merges adjacent low-level features first, then gradually incorporates high-level features to reduce the semantic gap between non-adjacent levels. To further enhance multiscale fusion, EFC [Xiao et al., 2024] addresses the inadequacies of simple concatenation or addition by emphasizing the contextual relationships between features at each level, improving feature relevance.

While current methods effectively combine high-level semantics with low-level details, they often ignore that activations at different scales correspond to objects of varying sizes. This can lead to activations from larger objects overshadowing smaller ones during direct fusion, limiting model performance. To address these challenges, we propose a new feature fusion module that considers activation intensity across different scales, reducing the risk of small objects being overwhelmed during fusion process.

3 Method

3.1 Preliminary

To facilitate the understanding of our subsequent paper, we first provide the mathematical definition of the co-activated region, the uni-activated region, and the boundary transitional region, as shown in Fig. 2, respectively.

Consider three consecutive feature maps, F3, F4, and F5, where each element f k i,j represents the activation intensity at position (i, j) on the k-th feature map, with i and j as the row and column indices, respectively, and k ∈ {3, 4, 5}. Given a threshold T, if the activation intensity f k i,j exceeds this threshold, the position (i, j) is considered strongly activated on the corresponding feature map [Selvaraju et al., 2017].

Specifically, let Ak denote the set of all normalized strong activation features on the k - th feature map. Then, the following definition holds:

Thus, as shown in Fig. 2(a), the co-activated region Φ can be defined as:

Then, with the help of the weakly activated region of A4 shown in Fig. 2(b), the uni-activated region Ω can be defined as:

Finally, as shown in Fig. 2(c), for the feature map F5 that simultaneously contains strongly and weakly activated regions, we define the boundary transitional region Ψ as:

where A+sin k and A+sout k denote the operations of expanding region Ak outward and inward by widths sin and sout, respectively. sin + sout defines the width of the boundary transitional region, i.e., the pooling kernel size.

3.2 Regional Adaptive Feature FusionModule

In multi-scale feature fusion, it is important to acknowledge that feature maps at different levels have distinct receptive fields, which activate objects of varying scales [Lin et al., 2017]. However, existing fusion strategies that simply merge adjacent levels often result in large objects in high-level features overwhelming small objects in low-level features.

To address these challenges, we propose the RegionAdaptive Feature Fusion (RAFF) module. The module carefully disentangles the co-activated regions and uni-activated regions across adjacent levels, effectively integrating the unique features from adjacent levels into the current level by leveraging the advantages of both fusion strategies. Specifically,as illustrated in the RAFF module in Fig. 3, we first align the high-level features F o i+1 and low-level features F o i-1 with the dimensions of the current-level feature map Fi. Specifically, the transformations of Fi+1 and Fi-1 are defined as follows:

where UP(·) denotes the upsampling operation, and DWConv(·) refers to depthwise separable convolution, which reduces both computational cost and parameter count while maintaining performance.

We then apply the channel attention mechanism to emphasize relevant channel information, compressing it into a single-channel feature map. As a result, the activation maps for the three adjacent levels are represented as δi-1, δi, δi+1:

where ECA(·) refers to the efficient channel attention mechanism [Wang et al., 2020].

For the low-level features, we enhance δi-1 and δi using the spatial attention mechanism [Woo et al., 2018], resulting in their enhanced features ηi-1 and ηi, respectively. Specifically, referring to Eq. (2), the co-activated regions Φsi and Φdi for the low- and high-level featuremaps, respectively, can be approximated as:

For the features within the co-activated regions, we apply addition to enhance them, reducing the risk of introducing redundant features and the potential for activated features to be overwhelmed. Consequently, the enhanced features F c i from the co-activated regions of the adjacent three levels can be defined as:

Following the definition in Eq. (3), we can further obtain the approximated uni-activated regions Ωsi and Ωdi in the low- and high-level feature maps, respectively, as follows:

For the features within the uni-activated regions, we apply concatenation to enhance them, thereby reducing the loss of unique features in adjacent levels. Therefore, the final feature map after region-adaptive fusion F final i is represented as:

The RAFF module effectively disentangles co-activated and uni-activated regions from adjacent levels, seamlessly fusing distinctive features from these levels into the current level through addition and concatenation. By substituting the traditional feature pyramid structure [Liu et al., 2018b] with the RAFF module, we achieve better integration of critical information across multiple feature scales, while mitigating the risk of larger object features overwhelming the details of small objects.

3.3 Boundary Transitional Region-enhanced Downsampling

Due to their limited features, small objects often produce weak activations, which increases the likelihood of losing detailed information during downsampling in existing backbone networks, particularly at critical boundaries [Hou et al., 2020; Chen et al., 2023]. This makes it challenging to capture sufficient and effective features for small objects, ultimately resulting in poor detection performance.

To address the aforementioned challenges, we propose the Boundary Transitional Region-enhanced Downsampling (BTRD) module, which enhances the boundary transitional regions that contain both strongly and weakly activated features. This approach minimizes the loss of crucial information related to small objects during downsampling. Specifically, as shown in Fig. 3, we utilize coordinate convolution to integrate positional information into the feature map Fi, resulting in the enhanced feature map F ′ i :

where AddCoords(·) denotes the coordinate convolution operation [Liu et al., 2018a] used to embed coordinate information, and Conv(·) refers to the convolution operation with a kernel and stride of 1.

Next, we compute the weights for the boundary transitional regions by calculating the difference between the maximum pooling and average pooling operations (see purple dotted box in Fig. 3). Specifically, we first transform the feature map F ′ i into a single-channel activationmap F a i , which is defined as F a i = SA(F N→1 (F ′ i )). Referring to Eq. (4), the boundary transitional region Ψi for the feature map F a i can be approximated as follows:

where SA(·) denotes the spatial attention operation [Woo et al., 2018], MP(·) denotes the maximum pooling operation, AP(·) denotes the average pooling operation, and σ represents the sigmoid function. Experimental results indicate that a pooling kernel size of 2 × 2 yields the optimal performance, as detailed in Sec. 4.5.

We then reconstruct the feature map F ′ i using the obtained boundary transitional weights Ψi. By applying average pooling to filter out noise and using maximum pooling combined with the weights to emphasize important regions, we effectively highlight significant features while minimizing the influence of irrelevant regions. The reconstructed feature map F re i is defined as follows:

Finally, in order to minimize information loss, we first split the feature map F re i into two channels (F re1 i , F re2 i ) and apply max pooling to preserve critical information in the transitional regions. Concurrently, we use convolution to capture detailed information from other activated regions. The downsampled feature map F down i is defined as:

where ⊕ denotes the concatenation operation, and EMA(·) refers to the efficient multi-scale attention mechanism [Ouyang et al., 2023], which helps focus the feature maps on the foreground object, thereby reducing background interference.

The BTRD module enhances the boundary transitional regions containing both strongly and weakly activated features, mitigating the risk of information loss for small objects. This method preserves more unique features of small objects, resulting in multi-scale feature maps with stronger representation capabilities for small objects.

4 Experiments

We have integrated our RAFDE module with the latest YOLO model and conducted experiments on two widely used drone image benchmarks: the VisDrone dataset [Du et al., 2019] and the Drone-vs-Bird dataset [Coluccia et al., 2021]. The VisDrone dataset consists of 7,019 high-resolution images (2000 × 1500) containing 10 classes of small, densely packed objects. Of these, 6,471 images are used for training, 548 for validation, and 1,610 for testing. The Drone-vs-Bird dataset includes 1,387 training images and 434 test images, featuring both UAV and environmental data. We evaluated the model's performance using Mean Average Precision (mAP ), where mAP50 corresponds to an Intersection over Union (IoU) threshold of 0.5, and mAP is calculated as the average across IoU thresholds ranging from 0.5 to 0.95. Due to space limitations, additional experimental results can be found in the supplementary materials.

4.1 Implementation Details

We implemented our RAFDE strategy using PyTorch [Paszke et al., 2019]. All models were trained for 150 epochs, with YOLOv11m serving as the baseline. Our approach employs the same loss function as YOLOv11 [Khanam R, 2024], which includes both object classification loss and bounding box regression loss. For the classification loss, we combine BCELoss [Zheng et al., 2020] and FocalLoss [Li et al., 2020], while for the regression loss, we use CIoULoss [Wang et al., 2023]. The input resolutions were set to 640 × 640 and 1280 × 1280 for the VisDrone dataset, and 640 × 640 for the Drone-vs-Bird dataset. All models were trained using the Adam optimizer with an initial learning rate of 0.01 and a decay rate of 1e-5. Training and testing were conducted on a single RTX A6000 GPU, with batch sizes of 8 and 2 for input resolutions of 640 × 640 and 1280 × 1280, respectively.

4.2 Comparison with State-of-the-Art Methods

Table 1 presents a comparison of our RAFDE with stateof-the-art methods on two widely used datasets. On the VisDrone dataset, using an input resolution of 1280 × 1280, our method was compared with SDPDet [Yin et al., 2024], EFC [Xiao et al., 2024], and YOLOv11 [Khanam R, 2024]. The results demonstrate that our method not only has fewer parameters but also delivers competitive performance, with improvements in mAP (36.80%→38.0%) and mAP50 (57.6%→59.6%). Additionally, we evaluated our method on the Drone-vs-Bird dataset using a 640 × 640 input resolution, where it achieved significant improvements over the baseline, with mAP rising from 52.30% to 54.60% and mAP50 increasing from 93.10% to 96.30%. These results demonstrate that our method is highly competitive compared to existing state-of-the-art methods.

4.3 Ablation Studies

We evaluated the effectiveness of each component of our RAFDE method using the VisDrone validation set. YOLOv11m with a 640 × 640 input resolution was used as the baseline, and the metrics mAP and mAP50 were employed for evaluation. As shown in Table 2, the incorporation of the RAFF module, which disentangles co- and uniactivated regions in adjacent low-level and high-level feature maps, effectively mitigates the risk of small object features being overwhelmed during feature fusion. This approach not only reduced the model parameters from 20.06M to 14.40M, but also significantly enhanced its performance, with mAP improving from 26.6% to 28.8%, and mAP50 increasing from 43.5% to 47.0%.

By incorporating the BTRD module to enhance the boundary transitional regions, which contain both strongly and weakly activated features, our method effectively reduces the loss of critical features for small objects. This modification led to a further reduction in model parameters from 14.40M to 11.02M, while also achieving a notable performance boost, with mAP improving from 28.8% to 30.0% and mAP50 increasing from 47.0% to 48.5%. Compared to the baseline, our RAFDE method significantly improves small object detection performance, with minimal increase in GFLOPs and a substantial reduction in model parameters.

4.4 Visualization

Figure 4 demonstrates the effectiveness of our method compared to the baseline on the VisDrone test dataset. Specifically, our approach improves the detection of distant small objects, addressing the issue of missed detections. In the first and second rows, YOLOv11 struggles with detecting distant small objects, missing several distant "people" (see first column, fourth row) and misclassifying "traffic cones" as "people" (see second column, second row). In contrast, our method detects more small objects (see third row, fourth and fifth columns) and reduces false detections (see second column, second and fourth rows). These results highlight the ability of our method to preserve the fine details of small objects, enabling the model to learn from a broader range of small object samples during training, which significantly enhances detection performance.

4.5 Pooling Kernel Size Selection in BTRD

In the BTRD module, a fixed region size is required for scanning in order to extract the maximum and average values within the region. These values are then used to identify the boundary transitional regions containing both strongly and weakly activated features, by calculating their differences. To determine the optimal region size, we tested scanning regions of 2 × 2, 3 × 3, and 5 × 5. The results, shown in Table 3, indicate that larger regions tend to introduce more noise, thereby reducing performance. Therefore, we selected the 2 × 2 region size for optimal performance.

4.6 Multi-level Semantics Selection in RAFF

When replacing the PAFPN [Liu et al., 2018b] with our RAFF module in YOLOv11, we investigated the influence of low-level detail information and high-level semantic information on the detection of both small and regular-sized objects. As shown in Fig. 1(b), the high-level feature map, used for detecting regular-sized objects, effectively activates with detailed information from adjacent levels. However, as seen in Table 4, the low-level feature map, used for detecting small objects, experiences a significant performance drop when deprived of high-level semantic information. Fig. 5 further demonstrates that the absence of high-level semantic information in the low-level feature map causes small objects to blend into the background, resulting in a considerable loss of detected small objects (highlighted in red boxes in Fig. 5). To address this issue, we directly concatenate high-level and low-level feature maps to retain critical semantic information (as shown in the red box in Fig. 3), leading to a significant improvement in the model's ability to detect small objects.

Basline

RAFDE

Table 4: Effect of low-level feature map with or without high-level semantic information on model performance.

4.7 Effectiveness of RAFDE on Other Backbones

Our plug-and-play method can be easily integrated into any model featuring an FPN structure. To showcase the versatility of RAFDE, we incorporated it into the popular MMDetection framework [Chen et al., 2019] and conducted tests on the VisDrone dataset. Specifically, we selected two twostage models: Faster R-CNN [Ren et al., 2016] and Cascade R-CNN [Cai and Vasconcelos, 2018]. All experiments were performed on a single RTX A6000 GPU, using a batch size of 4 and an input resolution of 1333 × 800. The models were trained for 12 epochs with configurations consistent with the baseline. Performance was evaluated using COCO metrics, including APs and APm, which represent the average precision for small and medium objects, respectively, at IoU thresholds ranging from 0.5 to 0.95. Additionally, ARs was used to measure the average recall for small and medium objects under the same IoU settings, while AP and AR represent the overall average precision and recall across the full IoU range. Additionally, AP50 denotes the average precision at an IoU threshold of 0.5. As shown in the table below, our method consistently improves model performance across different backbones, particularly for small objects (e.g., Faster R-CNN: 13.3% → 16.5%).

5 Conclusion

In this study, we present a lightweight approach to enhance UAV object detection by investigating the impact of activation strengths of objects at different scales in adjacent levels of feature maps on downsampling and fusion. First, we employed the BTRD module to strengthen the boundary transitional regions, mitigating the loss of crucial boundary features for small objects during downsampling. Second, we introduced the RAFF module, which replaces the conventional FPN structure. This module performs region-adaptive fusion of the co- and uni-activated regions, which are disentangled from adjacent levels, into the current level. This approach reduces feature loss and mitigates the risk of small objects being overwhelmed. Extensive experiments confirm that our RAFDE strategy reduces model parameters while remaining competitive with other state-of-the-art methods.

Acknowledgements

This work is supported in part by the Major Basic Research Project of Shandong Provincial Natural Science Foundation (ZR2024ZD03), National Natural Science Foundation of China (62176141), Taishan Scholar Project of Shandong Province (tsqn202103088), Major science and technology innovation project of Shandong Province (2021CXGC11204), Natural Science Foundation of Shandong Province (ZR202103010201), Shandong Excellent Young Scientists Fund (ZR2024YQ006), Shandong Province Higher Education Institutions Youth Entrepreneurship and Technology Support Program (2023KJ027).