GSDet: Gaussian Splatting for Oriented Object Detection

1 Introduction

Oriented object detection aims to identify and locate objects with arbitrary orientations in images [Murrugarra-LLerena et al., 2025; Xie et al., 2024]. Unlike objects in horizontal object detection, oriented objects often appear in various scales, aspect ratios, and orientations, which significantly increases the difficulty of accurate detection [Cheng et al., 2022a; Xia et al., 2018]. These challenges necessitate the development of techniques that can effectively localize and classify oriented objects regardless of their complexity.

Over the years, oriented object detection has evolved significantly, driven by advancements in both CNN-based [Xie et al., 2024; Han et al., 2021] and transformer-based [Zeng et al., 2024] methods. CNN-based object detectors can be broadly categorized into one-stage and two-stage detectors. One-stage detectors [Han et al., 2021] predict object categories and bounding boxes directly from the feature map in a single pass, utilizing dense object candidates like anchors to cover potential object locations. In contrast, two-stage detectors [Xie et al., 2024] first generate region proposals using a region proposal network and then refine these proposals to produce the final results. Recently, transformer-based methods [Zeng et al., 2024] have gained traction in this field, which treat object detection as a set prediction problem and use a set of learnable queries from the encoder to directly predict the final bounding boxes and class labels.

∗Corresponding author

Although CNN-based and transformer-based methods have demonstrated promising performance, they still rely on manually designed object candidates, such as anchors, region proposals, and queries. Moreover, these candidates, which represent potential objects, struggle to effectively capture spatial information of objects.

To address these issues, we propose a novel framework that directly detects oriented objects from random 3D Gaussians.

Anchor Query

(c) Query

The random initialization does not involve learnable neuron parameters. Regarding spatial information, a 3D feature space is constructed from image features. Within this space, the 3D Gaussians are optimized to represent oriented objects and can be projected onto the image plane to form 2D Gaussians, which are then transformed into oriented boxes. This Gaussian-to-box method eliminates the need for prior anchors, proposals, or queries, which simplifies the generation of object candidates and establishes a streamlined pipeline.

Our method is inspired by Gaussian splatting (GS), as illustrated in Figure 1. The philosophy of Gaussian-to-box paradigm is analogous to the Gaussian-to-image process in Gaussian splatting [Kerbl et al., 2023], where a scene is represented by 3D Gaussians and high-quality rendering is achieved by projecting these 3D Gaussians onto the 2D image space. Gaussian splatting has demonstrated significant advancements in 3D scene representation tasks. However, to the best of our knowledge, no prior work has successfully applied Gaussian splatting to oriented object detection.

In this paper, we propose GSDet, a novel approach that formulates oriented object detection as Gaussian splatting. A 3D feature space is constructed by treating multi-scale feature maps from the encoder as supporting planes within a 3D coordinate system. In this space, 3D Gaussians serve as a flexible representation of objects and are initialized randomly, without relying on structure-of-motion (SFM) points. The Gaussian attributes, including the mean, anisotropic covariance, and confidence scores, are optimized using a decoder that incorporates 3D Gaussian sampling to extract spatial features. The predicted 3D Gaussians are projected onto the image plane to form 2D Gaussians, which are further transformed into predicted boxes via decomposition.

Our GSDet exhibits remarkable flexibility during inference. Firstly, it allows for a dynamic number of initial 3D Gaussians as input. We can train the model with a certain number of Gaussians and then use a different number during inference. Secondly, GSDet enables iterative reuse of the decoder to refine the predicted Gaussians through adaptive control, allowing Gaussians to focus on foreground objects.

Experiments on 3 datasets validate the effectiveness of GSDet. Its performance is further improved through the use of adaptive control and an increased number of Gaussians during inference. Contributions are summarized as follows:

• We propose a detector GSDet, which is the first work to formulate oriented object detection as Gaussian splatting

to the best knowledge of our knowledge.

• We perform detection within a 3D feature space constructed from image features, where the designed 3D Gaussian sampling captures spatial information.
• Anisotropic 3D Gaussians are employed to represent oriented objects through their attributes. They are projected onto the image plane to form 2D Gaussians, which are then transformed into oriented boxes.
• Our Gaussian-to-box detection paradigm exhibits favorable properties, including adaptive control and dynamic Gaussians during inference, owing to the random initialization of 3D Gaussians.

2 RelatedWork

Oriented object detection. The mainstream methods in oriented object detection can be broadly categorized into CNN-based and transformer-based approaches. CNN-based methods, which have achieved significant success, are further divided into single-stage and two-stage methods. Twostage methods [Ding et al., 2019; Xie et al., 2024] utilize a region proposal network (RPN) to generate dense proposals, which are then refined by a region of interest (RoI) head. In contrast, single-stage methods [Hou et al., 2022; Nie and Huang, 2023] bypass the need for region proposals and detect objects directly from dense anchors.

To eliminate the need for dense anchor or proposal candidates, transformer-based methods establish a paradigm based on a set of queries [Zeng et al., 2024; Hu et al., 2023]. These queries represent both the semantic and location information of objects, and are selected through the encoder [Zhu et al., 2020]. Mainstream transformer architectures consist of an encoder and a decoder, with attention mechanisms serving as their core. Self-attention in the decoder enables queries to interact with one another, while cross-attention integrates encoder features with queries.

Unlike these methods, our approach is the first to predict oriented boxes directly from random Gaussians using Gaussian splatting, eliminating the need for pre-designed anchors, proposals, or learned queries, as illustrated in Figure 2.

Gaussian splatting has emerged as a significant technique in computer graphics, offering explicit scene representation and efficient rendering capabilities [Kerbl et al., 2023]. It represents objects and surfaces as collections of Gaussians, allowing for high-quality image synthesis. This method has

Proposal Decoder been explored in various applications, including scene reconstruction [Yang et al., 2024; Wu et al., 2024], SLAM [Yan et al., 2024; Matsuki et al., 2024], and autonomous driving [Zhou et al., 2024b; Zhou et al., 2024a], demonstrating its versatility and potential for real-time rendering. Despite the success of Gaussian splatting in these domains, there are no previous solutions that successfully adapt Gaussian splatting for oriented object detection. We believe that the slow progress of Gaussian splatting in oriented object detection is primarily due to the different optimization objectives, i.e., Gaussians and oriented boxes. To the best of our knowledge, this is the first work that adopts Gaussian splatting for oriented object detection.

3 Method

Oriented object detection identifies oriented boxes and class labels for objects in images. The core of our GSDet is to address oriented object detection through Gaussian splatting. In this section, we provide a detailed description of our detector. The overview of our GSDet is illustrated in Figure 3.

3.1 Gaussian for Oriented Object

Oriented box. The objects rotate arbitrarily in oriented object detection. Typically, an oriented box is represented as (cx, cy,w, h, θ), where cx and cy denote the center coordinates, w and h represent the width and height, and θ specifies the angle of orientation. To make better use of spatial features, we introduce another representation (cx, cy, cz, s, θ), where cz denotes the scale of objects and s denotes the aspect ratio. The cz and s are calculated as:

where wh denote the area, and hw denote the aspect ratio of objects. The log2(·) is used to limit the range of value. Furthermore, the (cx, cy, cz) directly represents the center of a 3D Gaussian in a 3D space, which will be introduced below.

3DGaussian representation. In our approach, anisotropic 3D Gaussians are initialized randomly. The basic 3D Gaussian N3d(µ3d, Σ3d) is described by its mean µ3d, covariance matrix Σ3d, and confidence score c.

Geometrically, the mean µ3d represents the center coordinates of ellipsoids, and can be defined as:

The covariance matrices Σ3d have geometric meaning only when they are positive semi-definite. Typically, it is difficult to constrain the learnable model parameters using gradient descent to generate such valid matrices. To avoid invalid matrices during training, we optimize the factorized form of the covariance matrix. The covariance matrix can be factorize into a rotation matrix R3d and scaling matrix Λ3d:

where the rotation matrix R3d and scaling matrix Λ3d are expressed as:

The rotation matrix R3d describes the rotation of the entire Gaussian by angle θ around a specified axis geometrically. In our setting, the 3D Gaussians rotate only around the zaxis, as the image is captured from a single perspective. The scaling matrix Λ3d changes the size of a Gaussian by scaling it along the x-, y-, and z-axis. The λ1, λ2, and λ3 are scaling factors in different eigenvector directions and determine how much the Gaussian is stretched or shrunk in each direction. Specifically, the λ1 and λ2 are calculated as:

where σ = 3 by default according to the 3-sigma rule. The λ3 is not explicitly defined, but features closer to cz receive more attention, which will be elaborated in Sec. 3.2.

The confidence score c is used for classification.

Splatting: 3DGaussian→2DGaussian. In oriented object detection, only the original image view is considered, not new viewpoints. Consequently, the 3D Gaussians are projected onto the image plane along the z-axis to produce 2D Gaussians N2d(µ2d, Σ2d). The mean µ2d is calculated as:

The covariance Σ2d can also be factorized by rotation matrix R2d and scaling matrix Λ2d, i.e., Σ2d = (R2dΛ2d) (R2dΛ2d)⊤. The R2d and Λ2d can be calculated as:

The confidence score c is consistent between each 2D Gaussian and its corresponding 3D Gaussian.

The time consumption in the splatting process is negligible, as 3D Gaussians can directly achieve orthogonal projection using these concise formulas.

Transformation: 2D Gaussian → Oriented box. Each Gaussian characterizes a potential object in our method, rather than multiple Gaussians jointly determining a potential object. Thus, given the mean µ2d, rotation matrix R2d, and scaling matrix Λ2d, the oriented boxes as final results are transformed from 2D Gaussians by the reverse process of Eq. 1, Eq. 5, Eq. 6 and Eq. 7 through decomposition.

3.2 Architecture

Our framework consists of an image encoder and a detection decoder, which is a simple paradigm. The image encoder provides 3D feature space and the detection decoder is trained to refine the random 3D Gaussians.

Image encoder. The image encoder is composed of a backbone [He et al., 2016; Liu et al., 2021] and a feature pyramid network (FPN) [Lin et al., 2017a]. The backbone takes in the raw image as its input and extracts image features for the subsequent FPN. The FPN is utilized to generate multi-scale feature maps with 256 channels. Specifically, we construct the pyramid with levels ranging from P2 to P6, where Pl has a resolution that is 2l times lower than that of the image. The l ∈ {2, 3, 4, 5, 6} represents the pyramid level.

3D feature space. Since oriented object detection lacks real depth information, we construct a 3D feature space based on the feature pyramid. The z-axis coordinates Zl of each feature map are computed as:

We rescale the feature maps P3 ∼ P6 to the same size [WF ,HF ] as P2 by interpolation. These feature maps, acting as supporting planes, are aligned along the x- and y-axes in the 3D feature space, denoted as {F2, F3, F4, F5, F6}.

3D Gaussian sampling. We implement 3D Gaussian sampling in a decomposition manner. We first sample in the feature maps and then integrate the Gaussian weights of the zaxis. The main sampling area Gxy (i, j) based on the 3-sigma rule for each feature map is calculated as follows:

Algorithm 1 GSDet Training

Input: images, GT

Output: loss L

• 1: {P2, . . . , P6}←Encoder(images)
• 2: {F2, . . . , F6}←3DFeatureSpace({P2, . . . , P6})
• 3: Initial3DGau←Random Initialization
• 4: Result3DGau←Decoder({F2, . . . , F6}, Initial3DGau)
• 5: Result2DGau←Splatting(Result3DGau)
• 6: GT2DGau←Transform(GT)
• 7: L←Loss(Result2DGau, GT2DGau)
• 8: return L

where (i, j) ∈ [0, WF ]×[0, HF ]. (·)◦2 denotes squaring each element. sum(·) denotes the summation of all elements. The Gaussian weight on the z-axis is calculated as follows:

The sampling area Gxy and weight Gzl assigns high importance to spatial locations near the center (cx, cy, cz). The 3D Gaussian sampling is calculated as follows:

where fl denotes the sampled features from Fl. Inspired by RRoIAlign() [Ding et al., 2019], we introduce GauAlign(), which first crops main regional features based on Gxy and then resizes them into a uniform size of 7 × 7 with 256 channels using bilinear interpolation. This operator naturally aligns the oriented Gaussians with axis-aligned features. Last, the sampled features f are obtained by summing fl from all levels, calculated as f = ∑6l=2 fl.

Detection decoder. The decoder stacks multiple layers, each integrating 3D Gaussian sampling, self-attention [Carion et al., 2020], and dynamic convolution [Sun et al., 2021]. The f sampled through 3D Gaussian sampling is passed into self-attention for the relations between Gaussians. Dynamic convolution further enhances feature fusion. Finally, a feedforward network outputs 3D Gaussians as predictions.

3.3 Training

The training procedure for GSDet is shown in Algorithm 1. Given input images and ground truth (GT), the model is trained to predict both locations and classes of objects.

Loss for 2D Gaussians. We assign multiple predictions to GT and apply the set prediction loss [Carion et al., 2020]. We measure the distance and overlap between the predicted and GT 2D Gaussians. The GT 2D Gaussians are transformed from GT annotations. Specifically, the Gaussian Wasserstein distance (gwd) is employed to quantify the distance [Yang et al., 2021b], and the SkewIoU is used to calculate the overlap [Yang et al., 2023]. Additionally, Focal loss [Lin et al., 2017b] is used for classification. The loss is formulated as:

where δgwd, δskewiou and δcls are weights of each component. We set δgwd = 2, δskewiou = 5, and δcls = 2 by default.

75.4 75.6 75.7 75.7 75.7

500 700 900

73.9 74.1

74.1 74.1

73.4 73.5 73.5 73.6

73.5

73.0

1

2

3

4

5

number of iteration steps

Figure 4: The dynamic number

of iterations in adaptive control

during inference.

ARS-DETR

GSDet

71.9

71.5

Ntrain > Neval Ntrain < Neval

500 700 900 1300 2000

number of Gaussians /

Q

ueries

Figure 5: GSDet vs. ARS-

DETR with dynamic number of

Gaussians during inference.

number of Gaussians

Figure 6: Dynamic number of Gaussians during inference.

3.4 Inference

Adaptive control of Gaussian. We can evaluate GSDet with a dynamic number of iterations by reusing the decoder during inference. After each iteration, some predicted 3D Gaussians focus on foreground objects, while others are located in the background. Predicted Gaussians with lower confidence scores are replaced with random Gaussians for the next iteration. Meanwhile, high-confidence Gaussians are retained for final results and then replaced with relatively random Gaussians centered within their areas for the next iteration.

Dynamic number of Gaussian. Thanks to the random initialization, we can evaluate GSDet with a dynamic number of random Gaussians during inference which do not need to be equal to the training stage. In contrast, previous methods rely on the same number of anchors or queries during training and inference, which is inflexible.

4 Experiments

4.1 Experimental Setups

Datasets. We conduct extensive experiments on three datasets DOTA-v1.0 [Xia et al., 2018], DOTA-v1.5 [Xia et al., 2018] and DIOR-R [Cheng et al., 2022a].

• DOTA-v1.0 [Xia et al., 2018] comprises 1,869 images in the trainval set and 937 images in the test set, annotated with 188,282 instances across 15 categories.
• DOTA-v1.5 [Xia et al., 2018] dataset extends the DOTA-v1.0 dataset by adding a new category named "Container Crane" while keeping the same images. The number of instances is increased to 403,318 in total.
• DIOR-R [Cheng et al., 2022a] dataset consists of 11,725 training images in the trainval set, 11,738 test images in the test set and 192,512 instances belonging to 20 categories.

Implementation details. Our code is built on MMrotate with pytorch. The optimizer AdamW [Loshchilov and Hutter, 2018] is used with the learning rate as 2.5 × 10-5 and the weight decay as 10-4 . All models are trained with the batchsize 4 on two Nvidia 2080ti (2 images per GPU). The training schedule is 24 epochs, with the learning rate divided by 16 and 22 epochs. Data augmentation strategies contain only random flips. For DOTA-v1.0 and DOTA-v1.5 datasets, images are cropped into patches of 1024×1024 with overlaps of 200. The predictions of DOTA-v1.0/v1.5 are submitted to the official website for evaluation.

Evaluation metrics. The average precision at various intersections over union (IoU) thresholds is utilized to measure the precision and recall rates of the detected objects, including AP50, AP75, and mAP. Additionally, frames per second (FPS) is employed to evaluate the speed of methods.

4.2 Properties of GSDet

Adaptive control. We investigate the effect of adaptive control by increasing the number of iterations from 1 to 5 during inference, as shown in Figure 4. The model is trained with 500, 700, and 900 Gaussians, respectively. As the number of iterations increases from 1 to 5, the model achieves a 0.3% AP50 improvement with 900 training Gaussians and a 0.6% AP50 improvement with both 500 and 700 training Gaussians.

Dynamic number of Gaussians. Figure 5 compares our GSDet with ARS-DETR [Zeng et al., 2024]. Both models are trained with 900 Gaussians or queries and evaluated using 500, 700, 900, 1300, 2000 Gaussians or queries. To enable ARS-DETR to evaluate with 1300 or 2000 queries, we employ two strategies: cloning existing queries with nonmaximum suppression and concatenating random queries. With the former strategy, ARS-DETR experiences a slight decrease in accuracy, whereas the latter strategy results in a significant accuracy drop. In contrast, as the number of Gaussians increases from 500 to 2000 during inference, GSDet achieves a 1.6 AP50 improvement.

Additionally, we train GSDet with 500, 700, and 900 Gaussians and evaluate its performance across varying numbers of Gaussians during inference. Figure 6 illustrates that as the number of Gaussians increases from 100 to 2000, the AP50 consistently improves, demonstrating the flexibility.

4.3 Main Results

We compare GSDet with modern CNN-based one-stage, twostage, and transformer-based detectors. We train and evaluate GSDet with 900 Gaussians in both Table 1 and 2. Table 1 presents the detailed results for each category on DOTA-v1.0. GSDet achieves an AP50 of 75.44% with one iteration and 75.74% with three iterations using the ResNet50 [He et al., 2016]. Notably, its performance improves further when using the Swin-T [Liu et al., 2021]. Furthermore, with a multi-scale training strategy, GSDet achieves an AP50 of 80.04%.

Table 2 provides the AP50, AP75, mAP, and FPS results on DIOR-R, DOTA-v1.0, and DOTA-v1.5. GSDet achieves a AP50 of 70.05% on DIOR-R, 75.74% on DOTA-v1.0, and 67.84% on DOTA-v1.5 with three iterations. These results confirm that Gaussian splatting for oriented object detection outperforms both CNN-based and transformer-based methods. Additionally, the inference speed of GSDet is faster than that of two-stage and transformer-based methods owing to random initialization and simple architecture.

4.4 Ablation Study

Gaussian initialization. We study the effect of different Gaussian initialization strategies in Table 3, including: (1) "Center" means all Gaussians are located in the central region, with a height and width set to 10% of the image size; (2) "Full" indicates that all Gaussians are initialized to cover the entire image; (3) "Grid" means that each Gaussian is distributed within each grid, with a total of 30×30 grids arranged across the image; (4) "Isotropic" signifies that all Gaussians are isotropic and randomly initialized. The performance of the "Center" and "Full" strategies is notably poor, with AP50 of only 11.60% and 15.11%, respectively. We argue that Gaussians under these two strategies can not effectively cover potential object regions. In contrast, the "Random" initialization achieves the best performance, with an AP50 of 75.44%, followed by the "Isotropic" and "Grid" strategies. The results indicate that anisotropic random Gaussians are more likely to align with potential object areas.

3D Gaussian representation and sampling. We evaluate the impact of 3D Gaussian representation and 3D Gaussian sampling in Table 4. This ablation begins with an additional angle branch [Yang and Yan, 2020] and a set of randomly SV

LV

PL

GTF

SBF

BD

BC

HB

Table 6: The comparison between cross-attention and dynamic conv. on DOTA-v1.0.

Method AP AP75 AP50

Feature pyramid 45.13 47.17 73.21 3D feature space 47.77 52.15 75.44

Table 7: The comparison between feature pyramid and 3D feature space on DOTA-v1.0.

initialized rectangles, achieving an AP50 of 67.01%. Under these conditions, learning object orientation proves challenging. Incorporating the 3D Gaussian representation results in a 7% improvement, demonstrating the feasibility of Gaussian splatting. Further applying 3D Gaussian sampling results in a 1.43% improvement, confirming that 3D Gaussian sampling captures spatial information more effectively than relying solely on GauAlign().

Self-attention and dynamic convolution. We check the effect of self-attention and dynamic convolution in the decoder, as shown in Table 5. Our method achieves an AP50 of 61.91% without self-attention and dynamic convolution. Sequentially incorporating self-attention results in AP50 of 66.21%. When both of them are applied, the model achieves an AP50 of 75.44%. Additionally, we compare an alternative implementation, replacing dynamic convolution with crossattention, as shown in Table 6. The performance degrades with cross-attention. We argue that the input keys and values here are not processed by a standard transformer encoder.

3D feature space. We compare the effect of the 3D feature space with the feature pyramid {P2, . . . , P6}, as shown in Table 7. In this ablation study, we represent oriented objects with 2D Gaussians and employ RRoIAlign() to extract features from transformed rectangles in {P2, . . . , P6}. The results indicate a 2.23% drop in AP50 when using the pyramid, attributed to the lack of spatial information.

Gaussian update strategy. Table 8 illustrates the effect of the update strategy for predicted high- and low-confidence 3D Gaussians in adaptive control. The results indicate a 0.66% drop in AP50 with more iterations, when the output of the current step is directly used as input for the next step. We argue that the output distribution is not learned during training. By gradually applying the two update strategies, our GSDet achieves performance gains with only a slight drop in FPS, which verifies the effectiveness of adaptive control.

Qualitative results. We visualize the Gaussians and predictions in Figure 7. The initial Gaussians vary each time due to randomness. After refinement, the Gaussians focus more effectively on potential object areas. The predictions demonstrate that GSDet accurately detects dense, large scale and aspect ratio objects. In addition, Figure 8 illustrates various randomly initialized Gaussians and their corresponding predictions. We observe that initial Gaussians positioned near objects contribute to the final predictions, and GSDet effectively handles diverse inputs. Interestingly, relatively largescale Gaussians are able to predict small-sized objects.

5 Conclusion

This paper introduces GSDet, a novel detector that is the first to formulate oriented object detection as Gaussian splatting. By leveraging anisotropic 3D Gaussians for object representation within a 3D feature space, GSDet effectively captures spatial information through Gaussian sampling to optimize Gaussians' attributes. 3D Gaussians are projected onto the image plane to form 2D Gaussians, which are then transformed into oriented boxes. Benefiting from the random initialization, the proposed Gaussian-to-box paradigm eliminates the reliance on pre-designed object candidates, allowing for adaptive control and dynamic number of Gaussians during inference. Extensive experiments validate the superior performance of GSDet over mainstream methods.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62272461, Grant 62172417, and Grant 62277046; in part by the Double FirstClass Project of China University of Mining and Technology for Independent Innovation and Social Service under Grant 2022ZZCX06; in part by the Six Talent Peaks Project in Jiangsu Province under Grant 2015-DZXX-010 and Grant 2018-XYDXX-044; and Funded by the Graduate Innovation Program of China University of Mining and Technology and the Postgraduate Research & Practice Innovation Program of Jiangsu Province under Grant 123625313.