SDDiff: Boosting Radar Perception via Spatial-Doppler Diffusion

1 Introduction

Millimeter-wave radar for all-weather perception is increasingly attracting widespread attention in robotics, computer vision, augmented reality, and autonomous driving [Harlow et al., 2024; Zhang et al., 2022; Xu et al., 2021; Adolfsson et al., 2021]. Point cloud extraction (PCE) and ego velocity estimation (EVE) are crucial pillars of radar perception. PCE acts as a low-level sensory process, extracting fundamental object information from reflected radar signals, including position, reflectivity, and Doppler velocity. Conversely, EVE

∗Corresponding author

serves as a high-level cognitive process, leveraging elemental point clouds to infer the radar's ego velocity. As Fig. 1 shows, comprehensive PCE and accurate EVE form a solid perceptual foundation for downstream tasks, including object detection, simultaneous localization and mapping (SLAM), path planning, and autonomous navigation.

Existing studies have started to focus on PCE [Prabhakara et al., 2023; Zhang et al., 2024] and EVE [Pang et al., 2024] separately, yielding promising results. These tasks are typically treated independently, with PCE relying solely on signal intensity and EVE anchored in sparse, chaotic points processed by onboard systems. However, addressing these tasks in isolation may overlook the interplay between radar's spatial and Doppler domain features, potentially introducing additional bias. Specifically, on the one hand, non-object regions may display high reflection intensity due to the notorious "multi-path effect" [Yataka et al., 2024]. Consequently, relying solely on signal intensity may mislead PCE, impeding the balance between detection density and clutter suppression. On the other hand, sparse, noisy points with poor vertical angle resolution severely degrade EVE performance.

One of our key observations is that there is a synergy between PCE and EVE, which can reciprocally evoke potential gains from each other. We conduct a feasibility study using the Coloradar Dataset [Kramer et al., 2022], as shown in Fig. 2(a). The orientation and Doppler velocity of 3D point clouds jointly define a surface, with its precise parameters determined by the radar's ego velocity. Given various points, we estimate these parameters by Random Sample Consensus (RANSAC) [Fischler and Bolles, 1981]. As shown in Fig. 2(b), sparse, noisy points after onboard CFAR

post-processing lead to an awkward inlier rate (about 25.9% at a threshold of 0.08). Points warped by dense LiDAR point clouds show over 5× improvement in EVE, with the inlier rate reaching 98.6% indoors. This demonstrates that the synergy offers reciprocal benefits: accurate ego velocity helps filter or refine points aligned with Doppler velocity. In turn, robust and dense PCE can further enhance EVE accuracy.

To fully unlock such inspiring reciprocal potential, we introduce a Spatial-Doppler Domain Representation (SDDR) for 3D radar perception. As shown in Fig. 2(c), the radar analog-to-digital converter (ADC) signal is encoded into a 3D tensor that embodies both spatial occupancy and Doppler velocity. PCE and EVE are jointly refined through SDDR Purification Process, sculpting coarse, ghost-prone representations into fine, uncontaminated ones.

Inspired by remarkable generative capabilities of diffusion models [Ho et al., 2020], we design a Spatial-Doppler Diffusion (SDDiff) model to generate an enhanced SDDR by effectively aligning spatial and Doppler features. Specifically, we propose two key designs that allow SDDiff to seamlessly tailor to 3D radar perception. On the one hand, naive diffusion model aims to transform multi-modality in latent space, requiring a plethora of sampling steps. This high demand is attributed to sampling from a standard Gaussian distribution, which is well-suited for image synthesis. However, it introduces sampling wastage and ambiguous outcomes for the SDDR purification task. To address this, we design a directional diffusion with radar priors, transforming consistent modality in SDDR. This significantly reduces the sampling steps and facilitates finer spatial occupation. On the other hand, we incorporate iteratively refined Doppler velocity profiles into the model as conditions and apply a physical constraint to bridge PCE and EVE based on the synergy. This enables the model to learn adaptive feature representations that are resilient to density variations and ghost effects.

We summarize the contributions below:

• We propose a novel Spatial-Doppler Diffusion model to

sculpt coarse representations into fine ones for 3D radar perception. To the best of our knowledge, this is the first attempt to simultaneously attain dense point clouds and accurate ego velocity from Spatial-Doppler Domain Representation.

• We design a directional Spatial-Doppler diffusion with radar priors to streamline the sampling. This deeply reduces sampling steps and mitigates ambiguous outcomes.
• We propose Iterative Doppler Refinement, leveraging the Doppler-consistency of refined spatial occupancy to enhance the model's adaptability to density variations and ghosting effects.
• Experimental results demonstrate that our method outperforms the previous state of the arts and achieves reciprocal benefits across PCE and EVE. Additionally, we will make our self-collected dataset publicly available to the research community. This dataset facilitates further advancements in 3D radar perception.

2 RelatedWork

2.1 Traditional Model for Radar Detection

In early radar detection, low-pass filtered intermediate frequency (IF) signals from millimeter-wave radar were processed via FFT, angle estimation [Schmidt, 1986; Li et al., 2003; Roy and Kailath, 1989], and Constant-False-AlarmRate (CFAR) [Nitzberg, 1972], resulting in sparse, artifactladen point clouds. Such sparse points are insufficient for downstream tasks such as state estimation, mapping, and navigation. To generate denser radar points, some research [Lai et al., 2024; Qian et al., 2020] leverages radar motion to construct virtual antenna arrays, similar to synthetic aperture radar (SAR) imaging [Ausherman et al., 1984], to enhance angular resolution. However, these methods depend on precise motion estimation or predetermined trajectory, hindering their translation into real-world implementation. Other studies [Cen and Newman, 2019; Cen and Newman, 2018] concentrate on eliminating clutter points caused by multipath effects to enhance the quality of radar point clouds. However, they are limited to high-resolution mechanical scanning radars and unsuitable for commercial off-the-shelf radar.

2.2 Generative Model for Radar Detection

Recently, numerous studies have focused on generating denser radar points through cross-modal supervision from high-resolution sensors, such as LiDAR or depth camera. Guan et al. [Guan et al., 2020] leverages a conditional Generative Adversarial Network (cGAN) [Goodfellow et al., 2014; Sun et al., 2021] to achieve object-level imaging and recover the high-frequency shape of the specific object. To materialize scene-level imaging, milliMap [Lu et al., 2020] and RadarHD [Prabhakara et al., 2023] employ a frame-stacking strategy, combining multiple CFAR-filtered Cartesian map patches or planar Range-Azimuth (RA) maps as U-Net [Ronneberger et al., 2015] inputs, with maps projected by LiDAR points for supervision. Similarly, Zhang et al. [Zhang et al., 2024] apply a diffusion model [Ho et al., 2020] to restore radar RA maps under LiDAR supervision, subsequently generating radar points in bird's-eye view (BEV). However, these intensity-only methods overlook the significant contributions of Doppler features to the spatial point cloud's contextual understanding. To enhance 3D point clouds, Luan et al. [Luan et al., 2024] encode CFAR-processed 3D sparse point clouds into BEV images for diffusion model. Unlike the direct approaches, other studies [Cheng et al., 2022; Fan et al., 2024] predict Range-Doppler (RD) maps, and then use conventional angle estimation to derive 3D points. Nevertheless, they still resort to traditional angle estimation.

2.3 Ego Velocity Estimation for Radar

Some methods estimate ego velocity by solving a transformation between two consecutive exteroceptive measurements. General registration techniques, such as ICP [Besl and McKay, 1992], NDT [Biber and Straßer, 2003], and their variants [Censi, 2008; Segal et al., 2009], are often applied for this. However, these sensor-agnostic approaches struggle with noisy radar point clouds, which provide poor point-to-point correspondence. While [Cen and Newman, 2019] introduces a keypoint extraction and data association scheme, it is unsuitable for low-resolution commercial radars. More methods leverage Doppler velocities for 2D ego velocity estimation, typically relying on RANSAC [Kellner et al., 2013; Kellner et al., 2014] or end-to-end networks like RadarEVE [Pang et al., 2024]. However, these approaches are built on sparse and chaotic points processed by the radar's onboard system. In open environments with few points, such methods suffer significant performance degradation or even complete failure. To address these challenges, we propose SDDiff, the first method, to the best of our knowledge, that enables both dense point cloud extraction (PCE) and accurate 3D ego velocity estimation (EVE) directly from the raw ADC data of a single-chip radar.

3 Spatial-Doppler DiffusionModel

In this section, we present the Directional Spatial-Doppler Diffusion Model with radar priors in Sec. 3.1 and Iterative Doppler Refinement in Sec. 3.2, followed by an overview of SDDiff model as illustrated in Sec. 3.3.

3.1 Directional diffusion with radar priors

Spatial-Doppler Domain Representation We focus on radar's information-rich ADC data rather than the sparse points from onboard systems. To simultaneously achieve dense PCE and accurate EVE, we introduce a Spatial-Doppler Domain Representation (SDDR) designed around two key principles: 1) Capturing both spatial occupancy and Doppler velocity with minimal information loss. 2) Eliminating redundant data to reduce computational cost and memory usage when feasible. Specifically, we transform raw radar data into a 4D tensor C′ ∈ RR×A×E×D via fast Fourier transform (FFT), where the dimensions correspond to range, azimuth, elevation, and Doppler. Higher values in the tensor typically indicate a higher likelihood of a point's existence. For a spatial position sk,i,j = (rk, ai, ej), the Doppler velocity vrk,i,j is uniquely determined. Empirically, the maximum value along the Doppler axis is typically at least 6× greater than the second when a point exists at sk,i,j . Using this property, we extract the indices of peak values along the Doppler axis to determine Doppler velocities as shown in Fig. 3. Subsequently, intensity u and Doppler velocity v are concatenated into a polar SDDR C = [u; v] ∈ RR×A×E×2 , where intensity represents the spatial occupancy and the index corresponds to the Doppler velocity. This can significantly reduce the data volume and computational overhead.

Similarly, LiDAR's spatial occupancy can be represented by wrapping the points into a polar format consistent with the radar configuration. Refining PCE and EVE is defined as the SDDR Purification Process, transforming coarse, ghostprone representations into fine, uncontaminated ones. The conventional diffusion process typically transforms the target distribution into a standard Gaussian distribution over long steps, and subsequently iteratively samples a new target starting from the Gaussian noise given conditional embeddings. However, it requires a plethora of sampling steps and produces ambiguous outcomes for the SDDR purification task. In response, we define a directional diffusion process, starting from an initial state with a homogeneous radar representation. The forward and reverse processes are comprehensively presented and analyzed as follows.

Forward process Similar to DDPM [Ho et al., 2020], we construct a parameterized Markov chain {x0, x1, · · · , xT} to model the transition between radar and LiDAR distributions as shown in Fig. 4. Given radar's spatial occupancy u0 aligned with LiDAR's x0, the forward diffusion process is defined as gradually adding Gaussian noise ϵ ∼ N(0, I) to the target representation along the direction toward the radar prior u0 as follows:

where αt, λt denote pre-defined weight, variance schedule.

Theorem 1. Given the directional diffusion process with control signal u0, sampling the state xt at an arbitrary timestep t are expressed in closed form:

The proof of Theorem 1 is included in the appendices of the supplementary material. Eqn. 1 shows that directional diffusion process starts sampling from the radar prior distribution and diffuses toward the LiDAR SSDR. This facilitates fewer sampling steps and mitigates ambiguous outcomes.

Reverse process The key of the reverse process is to estimate the posterior distribution pθ (xt-1|xt,u0) with the radar prior u0. Following most of the literature on generative models [Rombach et al., 2022], we aim to choose a tractable distribution N (xt-1; µθ (xt, u0) , Σθ (xt, u0)) by optimizing the variational bound on negative log likelihood of p(x0|u0).

where DKL[·||·] denotes the Kullback-Leibler (KL) divergence.

Theorem 2. For directional diffusion, with the forward process defined in Eqn. 2, the posterior distribution of the latent variable is tractable when conditioned on x0 and radar's prior u0, and is given in explicit form as follows:

Derivation details on Theorem 2 are provided in the supplementary material appendix due to space limitations.

According to Eqn. (2)(4)(5), the mean ˜µt (xt,u0) of the posterior pθ (xt-1|xt, u0) conditioned by the previous latent state xt and the radar prior u0 is further parameterized as follows:

Therefore, given homogeneous radar prior u0, the objective function in Problem 3 is simplified as:

3.2 Iterative Doppler Refinement

Point cloud density variations across indoor and outdoor environments and multi-path-induced ghost points hinder model generalization. Drawing from Fig. 2, we design Iterative Doppler Refinement to mitigate ghost artifacts while progressively enhancing spatial representations for more accurate ego-motion estimation. Specifically, Doppler velocity, i.e. relative radial velocity vr i,j , is determined by the radar's ego-velocity vego and the target's azimuth ai and elevation angle ej when the target is stationary relative to the ground.

According to the Cauchy-Schwarz inequality, the peak height of the distribution surface of vr w.r.t. ai and ej corresponds to the magnitude of the radar's velocity, while the peak position indicates the azimuth and elevation angles of the radar's motion as shown in Fig. 4(d). Considering that static targets dominate the radar field of view in scene-level PCE, we utilize Doppler-consistency as a critic to refine spatial occupancy. Based on Eqn. 8 we train a differentiable model fψ (v) for velocity regulation. Given a progressively refined spatial representation xt, a reduction operation along range followed by softmax is applied to obtain a soft maskMt. Further, we define a Doppler-consistency loss to jointly refine spatial occupancy and ego velocity.

where ⊙ donates Hadamard product, the weights are determined by Eqn. 5.

3.3 Overview of Our SDDiff

Fig. 4 presents the overview of our SDDiff. During training, the LiDAR representations are softened with a Gaussian kernel to transition to a probabilistic spatial occupancy x0. Subsequently, they and radar priors u0 are fed into the directional spatial-Doppler diffusion module. We adopt SDDNet as the denoising model, designed on a standard 3D U-Net architecture with paird up-down blocks incorporating ResNet and attention mechanism. To optimize computational efficiency, radar Doppler profiles v are embedded w.r.t. chirp channels before injected into each 3D U-Net layer for cross-attention. Subsequently, Iterative Doppler Refinement with the latent variable xt and Doppler profile v as inputs, is initialized to mitigate density variations and ghosting effects. The overall training loss for SDDNet is formulated as:

where ω denotes the weight balancing spatial and Doppler losses. Notably, only the parameters of SDDNet are optimized during training. The inference process resembles Stable Diffusion, iteratively sampling the SDDR through Markov chains. The refined SDDR is subsequently employed for ego velocity estimation.

4 Experiments Setup

4.1 Dataset

We evaluate the proposed method using both the publicly available Coloradar dataset and a self-collected dataset across different indoor and outdoor scenarios.

ColoRadar Dataset. We conduct our method on ColoRadar Dataset [Kramer et al., 2022], which includes raw ADC sample data for single-chip radar, dense LiDAR point clouds, and odometry pose information. We use LiDAR point clouds as the ground truth for PCE and velocity solved through odometry for EVE. The ColoRadar contains both indoor and outdoor scenes with a total of 52 sequences. For a fair comparison with other learning-based baselines, we select the same 36 sequences as the training set and others for testing.

Self-Collected Dataset. We collected real-world data from diverse environments using the platform shown in Fig. 5 to assess the model's generalization. The dataset comprises 10,371 frames, with 10% used for fine-tuning and 90% for testing. Reliable odometry for EVE ground truth was obtained using Fast-Livo [Zheng et al., 2022].

4.2 Implementation Details

We implement our SDDiff using Pytorch 1.11.0 with CUDA 12.4. The parameters ω of the weighted spatial and Doppler loss are set to 0.01. We apply Gaussian filtering with sigma values of [0.2, 0.5, 1] as the filter encoder and perform offline training for the associated decoder. The Ego-Velocity Decoder is trained according to Eqn. 8, with the AzimuthElevation (AE) Doppler profile as input. The forward process variances are set to constants increasing linearly from ¯α1 = 0.01 to ¯αT = 0.99 and the noise scale λk is set to 0.1. To illustrate the effectiveness of the directional diffusion strategy, we set the sampling step size T = 20. We train SDDNet for 100 epochs on ColoRadar Dataset with AdamW optimizer and a learning rate 10-4 . It takes about 5 days to train our model with a machine using three NVIDIA GeForce RTX 4090 GPUs and Intel Xeon Gold 6226R CPU.

4.3 EvaluationMetric

Baseline. We evaluate the PCE performance of SDDiff compared to traditional method OS-CFAR [1988], as well as generative methods RPDNet [2022], RadarHD [2023] and RadarDiff [2024]. Since no generative models exist for scene-level 3D PCE, we modify the naive diffusion model with radar spatial occupancy as input for comparison with our approach, denoted as modified 3D-Diff. For EVE performance evaluation, we compare our method with ICP [1992], RANSAC [2013], and RadarEVE [2024].

PCE Metrics. We use Earth Mover's Distance (EMD) and Chamfer Distance (CD) to quantify the similarity between the extracted point clouds P and the ground truth Q. Additionally, we adopt the definition of clutter points as provided in RPDNet [Cheng et al., 2022], as follows:

where d (q, p) denotes the Euclidean Distance between p and q. Similarly, we define the 'shot' LiDAR points as those generated points that can effectively represent the scene.

We define scene representation level and valid point ratio for point cloud generation quality assessment, and effective generation density for density evaluation.

• Valid Point Ratio (VPR) is defined as 1 - |Pclutter|/|P|, reflecting the reliability of the generated point clouds.
• Scene Representation Level (SRL) is defined as |Qshot|/|Q|, quantifying the effectiveness of the point cloud generation.
• Effective Generation Density (EGD) is characterized as |(P - Pclutter|)/|Qshot|, referring to the density of the point clouds.

EVE Metrics. We evaluate EVE using Mean Absolute Error (MAE) and cumulative velocity error density in both indoor and outdoor scenarios.

5 Results and Analysis

5.1 Point Cloud Extraction Results

Qualitatively, we showcase the PCE results of various approaches, as shown in Fig. 6. Our method demonstrates superior density and accuracy in PCE compared to alternatives. Quantitatively, we compare the point clouds generated by different schemes on the Coloradar Dataset against the LiDAR ground truth, using metrics EMD and CD. As presented in Tab. 1, points generated by our method exhibit the closest alignment with the LiDAR ground truth. We also evaluate our method on the self-collected dataset to demonstrate its generalization. As shown in Tab. 2, our approach consistently outperforms others when applied to new scenarios.

Reliability and Effectiveness Analysis: Different models show variations in the number of points and clutter at different thresholds. To comprehensively assess the reliability and effectiveness of the generated point clouds, we evaluate the clutter rate, representation level, and density of various PCE methods. As illustrated in Fig. 7, 2D methods exhibit high VPR and low SRL due to the lack of elevation information. our method achieves a balanced trade-off between effectiveness and reliability, outperforming the state-of-theart baseline with a 30% improvement in VPR and a 33% increase in SRL. Moreover, our method ensures the highest valid point cloud density compared to other approaches. This is attributed to the reciprocal benefit mentioned in Fig. 2, which enhances the model's robustness to density variations and ghosting effects.

5.2 Ego Velocity Estimation Results

As shown in Tab. 3, our method improves EVE by 30% and 59% over the state-of-the-art in indoor and outdoor scenarios, respectively. This is attributed to our PCE process, which provides more accurate and reliable point clouds for EVE. In contrast, other methods degrade outdoors due to relying on the sparse points processed by the data capture card.

5.3 Ablation Study

We conduct an ablation study to validate the gains introduced by PCE module i.e. Directional Spatial-Doppler Diffusion and EVE module i.e. Iterative Doppler Refinement. As shown in Tab. 4, the PCE without Doppler information incurs a 15% reduction in PCE performance compared to the whole SDDiff. Nevertheless, it still outperforms Modified 3D-Diff, which suffers from ambiguous outcomes due to its diffusion scheme. Moreover, it speeds up inference by 3.13× compared to Modified 3D-Diff, owing to the more efficient sampling prior distribution. The EVE without reliable spatial occupancy leads to a 1.2× and 3.4× increase in indoor and outdoor errors, respectively. This demonstrates that SDDiff effectively harnesses the potential gains from the synergy between PCE and EVE.

6 Conclusion

This paper introduces SDDiff, a directional Spatial-Doppler diffusion model to simultaneously enable dense point cloud extraction and accurate ego velocity estimation. To reduce sampling wastage and mitigate ambiguous outcomes, we design a directional diffusion with radar priors to streamline the sampling. Additionally, we design Iterative Doppler Refinement to defend against density variations and ghosting effects. The experiments demonstrate that SDDiff enhances EVE accuracy while improving both the effectiveness and reliability of PCE.

Acknowledgments

This work was supported in part by National Natural Science Foundation of China with Grant 62471194, and the Fundamental Research Funds for the Central Universities.