Beyond the Map: Learning to Navigate Unseen Urban Dynamics Using Diffusion-Guided Deep Reinforcement Learning

1 Introduction

In recent years, Autonomous Driving (AD) has gained significant research attention due to advancements in Imitation Learning (IL) and Deep Reinforcement Learning (DRL) techniques. AD systems require a comprehensive understanding of the driving environment to ensure safety and effective generalization for deployment in real-world settings. IL uses supervised learning to make a driving policy based on expert datasets. [Shafiullah et al., 2022; Wang et al., 2023; Jain and Unhelkar, 2024]. The goal of IL is to train an agent to replicate the behavior of human experts as closely as possible. However, IL algorithms have the following limitations: 1) The driving policy is inherently restricted to the expert's performance, and it is impractical to gather expert data for every conceivable driving situation, and 2) IL algorithms often experience a distribution mismatch between training data and real-world scenarios, as they rarely encounter failure situations during training. This lack of exposure prevents the system from learning how to manage such cases effectively.

To address the limitations of IL, researchers have shifted their focus to DRL techniques to develop driving policies through direct interaction with the environment. These techniques utilize reward mechanisms to evaluate optimal actions in various environmental states, such as dense traffic and dynamic weather conditions (e.g., rain, fog, and cloudy) [Ahmed et al., 2021; Buddareddygari et al., 2022; Wu et al., 2022; Zhang et al., 2023b; Coelho et al., 2023; Chowdhury et al., 2024]. The main goal of DRL techniques is to maximize expected cumulative rewards. Since the agent learns by interacting with the environment, DRL alleviates distribution mismatch issues and is not constrained by the performance of any expert. However, DRL suffers from low sample efficiency and requires significantly more data than IL to achieve convergence due to the extensive exploration needed during training to understand and navigate the environment completely. In advanced methodologies, researchers are focusing on integrating data augmentation, IL, and DRL techniques to develop robust and efficient DRL systems capable of effectively managing the complexities of real-world driving environments [Yuan et al., 2022b; Liu et al., 2024; Wang et al., 2024b; Hansen and Wang, 2021]. Despite recent progress, the widespread deployment of Autonomous Vehicles (AVs) on public roads remains a challenge. The main difficulties stem from dense urban traffic environments characterized by numerous dynamic entities, such as cars, bicycles, and pedestrians, as well as complex road layouts and intricate user interactions. Existing solutions sometimes make incorrect decisions in these complex and dynamic settings, potentially leading to serious accidents or traffic violations.

A survey report by the AAA Newsroom highlights the significant challenge of building public trust in autonomous vehicles, particularly due to recent incidents that have heightened safety concerns among the public. The statistics indicate potential risks for the future of the autonomous industry. Therefore, this research aims to develop a generalized solution for handling environmental uncertainties in AD systems, primarily focusing on enhancing their performance in dynamic environments. We use the CARLA simulation environment to emulate real-world conditions like varied towns, weather, and highway scenarios. To advance the performance of AVs, we propose the following contributions:

• We propose the Diffusion-Guided Deep Reinforcement Learning method to enhance generalization in the occurrence of unseen events and reduce dependency on imitation learning by integrating a diffusion model with the Soft Actor-Critic (SAC) method.
• We introduce a novel modified Partially Observable Markov Decision Process (mPOMDP) to optimize the behavior of the SAC policy for managing uncertainty in the dynamic environment. This design enhances the policy network's ability to make optimal decisions in closely related states.
• We conducted extensive testing to evaluate the effectiveness of the proposed method in various CARLA environments. Our approach outperformed the benchmarks in NoCrash scenarios (Empty, Regular, Dense) in both the Towns (Town01 and Town02), resulting in significant improvements in pedestrian safety. The pedestrian impact was reduced to 0.01/km, and the task completion success rate increased to 95%.

2 RelatedWork

This section presents a thorough review of existing literature on IL and explores methods to improve generalization through DRL techniques.

2.1 Imitation Learning in Driving

The effectiveness of IL has been widely acknowledged in AD, enabling the development of driving policies applicable in simulated and real-world urban driving scenarios. Learning from All Vehicles (LAV) [Chen and Kr¨ahenb¨uhl, 2022] enhances sample efficiency by incorporating behaviors from all vehicles in the scene. Similarly, Learning by Cheating

(LBC) [Chen et al., 2020a] and Roach [Zhang et al., 2021] adopt a RL coach, trained on-policy with privileged information, to guide the learning process. The CaT (Coaching a Teachable student) framework facilitates efficient knowledge transfer from a privileged teacher to a sensorimotor student, utilizing ResNet architectures for effective learning [Zhang et al., 2023a]. The Behavior-Aware Trajectory (BAT) prediction model [Liao et al., 2024b] employs LSTM and attentionbased techniques to manage prediction uncertainties and understand interactions. It mainly addresses issues with noncontinuous behavior labeling. The RLfOLD [Coelho et al., 2024] integrates IL with DRL by leveraging online demonstrations to bridge the distribution gap between demonstration and training environments. Furthermore, researchers have also focused on integrating diffusion models with IL. The Diffusion Q [Wang et al., 2022] conceptualizes the diffusion model as a policy, incorporating regularization and maximizing the action value function to optimize action selection. Diffusion-BC [Pearce et al., 2023] improves traditional behavior cloning by employing the diffusion model as a policy, addressing expressiveness limitations. Additionally, CDSTraj [Liao et al., 2024a] combines the Diffusion Module and the Spatio-Temporal Interaction Module to predict future traffic scenarios, enabling safe and efficient navigation in dynamic environments. DiffAIL [Wang et al., 2024a] enhances adversarial IL by integrating the diffusion model, strengthening the discriminator's ability to capture and represent complex distributions effectively. However, despite the advancements in IL approaches, a significant challenge persists in addressing the distribution gap between demonstration datasets, and few works have explored the effectiveness of IL models for generalization. This gap remains a key challenge to the effective deployment of AD systems in real-world scenarios and highlights the need for further research.

2.2 Enhancing Generalization Using DRL

Researchers have explored various approaches to enhance generalization capabilities in visually diverse environments. Data Augmentation (DA) and Domain Randomization (DR) have proven two effective methods for this purpose. [Hansen and Wang, 2021] employed a BYOL-like [Grill et al., 2020] architecture to separate augmentation from policy learning, enhancing generalization. To manage high variance during DA, DrQ [Yarats et al., 2020] updated the temporal difference loss using augmented Q-values and target Q-values. SVEA [Hansen et al., 2021] identified two pitfalls causing instability in DrQ and proposed augmenting Q-values only during training. [Yuan et al., 2022a] proposed a task-aware data augmentation method using the Lipschitz constant to maintain training stability. Additionally, [Hafner et al., 2023] worked on improving sample efficiency for learning from pixel-based observations. [Ma et al., 2024] proposed Random PadResize (Rand PR) and Cycling Augmentation (CycAug) to enhance the efficacy of DA by improving spatial diversity and maintaining data distribution consistency. The most popular approach in domain randomization involves training a DRL agent on numerous source domains to develop a generalized policy, as demonstrated by Andrychowicz et al. [Andrychowicz et al., 2020]. This method helps the agent to ignore irrelevant variations and focus on common features. [Lee et al., 2024] introduced a spatial domain adaptation algorithm that enhances images through random transformations such as flipping, cropping, and rotating. These techniques use deterministic transformations, which sometimes fail to capture the continuous nature of state transitions and unexpected events in a dynamic environment. As depicted in Figure 2c, the proposed method learns smooth interpolation between states and allows the model to develop more robust representations of the states.

3 Methodology

In this section, we introduce the modified Partially Observable Markov Decision Process (mPOMDP) method for improving generalization. Subsequently, we detail the integration of the diffusion model with the Soft Actor-Critic Deep Reinforcement Learning framework.

3.1 ProblemDefinition

Autonomous vehicle (AV) frequently encounters challenges in interpreting scenes from images due to the dynamic nature of the environment (e.g., rapid movements of vehicles and pedestrians) and its inherent uncertainties. Consequently, the sequential decision-making process for AVs can be modeled as a mPOMDP, characterized by the tuple (S,O,A, r, P, γ). Here, S = {s1, s2, . . . , sNS } represents the finite set of possible states of the vehicle and its environment, O = {o1, o2, . . . , oNO } denotes the finite set of possible observations derived from sensor data, where o2 is derived from the diffusion generated latent representation of o1 and the initial original latent representation (o1), and A = {a1, a2, . . . , aNA } is the finite set of actions the vehicle can take. The reward function r : S×A → R assigns rewards based on actions taken in specific states, such that r(s, a) ∈ [0,Rmax], where Rmax ∈ R+ is the maximum possible reward. The state transition function P : S ×A× S → [0, 1] gives the probability P(st+1 | st, at) of transitioning from state st to state st+1 after taking action at. The observation function O : S×O → [0, 1] defines the probability O(ot | st) of making observation ot given the current state st. Finally, γ ∈ [0, 1] is the discount factor that places higher importance on immediate rewards over future rewards. Our main objective is to determine an optimal policy π∗ that maximizes the expected cumulative reward over time. Mathematically, this can be defined as:

where the actions at are sampled from the policy π(· | st) and the states st evolve according to the transition probabilities P(st+1 | st, at), starting from an initial state s0 ∈ S. The optimal policy π∗ specifies the best action at to take in each state st to maximize the expected discounted sum of rewards over a time horizon T .

3.2 Use Case

Making complex sequential decisions in a non-deterministic, stochastic environment is a key challenge in autonomous driving (AD). Traditional methods often approach this task using Markov Decision Processes (MDPs) or POMDPs. These methods typically assume that the next state is fully determined by the current state and the action taken, as shown in Figure 2a. However, this assumption can be inadequate in AD, where sudden or significant state transitions may lead to unsafe conditions, such as collisions. Real-world driving dynamics require state transitions to be smooth and constrained to ensure safety and stability, as represented in Figure 1. To address these challenges, the proposed mPOMDP framework allows the vehicle to make optimal decisions by analyzing original and diffusion-generated observations as illustrated in Figure 2b. These generated observations enable the model to learn a more robust, generalized policy by exposing it to uncertainty. The primary objective of integrating the diffusion model within the mPOMDP is to create anticipated future states with controlled uncertainty.

3.3 Encoder Block

The AV decision-making process involves interpreting observations ot and selecting actions at to maximize cumulative rewards over time. Each observation ot at timestep t is defined as: ot = {(It-k,Wt-k,Vt-k)}1k=0 , where It-k ∈ R3×256×256 represents the image from the vehicle's camera, Wt-k ∈ RN×2 denotes the 2D coordinates of N = 10 future waypoints, and Vt-k ∈ R2 includes the vehicle's speed and steering angle. The agent computes an action at = (throttle, brake, steering), where throttle, brake ∈ [0, 1] and steering ∈ [-1, 1], based on these observations. The environment provides a reward rt and the next observation ot+1. The throttle and brake values are determined by a Proportional-Integral-Derivative (PID) controller that aligns with the predicted target speed from the policy network. To handle partial observability, we transform observations into latent representations. We used a similar setup for the encoder configurations based on the approach defined in RLfOLD [Coelho et al., 2024]. The image encoder fimg processes the image sequence to generate latent features it. This encoder uses convolutional layers enhanced with Adaptive Local Signal Mixing (A-LIX) and data augmentations such as color jittering and Gaussian blur to improve feature extraction and reduce overfitting. The waypoint encoder fwp applies 1D convolutions to the waypoint coordinatesWt, producing a latent representation. The vehicle measurements encoder fveh uses a Multi-Layer Perceptron (MLP) to process the concatenated vehicle data. The overall latent state ot is formed by concatenating these individual latent features: ot = (Ih,Wh, Vh)t, where ot ∈ Rdtotal , with dtotal representing the combined dimensionality of the latent features from the image, waypoint, and vehicle measurement encoders. These latent representations ot encapsulate the critical information from the environment and vehicle state, allowing the policy network to take optimal actions.

3.4 Diffusion for Latent Space Uncertainty

Diffusion models are generative methods that iteratively transform Gaussian noise into a desired target distribution, potentially incorporating contextual information as needed. We modified the output layer of the DiffAIL [Wang et al., 2024a] model to process encoded latent representations and generate representations that are closely aligned with some uncertainty. For simplicity, define ot = (Ihi,Whi, Vhi)t, where t denotes the time step in the diffusion process, and i indicates the agent environment steps at which a latent representation is used for action in the trajectory. The forward diffusion process is parameter-free and is given by:

where, γt represents the variance at time step t. The reverse process of the parameterized diffusion model is defined as:

The covariance matrix is fixed as Σψ(ot, t) = σ2 t I = γtI, making it non-trainable and represented by a predefined schedule. According to the forward process, ot can be derived from ot-1 at any time step t. The mean of the reverse process is expressed as:

where, δt = ∏ts=1 (1 - γs) and ϵψ(ot, t) represents the predicted noise at time step t. The diffusion model over stateaction pairs is formulated to predict the noise at each time step in the reverse diffusion process:

Equation (5), is used to train the diffusion model to align with ot. We incorporate the predicted noise into the latent features to enhance the policy network's ability to generalize across different scenarios. The term (√δt ·ot+√1 - δt ·ϵ) represents the noisy latent state generated by the forward pass, referred to as opred t . In the reverse pass, the diffusion model learns the added noise, denoted as ϵpred. Therefore, we can obtain o′t by subtracting the learned noise ϵpred from opred t . Once the diffusion model is end-to-end trained, o′t will converge to ot. Therefore, o′t can be mathematically represented as:

where, o′t represents the adjusted latent feature, δt is the parameter of the diffusion process, and ϵpred is the predicted noise. Eq. (6), then used to guide the policy network in selecting new actions based on this input, effectively reducing the distribution gap between the actions taken by ot and o′t . This process is detailed in the following subsection.

3.5 Diffusion-Guided Soft Actor-Critic Method

Soft Actor-Critic [Haarnoja et al., 2018] method is a modelfree, off-policy actor-critic DRL algorithm designed to optimize a maximum-entropy objective with a discount factor γ. It learns two Q-functions Qθ1 and Qθ2 , a stochastic policy πϕ, and a temperature parameter α. Our approach bypasses the use of a value network and instead trains two Q-networks by optimizing them based on the one-step soft Bellman residual:

The policy loss function for the SAC method is enhanced by integrating diffusion-based latent features from Eq. (6), aiming to increase the expected return and promote exploration via entropy regularization. Therefore, the policy loss function can be defined as:

The policy loss function Lπϕ in (9), combines the expected return, which drives the policy network to maximize rewards using KL divergence, weighted by λ, to regularize the policy's distribution. For Gaussian policies, the KL divergence between the two distributions can be modeled as πϕ(·|ot) ∼ N(µot , σ2 ot ) and πϕ(·|o′t ) ∼ N(µo′t , σ2 o′t ). The higher KL divergence value helps the agent stay consistent with previously learned policies, particularly in new or uncertain states. As the diffusion model continues to learn over time, the gradient of the KL divergence with respect to the policy parameters ( ∂DKL ∂ϕ ), progressively decrease to zero. Therefore, we can draw the following observations: (µot - µo′t ) → 0 and σot ≈ σo′t . Hence, we can formulate:

Thus, the gradient of the policy loss with respect to ϕ simplifies, as the KL divergence term no longer majorly contributes to the gradient. Hence, the policy optimization reduces to:

Reducing KL divergence weakens the penalty on policy differences between states, allowing the policy to focus more on optimizing actions based on Q-value differences rather than maintaining consistency. This smooth transition from high to low regularization reflects the policy's shift from broad exploration to specialization, initially exploring and generalizing across states and later focusing on exploiting optimal actions as confidence grows.

4 Experiments and Result Discussion

We trained and evaluated our model using an NVIDIA L40S GPU with 46GB of RAM in the endless-full-nocrash Carla environment. The training process took 6 days and 19 hours to complete 106 environment steps. Evaluations were conducted every 20k steps, with episode returns averaged over 10 episodes in each assessment. We utilized the reward function mechanism described in [Zhang et al., 2021]. In this study, we utilized CARLA [Dosovitskiy et al., 2017] version 0.9.10, an open-source simulator designed for AD research. CARLA features high-fidelity maps with static objects, such as buildings, traffic signs, and dynamic objects. In vehicle avoidance scenarios, the ego vehicle must stop when a stationary vehicle appears 20 meters ahead and proceed once cleared. The hyperparameter configurations used during model training are provided in the supplementary file.

4.1 EvaluationMetrics and NoCrash Benchmark

To evaluate the generalization ability of the proposed model, we utilized the NoCrash benchmark [Codevilla et al., 2019a]. Which provides three traffic conditions of varying difficulty: empty (no dynamic objects), regular (moderate numbers of pedestrians and vehicles), and dense (high volume of pedestrians and vehicles). Additionally, it defines six weather conditions and features 25 routes in Town01 (characterized by one-lane roads and T-junctions) for training, while Town02, a scaled-down version with distinct textures, is used for testing. The evaluation focused on the success rate, indicating the proportion of routes completed without collisions.

4.2 Model Comparison

We compare the proposed method with recent state-of-theart (SOTA) approaches, including CILRS [Codevilla et al., 2019b], LBC [Chen et al., 2020b], WOR [Chen et al., 2021], GRIAD [Chekroun et al., 2023], CADRE [Zhao et al., 2022], and RLFOLD [Coelho et al., 2024]. Table 1, presents a comprehensive comparison of success rates on the NoCrash benchmark across these methods. LBC, CADRE, and RLFOLD utilize single-camera approaches, while GRIAD and WOR use three cameras as input. Most of these methods rely on IL in offline or online demonstration mode. A major drawback of IL is its limited generalization to unseen scenarios. IL methods mainly rely on the quality of demonstration data, which can result in unsafe or suboptimal decisions when faced with rare or unpredictable events in dense traffic scenarios. The proposed method efficiently handled this problem and choose the optimal decision close to the previous decision instead of following predefined actions. The DGDRL demonstrates remarkable efficacy across diverse traffic conditions. In the Train(W01)-Train(T01) scenario, DGDRL achieves a success rate of 95.52% in the dense task, outperforming other methods. However, it performs relatively lower in the Regular task, where GRIAD achieves a higher success rate of 98.43%. In the Test(W02)-Test(T02) scenario, where other methods struggle to handle dense traffic conditions effectively, DGDRL continues to demonstrate outstanding performance. In dense traffic scenarios, it achieves an 82.12% success rate, representing a substantial 24% improvement over RLFOLD. Similarly, in regular task conditions, DGDRL attains a 92% success rate, surpassing RLFOLD by 6%. Notably, the method maintains an impressive average success rate of 95.04%, underscoring its robust performance and superiority compared to existing state-of-the-art approaches.

Handling Unseen Tasks and Unexpected Events. Scene understanding is a crucial task for AVs operating in highly dynamic environments, where even a minor deviation in perception and interpretation of surroundings can lead to significant safety challenges for pedestrians. Figures 4 and 3 present experimental results evaluating the generalization capability of the proposed method during unexpected events and its ability to handle unseen tasks. The results show that the vehicle successfully detects pedestrians and executes appropriate safety protocols by reducing speed to a complete stop, ensuring safe road crossing for pedestrians with zero collision incidents. The vehicle then resumes its planned trajectory once the road is clear. Furthermore, we critically analyzed the captured testing videos and observed that the model, despite not being explicitly trained for lane-switching, executes smooth lane-switching tasks. These findings demonstrate the vehicle's reliability in handling unexpected events and highlight its potential for real-world AV applications. Additionally, we investigated the root cause of the 5% gap in the success rate from the perfect score. Our analysis revealed that the vehicle's behavior is inconsistent when encountering traffic lights, as shown in Figure 5. The agent stops when it detects another stopped vehicle in the lane. However, the agent sometimes passes through a red light when the road is clear. This behavior highlights the limitations of the proposed method and requires further improvements.

5 Ablation Study

In the ablation study, we evaluate the impact of varying the diffusion regularization parameter (λ), given in Eq. (9), on task completion and safety metrics. As shown in Table 2, increasing λ leads to a substantial improvement in the task success rate. Specifically, the success rate rises from 86% for the RL baseline to 92% when λ is set to 0.1. This improvement is due to the additional regularization term in the loss function, which helps the policy network generate more robust actions to state perturbations. We observed a significant reduction in pedestrian impacts and vehicle collisions, with values of 0.01 and 0.14 per km, respectively, when λ was set to 0.1. This indicates that the model generalizes effectively to unseen events, such as sudden pedestrian movements. However, when λ exceeded 0.1, the diffusion loss became dominant over the policy loss, causing the model to deviate from optimal sequential decisions. In real-world driving scenarios, state transitions are continuous in nature. For instance, when moving from sunny to rainy weather, the change occurs gradually: sunny change to cloudy, followed by light rain and heavy rain. Considering these factors, we utilized linear noise scheduling, which produced state-of-the-art results. We also tested exponential noise scheduling, which led to aggressive exploration and caused learning instability. We set the timesteps (t) to 1000 and observed that reducing the t to 600 made the model less generalized, while increasing it to 1200 resulted in no further improvement in results. Additionally, we conducted a series of testing experiments using Town03, Town04, and Town05 to evaluate the generalization capability of the proposed model to unseen dynamic environments, as detailed in the project page supplementary file.

6 Conclusion

This work introduces the Diffusion-Guided Deep Reinforcement Learning (DGDRL) framework, which integrates a Diffusion model with a Soft Actor-Critic DRL approach to enhance autonomous driving performance in complex urban environments. The framework follows a novel modified Partially Observable Markov Decision Process (mPOMDP) principle. It effectively addresses the limitations of traditional Imitation Learning by incorporating a diffusion process, which improves the model's ability to handle environmental uncertainties and unforeseen events. Empirical results demonstrate DGDRL's significant superiority over existing state-of-the-art methods, achieving an average success rate of 95.04% across diverse towns and weather conditions. The ablation study reveals that increasing the diffusion regularization parameter λ to 0.1 shows substantial improvements in task completion rates and reduces collision rates, underscoring the critical role of the diffusion model in developing robust and generalized autonomous driving policies. In the future, we plan to focus on enhancing DGDRL's capabilities to manage complex tasks, such as handling traffic lights and tunnels, along with exploring its application in other domains.

Acknowledgements

This work is partially supported by the DST-NSF Project, P-13, TPN-104422, and Meity CPS Security project AAA.22/12/2021-CSRD-MeitY.