Crowd-Steer: Realtime Smooth and Collision-Free Robot Navigation in Densely Crowded Scenarios Trained using High-Fidelity Simulation

1 Introduction

Mobile robots are frequently deployed in indoor and outdoor environments such as hospitals, hotels, malls, airports, warehouses, sidewalks, etc. These robots are used for surveillance, inspection, delivery, and cleaning, or as social robots. For such applications, robots must smoothly and reliably navigate in these scenarios by avoiding collisions with obstacles, including dynamic agents or pedestrians. The crowd density in these scenarios generally vary between 1-3 pedestrians per square meter.

In recent years, there has been significant work on learning-based collision avoidance for mobile robots operating in such dense scenarios. These include techniques based on end-to-end deep learning [Kim et al., 2018; Pfeiffer et al., 2017], generative adversarial imitation learning [Tai et al., 2018], and Deep Reinforcement Learning (DRL) [Long et al., 2017; Fan et al., 2018b]. Specifically, DRL-based methods [Chen et al., 2017; Everett et al., 2018; Long et al., 2017] have demonstrated superior collision avoidance rates, lower time-to-goal, and higher average speed of the agent when compared to traditional Velocity Obstacle (VO) methods. These methods typically use a single sensor such as a lidar, depth camera, or RGB camera for obstacle avoidance.

∗Authors contributed equally.

In practice, the DRL-based methods are trained in a simulator and the trained policy is transferred to a real robot. This strategy of DRL policy training suffers from the sim-to-real gap, i.e., the policy's navigation performance in real-life can be worse than its performance in the simulator. The main reasons for the sim-to-real gap are 1) not modeling noisy sensor data readings, friction and delays in robot hardware [Neunert et al., 2016] and 2) difficulty in recreating real-life robot-pedestrian interactions with varying complexity in the simulator.

In addition to sim-to-real transfer issues, the choice of perception sensors that were used to train the policy also affects the performance of the navigation methods. For instance, algorithms that are trained using a single lidar [Long et al., 2017] may perform well in dense scenarios but lack the ability to differentiate between animate and inanimate obstacles. Methods which use several perception sensors [Chen et al., 2017; Everett et al., 2018] exhibit good performance in moderately dense crowds but not in high density crowds, and are susceptible to perception errors. Their trajectories also tend to be jerky and oscillatory.

MainResults. We present a novel high-fidelity 3-D simulator for training learning-based navigation policies for densely crowded scenarios. In addition, we use our simulator to train CrowdSteer, a novel collision avoidance policy that fuses data from inexpensive perception sensors such as a 2-D lidar and a depth (RGB-D) camera to sense obstacle features. Our simulator helps the policy implicitly learn different kinds of robot interactions with the pedestrians, and significantly reduces the sim-to-real gap. The novel components of our work are:

• A high-fidelity 3-D simulator which accurately models real-world crowd behavior, sensor noise, frictions and delays of the robot, to close the sim-to-real gap while training DRL policies for dense crowd navigation. Our simulator also allows to incrementally vary the complexity and randomness of the training scenarios. The robot's performance in simulation and in real-world scenarios matches over 80% of the time, demonstrating our simulator's sim-to-real transfer capabilities. Previous methods perform well in simulations but exhibit oscillatory/jerky motions on real hardware.
• A DRL-based collision avoidance policy trained in our simulator that fuses inputs from a 2-D lidar and depth camera for implicitly characterizing the robot's interactions with pedestrians. The policy smoothly navigates through complex real-world scenarios.
• Reward function shaping for crowd navigation that leads to highly smooth robot trajectories which are less obtrusive to pedestrian motion. Our ablation study (Table 2) shows more than 4 times reduction in oscillations.

We evaluate the generalization and sim-to-real capabilities of the trained DRL policy on two real robots, Turtlebot and ClearPath Jackal in indoor environments such as corridors, L and T-junctions with varying pedestrian density (1-3 humans/m2 ). We also compare its performance with prior traditional methods such as the Dynamic Window Approach (DWA) [Fox et al., 1997] and a state-of-the-art learning-based crowd navigation algorithm [Long et al., 2017]. We observe that our approach surpasses these methods in terms of success rates, smoothness and shows a reduction of up to 68.16% in time to goal, and 6.12% reduction in trajectory length when compared to the current state of the art in DRL-based solutions [Long et al., 2017].

2 RelatedWork

In this section, we give a brief overview of the prior work on crowd simulation, and traditional and learning-based collision avoidance algorithms.

2.1 Sim-to-Real and Crowd Simulation

Reasons for the sim-to-real gap and methods to mitigate them [Neunert et al., 2016; Yu et al., 2019] have been discussed in several works. For training a policy for crowd navigation in simulation, the crowd's behavior must mimic real-world behavior. Works such as [Narang et al., 2015; Pelechano et al., 2007; Narain et al., 2009; Braun et al., 2003] discuss several dynamics, physiological and psychological factors that should be accounted to make crowd simulation more realistic. [Curtis et al., 2016] demonstrates a modular structure for developing a crowd simulator. We extend the concepts from these works for our crowd navigation policy training.

2.2 Navigation in Dynamic Scenes

There is extensive work on collision avoidance in dynamic scenes for robots. These include techniques based on potential-field methods [Tilove, 1990], social-forces [Helbing and Molnar, 1995], velocity obstacles and its extensions [van den Berg et al., 2009; Alonso-Mora et al., 2010; Bareiss and van den Berg, 2015; Alonso-Mora et al., 2018], etc. In terms of real-world scenarios, these methods require accurate sensing of obstacles' positions and velocities and parameter tuning that is scenario-dependent. These requirements make it difficult to directly apply them for navigation in dense crowds. The Dynamic Window Approach (DWA) [Fox et al., 1997] is another widely used method which calculates reachable dynamically-constrained velocities for collision avoidance within a short time interval. However, it does not scale well to large numbers of dynamic obstacles (Section 5.5).

2.3 Learning-based Collision Avoidance

Several works have used a single perception sensor such as 2-D lidar or raw data from a depth camera to train collision avoidance behaviors in a robot using expert demonstrations [Pfeiffer et al., 2016] and imitation learning [Tai et al., 2018]. End-to-end methods [Kim et al., 2018], and methods using deep double-Q networks [Xie et al., 2017] use RGB image data for training.

DRL-based Methods. A decentralized collision avoidance method was trained with Proximal Policy Optimization (PPO) [Schulman et al., 2017] using a 2-D lidar in [Long et al., 2017]. This approach was extended to a hybrid control architecture [Fan et al., 2018b], which switched between different policies to optimize the navigation. This approach works well in open spaces, but exhibits oscillatory and jerky motions in dense scenarios since the 2-D lidar only senses the proximity data and fails to sense more complex interactions. [Chen et al., 2017] presents a decentralized agent-level policy that utilizes a trained value network that models the cooperative behaviors in multi-agent systems. An LSTM-based strategy that uses observations of arbitrary numbers of neighboring agents during the training phase is described in [Everett et al., 2018]. Other methods [Chen et al., 2019] explicitly model robot-human interactions in a crowd for robot navigation. These algorithms use a 2-D or 3-D lidar along with several RGB cameras for pedestrian classification and have been tested in moderately dense crowds.

These methods use either basic 2.5D simulators or trajectories generated by methods such as ORCA [van den Berg et al., 2009] for policy training, which may not translate well to real-world scenarios.

3 Overview

In this section, we provide a brief overview on the multisensor based navigation policy trained in our simulator.

3.1 Multiple Sensors

To test the capabilities of our simulator, we train a policy that exploits data from two sensors (2-D lidar and depth camera) simultaneously. Our simulator is also extendable to other sensor/robot models. The lidar's raw data does not provide sufficient information to differentiate between animate and inanimate obstacles or sudden changes in the orientation of obstacles which makes it difficult to infer the direction in which obstacles are moving. Therefore, we also use images from a depth camera to capture such interactions that are useful for a robot's navigation.

2-D Lidar. Each scan/frame from a 2-D lidar consists of a list of distance values on the plane of sight of the lidar (See left scenario in Fig. 2). With its high accuracy, Field Of View (FOV), and low dimensional output data, the 2-D lidar allows us to detect clusters of closest points in the robot's surroundings.

Depth Camera. Depth images possess an additional dimension over and above a 2-D lidar. Therefore, features such as obstacle contours and changes in obstacles' poses are more prominently recorded even in low resolution images. If we consider several consecutive frames from the camera, the approximate positions, orientations and velocities of all obstacles/pedestrians in the frame can be extracted.

3.2 Robot Navigation

The dynamics of the robot we train is formulated using nonholonomic constraints [Alonso-Mora et al., 2010]. The robot knows its environment only through local observations, and it has no global knowledge of the environment, and cannot access hidden parameters of pedestrians, such as their goals. At each time step t, the robot has access to an observation vector ot which it uses to compute a collision-free action that drives it towards its goal. We also assume that the pedestrians cooperate with the robot to avoid collisions, and that data from the lidar and depth camera are synchronized.

We split the robot's observation space into four components, ot = [otlid , otcam, otg , otv ], where otlid denotes raw noisy 2-D lidar measurements, otcam denotes the raw image data from a depth camera, otg refers to the relative goal location with respect to the robot, and otv denotes the current velocity of the robot.

The action space of the robot is composed of its linear and angular velocities at = [vt , ωt ]. At each instant, the trained navigation policy πθ selects an action at to drive the robot towards its goal while avoiding collisions with pedestrians and static obstacles, until a new observation ot+1 is measured. During training, we optimize the policy πθ to minimize the arrival time of the robot to its goal. We choose Proximal Policy Optimization [Schulman et al., 2017], a policy gradient based method to train our policy in simulation. Since the current state-of-the-art DRL-based collision avoidance policy uses PPO, it helps us compare our policy against it.

4 Our Simulation Approach and CrowdSteer

In this section, we discuss the development of our simulator and our multi-sensor based collision avoidance method that is trained in it.

4.1 Pedestrian Behavior Modeling

Pedestrian velocities in crowds are dictated by several complex factors such as crowd density, stride length of a person, and need for personal space. The fundamental diagram [Narang et al., 2015] provides an inverse relationship between individual pedestrian velocities and the crowd density. To create realistic crowd navigation training scenarios, we need to account for the density-dependent behaviors based on the above said factors.

Each pedestrian in our simulation is modeled as a disk in 2-D plane with the following state space: [r ⃗ p ⃗vcurr ⃗vpref ] ∈ R7 . Here, r denotes the disc radius. ⃗ p,⃗vcurr and⃗vpref represent the current 2-D position, current velocity and the preferred velocities of the pedestrians respectively. We relate a pedestrian's natural walking velocity (V) with physiological (pedestrian's height and stride length), and psychological (need for personal space) factors using the following equation:

where S is the available space in front of the pedestrian, H (height/1.72) is a height normalization factor, α and β are constants that account for stride of the pedestrian. Accounting for these factors during policy training along with appropriate reward function shaping lead to navigation that is more pedestrian-friendly.

4.2 Scenario and Robot Model Development

An important challenge when designing a simulator and scenario for training navigation policies is to strike a balance between: (i) the generality of the scenario such that overfitting is avoided, (ii) maintaining the complexity of the training scenarios such that the policy converges. We achieve this by training the policy starting from simple to complex scenarios. We vary each scenario's complexity and randomness by using a complexity factor (c) for the different scenarios, and by correlating it with a certain attribute of the scenario.

We classify our scenarios into 3 broad categories and explain the complexity factor for each category.

Static Scenario. This category of scenarios contains only static obstacles in an enclosed space of constant area and the policy learns goal-reaching and static obstacle avoidance behaviors in these scenarios. The complexity factor c controls the number of obstacles in the scenario. As c is increased, obstacles are randomly placed in the scenario, which reduces the free space available for the robot to traverse through.

Random Static and Dynamic Obstacles. In this category of scenarios, c controls the number of static obstacles, and the magnitude of velocity of dynamic pedestrians. The simulated pedestrians use the velocity directions computed as mentioned in Section 4.1. As (c) is increased, the training scenario has more randomly placed static obstacles (less traversible space for the robot) and robot needs to handle dynamic pedestrians.

Scenario with Occluded Obstacles. In this category of scenarios, the robot learns to perform several sharp turns and avoid pedestrians who can only be observed in close proximity. This trains the robot to perform quick maneuvers in real-world scenarios. Here c is correlated with the occlusion % in the environment, and the number of dynamic obstacles. Depending on the FOV of the depth-camera, the occlusion % is defined as ( Angle not visible Camera's FOV ) × 100%. As c increases, the occlusions become more severe and the robot faces more dynamic obstacles in random intervals.

The complexity factor c gives more control on incrementally changing the randomness and complexity of the scenarios. This aids in training convergence, and since all scenarios include some randomness, overfitting is prevented.

Robot Model. We use the model of a Turtlebot in simulation for training. We compare the velocities of a real Turtlebot mounted with a laptop and sensors with a simulated robot, and use system identification techniques to deduce additional motor friction and inertia parameters that the simulated robot model must include. We also add Gaussian noise N (0, 0.2) to the depth images, and the goal location in the simulation and increased the delay in subscribing to current velocity observations. The mounting position of the depth camera is also varied during training to improve generalization. These modifications help more accurately recreate hardware phenomenon in simulation.

4.3 Reward Function Shaping

To train the basic objectives of navigation, the robot is rewarded for heading towards and reaching its goal, and penalized for moving too close or colliding with an obstacle. In addition, the robot is penalized for oscillatory velocities, and rewarded for reaching intermediate waypoints (rwp) provided relative to the robot. No global information is available.

Formally, the total reward collected by a robot i in the simulation at time instant t can be given as:

where the reward for reaching the goal (rg)ti or an intermediate waypoint is given as:

Here, pti and pwp denotes the position of the ith robot at time t and the relative position of its next waypoint respectively, and gi denotes its final goal location. The collision penalty (rc)ti is given as:

The oscillatory behaviors (choosing sudden large angular velocities ωt i ) are penalized as:

Although the penalty for collision teaches the robot not to collide, it does not specifically result in the robot maintaining a safe distance from pedestrians. The penalty for moving too close to an obstacle is given by:

where dthresh and drob denote the threshold distance that the robot needs to maintain from an inanimate obstacle at any time, and the actual distance that the robot maintains from an obstacle. S is the space needed for a pedestrian to walk without changing his/her velocity, calculated from equation 1 for a given V, α and β. This reward term is dynamic, as the value of S varies for different pedestrians. This novel addition trains the policy to navigate among humans while causing minimum disturbance to them, which previous methods have not accounted for in their training.

4.4 Network Architecture

The network architecture used for training our DRL policy is shown in Fig.3. The network has 4 branches to process each observation of the robot. We use a deeper branch with several 2-D convolutional layers to process depth images.

5 Results and Evaluations

In this section, we describe our implementation and highlight its performance in different scenarios, compare it with prior methods and perform ablation studies to highlight the benefits of our simulator, using multiple sensors and our reward function.

5.1 Implementation

We train our model in simulations that were created using ROS Kinetic and Gazebo 8.6 on a workstation with an Intel Xeon 3.6GHz processor and an Nvidia GeForce RTX 2080Ti GPU. We use Tensorflow, Keras and Tensorlayer for implementing our network. We use models of the Hokuyo 2-D lidar and the Orbbec Astra depth camera in Gazebo to simulate sensor data during training and evaluation. We mount these sensors on a Turtlebot 2 and a Clearpath Jackal robot to test our policy in real-world scenarios such as crowded corridors and occluded scenes.

5.2 Training Convergence and Data Efficiency

The convergence of our reward function versus the number of iterations for different training scenarios is shown in Fig.4. We gradually increase the complexity factor of the training scenarios and the training starts to converge at 100 iterations, and stabilizes at 200 iterations. The total process completes in six days. This increase in training time when compared with previous methods is due to the complexity and number of scenarios, and the dimensionality of the 3-D depth data used during training. However, the training time does not affect our run-time performance, which can be observed from our real-world tests.

5.3 Testing Scenario

We consider five different test scenarios that have narrower or different sections, as compared to our training scenarios. The scenarios demand tight maneuvers from the robot to reach the goal and are more challenging than existing simulation benchmarks and help to better test CrowdSteer's sim-toreal, and generalization capabilities. We define Least Passage Space (LPS) as the minimum space available to the robot inbetween obstacles when moving towards the goal. The scenarios we consider are:

• Narrow-Static: Scenario with only static obstacles where the LPS is < 0.7 meters.
• Narrow-Ped: Scenario with 8 pedestrians walking in

Figure 5: Testing the generalization of our training in a scenario with four agents (in blue) moving towards antipodal points on a circle (in yellow).

the opposite direction to the robot's motion in a narrow corridor, with an LPS of < 1.5 meters.

• Occluded-Ped: Scenario with sharp turns with static obstacles and pedestrians that are occluded by walls. The LPS is < 1 meter.
• Dense-Ped: Scenario with 18 pedestrians in a corridor of width 6 meters. Pedestrians may walk together as pairs, which require a robot to make sharp turns in the presence of multiple dynamic obstacles. The LPS is < 1 meter.
• Circle: To test the generalization of our method for multi-robot collision avoidance, we make 4 robots move towards antipodal positions on a circle. The trajectories of the 4 robots is shown in Fig. 5.

5.4 Performance Benchmarks andMetrics

We compare the benefits of our multi-sensor policy with three prior algorithms: (i) Dynamic Window Approach [Fox et al., 1997] along with a global planner which requires a map of the environment.(ii) An implementation that uses a single depth camera that was trained using PPO and our reward functions; (iii) Long et al. [Long et al., 2017], the current state-of-the-art DRL-based collision avoidance policy for dense crowd navigation, trained in a 2.5D simulator. Its real-world implementation is shown in [Fan et al., 2018a]. We use the following metrics to evaluate the performance of different navigation algorithms:

• Success Rate - The number of times that the robot reached its goal without colliding with an obstacle over the total number of attempts.
• Average Trajectory Length - The trajectory length traversed by the robot until the goal is reached, averaged over the total number of attempts. In cases where the robot never reached the goal, we report the trajectory length until it collided or started oscillating indefinitely.
• Mean Time - Average time taken to reach the goal over all attempts. If the goal is never reached in all attempts, we report the mean time until a collision or indefinite oscillation.
• Average Velocity - The average velocity of the robot until a collision occurs or the goal is reached over all attempts.

The values for the trajectory length, mean time and velocity reported when Long et al's method always failed, are until a collision or indefinite oscillation to give a sense of how much the robot traversed towards the goal before failing.

5.5 Analysis and Comparison

The results of our experiments and our ablation study to check the benefits of using multiple sensors (both lidar and depth camera) versus using one sensor (only depth camera) are shown in Table 1.

Comparisons with DWA and [Long et al., 2017]. All methods perform well in static scenarios. However, as the number and density of dynamic agents increases, both DWA and Long et al's method's performances drop considerably. This is mainly due to the high replanning time in DWA and the robot freezing problem and oscillations in the case of Long at al. DWA performs well in occluded areas due to it using a global map, while CrowdSteer has comparable performance using only local knowledge. Apart from having a much better success rate, CrowdSteer has similar performance as prior methods in terms of average velocities, better trajectory lengths and mean time to Long et al.'s method. Therefore, our method outperforms the current state-of-theart in dense and occluded scenarios. This is a testament to the benefits of our training scenarios, reward function and use of multiple sensors.

Comparing Smoothness. We observe that CrowdSteer has significantly less oscillations (sudden changes in the angular velocity of the robot) than Long et al's method [Long et al., 2017] resulting in smoother robot trajectories. CrowdSteer's novel reward structure trains the policy to maintain a safe and comfortable distance from pedestrians and avoids oscillations.

Ablation Study forMulti-Sensor Fusion. We compare the effects of using fused data from two sensors (CrowdSteer) versus a policy which uses one sensor (Depth Camera) trained with our training scenarios and reward function. Due to limited depth camera FOV, the depth camera model does not have a 100% success rate in any scenario. This also reflects in the drop in success rate in the Occluded-Ped and Dense-Ped benchmarks. However, it still manages to succeed in the dynamic testing scenarios due to our realistic training scenarios and reward function. CrowdSteer takes advantage of both the high accuracy of the lidar and the complex features extracted from the depth images and demonstrates significantly better success rates, lower mean time, and higher average velocities.

Ablation Study for Smoothness. We study the effect of our reward function on the robot's trajectory smoothness. We trained two policies, one including the penalty for oscillations in the reward function (Eqn. 5), and the other without it. The average number of oscillations in the robot's trajectory are summarized in Table 2. The models are evaluated in two scenarios: (i) Scenario without any obstacles, and the robot moves in a straight line for 10 meters towards the goal, (ii) Scenario where the robot must maneuver to avoid a static obstacle before reaching the goal. Empty/sparse scenarios are used so that turns during dynamic obstacle avoidance do not affect the number of oscillations. We observe that there is a significant reduction in the number of oscillations when the penalty is included.

Real-World Scenarios. We use CrowdSteer to navigate a Turtlebot and a Clearpath Jackal robot in crowds with varying densities (1-3 person/m2 ), as shown in the video. 1 The robots face high randomness in terms of the direction and velocities of pedestrians, which was not encountered during training. We compare the motion of CrowdSteer with Long et al.[Long et al., 2017] method in similar scenarios. Compared to Long et al., we observe that CrowdSteer has smoother trajectories in both robots and avoids all collisions with the obstacles. In occluded spaces such as corridors, CrowdSteer was able to avoid sudden obstacles which appear in places such as T and L junctions. These tests again highlight the benefits of our extensive training in occluded places, and implicit sensor fusion. Our real-world tests also demonstrate our simulator's strong sim-to-real and generalization capabilities.

1 https://www.youtube.com/watch?v=3NrhuQDsAoc

Failure Cases. CrowdSteer may not work well certain cases. It might fail in highly acute angled turns and in environments with reflective or transparent surfaces, and high interference from infrared light in the surroundings. In crowds with density > 4 people/m2 or scenarios with very minimal or narrow space for navigation, the robot may not find a collision-free path.

5.6 Improved Sim-to-Real Transfer

We compared the linear and angular velocities between a simulated and real robot when both are made to navigate in the similar scenarios with static obstacles. We observed that the velocity changes in both robots correlated over 80 % of the times. This demonstrates the accurate modeling of real-world interactions, and hardware parameters such as friction, inertia and delays. Previous DRL policies, when implemented in real hardware exhibit unnatural oscillatory behaviors since their training could not model such phenomenon due to the simplicity of their simulators and datasets. We also note that the simulator can be used to extensively collect trajectory/sensing datasets.

6 Conclusion, Limitations and FutureWork

We present a novel 3-D, high-fidelity simulator for training learning-based collision avoidance policies, with systematically increased complexity. We trained CrowdSteer, a policy that simultaneously uses multiple sensors such as 2-D lidars and cameras. In practice, our approach works well in complex, occluded scenarios and results in smoother trajectories. Our approach has some limitations and failure cases. It is susceptible to freezing robot problem in very dense settings and the computed trajectories are not globally optimal. Furthermore, the current sensors may not accurately handle glass or non-planar surfaces. These are avenues for future work. In addition to overcoming these limitations, we need to evaluate its performance in other scenarios and outdoor settings.

Acknowledgments

This work was supported in part by ARO Grants W911NF1910069 and W911NF1910315, and Intel.