DenseLight: Efficient Control for Large-scale Traffic Signals with Dense Feedback

1 Introduction

Reducing traffic congestion is an essential task for efficient modern urban systems. As the number of vehicles in the cities increases year by year, backward traffic coordination not only does harm to the driving experience but also aggravates air contamination with more harmful fuel emissions [Zhang and Batterman, 2013]. Alleviating traffic congestion by efficient traffic signal control (TSC) [Mirchandani and Head, 2001] is one of the most practical and economical approaches [Taylor, 2002]. Specifically, TSC aims at coordinating the traffic lights of a road network to regulate the traffic flows to minimize the average travel time of vehicles. However, the traffic data collected from a road network is usually massive yet incomprehensible [Liu et al., 2020]. Therefore, the widelyadopted TSC strategies either fix signal routine [Roess et al., 2004] or adapt the traffic signal plans in real-time according to traffic flow patterns [Cools et al., 2013].

To automatically mine useful information from the massive traffic data, more and more studies leverage the powerful representation capability of deep neural networks [LeCun et al., 2015] to learn TSC agents [Van der Pol and Oliehoek, 2016; Zhang et al., 2021; Zheng et al., 2019; Oroojlooy et al., 2020; Wei et al., 2018; Wei et al., 2019a; D'Almeida et al., 2021] through the advanced reinforcement learning (RL) methods [Mnih et al., 2015; Schulman et al., 2017; Liang et al., 2021]. A proper reward function is required for applying RL to the TSC problem and improving the policy. Since the travel time is only feasible after a vehicle leaves the road network and fails to provide instant/dense feedback to the TSC policy in time during its journey, many previous works [Varaiya, 2013; Wei et al., 2019a; Wei et al., 2019b; Xu et al., 2021] draw the traditional characteristics of traffic intersections for the reward design, such as traffic pressure and queue length at the intersection. However, most of these heuristic rewards may be biased from the ultimate goal of TSC, i.e., the average travel time minimization. The introduced biases could lead the RL methods to adjust the TSC policy in an incorrect direction.

Apart from the reward design, it is also critical to endow the RL agents with the capability of precisely predicting the future dynamics of the environment to make well-founded decisions [Van Hasselt et al., 2016; Fujimoto et al., 2018; Yang et al., 2016; Lou et al., 2020]. However, it is non-trivial for an RL agent to capture the future traffic dynamics of the intersections in the context of TSC [Chen et al., 2022]. The future arriving vehicles of one intersection may be running in a distant intersection at present. To this end, the local observations of either the intersection itself (i.e., a snapshot of vehicles at the intersection) [Varaiya, 2013; Wei et al., 2019a; D'Almeida et al., 2021] or neighboring intersections [Wei et al., 2019b; Xu et al., 2021] might be insufficient to predict the long-horizontal traffic dynamics of the intersection. Additionally, the location and traffic flow trend of an intersection also play a role in predicting the traffic dynamics of the intersection. For example, downtown intersections tend to be more crowded than suburban intersections. For example, a large number of vehicles entering an empty intersection may indicate the beginning of the rush hour. Therefore, the nonlocal observations and the location of the intersections, and historical traffic conditions are all important for a TSC agent to estimate future traffic situations more accurately.

To address the issues discussed above, we propose a novel RL-based TSC method named DenseLight, which improves traffic light coordination by exploiting dense information from both an unbiased reward and non-local intersection information fusion. Specifically, to provide dense and unbiased feedback for the policy improvement, we propose an equivalent substitute for travel time, i.e., the gap between the ideal traveling distance (i.e., the ideal distance a vehicle could have traveled at the full speed during its journey) and the actual traveling distance during the whole journey, namely IdealFactual Distance Gap (IFDG). Since the length of the factual journey of a vehicle is fixed according to its traveling lanes of the road network, minimizing IFDG is equivalent to minimizing travel time. Most importantly, IFDG can be calculated at each intersection and at each timestep. Therefore, IFDG can also provide instant feedback on the effectiveness of the control policy at each intersection.

Besides an unbiased and dense reward, DenseLight also features a Non-local enhanced Traffic Signal Control (NLTSC) agent to benefit TSC from the spatial-temporal augmented observation and the non-local fusion network architecture. Specifically, the NL-TSC agent supplements the original observation of an intersection with its location information and the previous observation so that each intersection can customize its own signal plans w.r.t. historical and spatial information. As for facilitating the TSC agents with a better awareness of the future traffic dynamics affected by other intersections, a non-local branch is proposed to enhance the local features of each intersection with the features of the nonlocal intersections. By learning to communicate the non-local information across non-local intersections, the NL-TSC agent can better predict the long horizontal traffic condition of each intersection and can thus make better coordination at present.

Overall, our contributions are three-fold: 1) we propose a novel RL-based TSC method, i.e., DenseLight, which is optimized by an unbiased and dense reward termed IFDG; 2) to better model the future accumulated IFDG of each intersection, we develop the NL-TSC agent, effectively gathering spatial-temporal features of each intersection and propagating the non-local intersection information to improve the multiagent RL policy; 3) comprehensive experiments conducted on different real-world road networks and various traffic flows show that DenseLight can consistently outperform traditional and advanced RL-based baselines.

2 RelatedWorks

Conventional traffic signal control methods. Traditional traffic signal control methods [Little et al., 1981; Roess et al., 2004; Koonce and Rodegerdts, 2008] typically set traffic signal plans with fixed cycle lengths, phase sequences, and phase splits. They heavily relied on expert knowledge and hand-crafted rules. Adaptive traffic signal control methods formulated the task as an optimization problem and made decisions according to the pre-defined signal plans and real-time data [Hunt et al., 1981; Luk et al., 1982; Mirchandani and Head, 2001; Cools et al., 2013; Hong et al., 2022]. Recently, researchers have made progress by comparing the requests from each phase through different traffic representations [Varaiya, 2013; Zhang et al., 2021].

RL-based traffic signal control methods. For individual intersection signal control, some studies investigated RL environmental settings [Van der Pol and Oliehoek, 2016; Wei et al., 2018; Wei et al., 2019a; Zhang et al., 2021], agent network design [Zheng et al., 2019], and policy transfer learning [Zang et al., 2020; Oroojlooy et al., 2020] to optimize the average travel time of all vehicles. Since the travel time can not be obtained until a vehicle leaves the road network, these methods presumed some objectives to be equivalent to maximizing the traffic throughput, thus minimizing travel time. For example, [Van der Pol and Oliehoek, 2016] utilized a combination of vehicle delay and waiting time as the reward, and PressLight [Wei et al., 2019a] introduced the pressure of intersection to be a new objective. However, these optimization objectives are not always aligned with the minimization of the average travel time. To resolve this conflict, [D'Almeida et al., 2021] defined an instant time loss for vehicles to guide the next action selection. This reward is the average loss of time at the moment when selecting traffic signals. Therefore, time loss is not able to reflect the later change in traffic congestion caused by the newly selected signals. Differently, our IFDG integrates speed over time after selecting a signal, and thus IFDG can reflect the effect of the signal.

Traffic signal coordination. To coordinate multiple intersections, a centralized controller could select actions for all intersections [Prashanth and Bhatnagar, 2010; Tan et al., 2019; Van der Pol and Oliehoek, 2016] but may suffer from the curse of dimension in combinatorially large action space. Therefore, some studies modeled single agents in a decentralized way and scaled up to multi-intersection settings by parameter sharing [Chen et al., 2020]. However, ignoring the influence of other intersections may affect the objective optimization above the whole network. Some studies used centralized training and decentralized execution [Rashid et al., 2018; Son et al., 2019; Wang et al., 2020; Tan et al., 2019]; some gathered neighboring information by concatenating neighborhood observations [Chu et al., 2019; Liu et al., 2022] or communicating through graph neural network [Wei et al., 2019b; Guo et al., 2021; Zeng, 2021; Huang et al., 2021]. However, these methods either treated other intersections equally or only considered the neighborhood, ignoring useful information from distant intersections.

3 Preliminaries

3.1 Traffic Signal Control

Conventionally, a TSC task D includes a road network and a set of vehicle flows. A road network is usually composed of N multiple intersections I = [1, ..., N] which can regulate the vehicles in their entering lanes via selecting different traffic signals. Specifically, an entering lane of an intersection is a lane where the vehicles enter the intersection from the north, south, west, or east direction. We use Lin i to represent all the entering lanes of the i-th intersection. And the exiting lanes Lout i are the downstream lanes of the i-th intersection where the vehicles leave the intersection. A traffic movement is the traffic moving from an entering lane lin i,j ∈ Lin i to an exiting lane lout i,k ∈ Lout i , which is generally categorized as a left turn, through and right turn.We define a combination of non-conflicting traffic movements as a phase a. An intersection with phase a gives the corresponding traffic movements the priority to pass through under a green light while prohibiting the others. According to the rules in most countries, right-turn traffic is allowed to pass unrestricted by signals. By convention [Zhang et al., 2021], we only consider a 4-way intersection, resulting in 12 movements and 4 candidate phases.

The goal of TSC is to learn a control policy πθ parameterized by θ for all intersections in the road network to minimize the average travel time of all the vehicles X in a road network, within a finite period TTSC, i.e. minθ 1 |X| ∑x∈X xl - xe, where xe and xl are the moments the vehicle x enters and leaves the road network, respectively. Following the conventional setting in [Zheng et al., 2019; Zhang et al., 2021], each phase persists for a fixed phase duration Tphase. To avoid confusion, we use d to denote the d-th decision, so that td+1 - td = Tphase. We use D to denote the total number of decision-making steps in each TSC episode, where TTSC = D × Tphase. We use vt(x) to stand for the speed of vehicle x at time t, and vmax to represent maximum speed.

3.2 Reinforcement Learning for TSC

In this paper, we consider a standard RL framework where an agent selects the phase for each intersection after every Tphase seconds. At td, a TSC agent observes the intersection state sd from an intersection. Then, the agent chooses a phase ad according to the policy π. The environment returns the reward rd and the next observation sd+1 ∼ T (sd, ad), where T is the environment transition mapping. The return of a trajectory after the d-th phase is ηd = ∑∞k=0 γ k rd+k, where γ ∈ [0, 1) is the discount factor. RL aims to optimize the policy to maximize the expected returns of future trajectories after observing sd.

Elements in observation and reward of TSC. In most previous TSC methods, observations and rewards originated from the traffic statistic characteristics of intersections, including the following examples. 1) Pressure: the difference between the number of upstream and downstream waiting vehicles (with a speed less than 0.1m/s) of the intersection in a time step; 2) Queue length: the average number of the waiting vehicles in the incoming lanes in a time step; 3) Time loss: sum of the time delay of each vehicle, i.e., 1-vtd /vmax, at the moment of phase selection td; 4) Step-wise travel time: the local travel time at each intersection within the duration Tphase. As proved in Appendix, step-wise travel time is an unbiased reward for TSC, however, provides sparse feedback for each intersection at each time step.

4 DenseLight

In our paper, we propose a novel RL-based TSC method, namely, DenseLight, which contains two key ingredients, i.e., Ideal-Factual Distance Gap reward and Non-local enhanced Traffic Signal Control agent, as shown in Fig. 1. In this section, we elaborate on each of them in detail.

4.1 Ideal-Factual Distance Gap

As mentioned in Sect. 3, the goal of TSC is to search for a traffic signal control policy to minimize the average travel time of vehicles. Before RL takes place in TSC, traditional methods, such as MaxPressure [Varaiya, 2013], have been developed and widely adopted in real-world road networks. The elements of traditional TSC methods like pressure and queue length have inspired the later development of RL-based methods [Wei et al., 2019a; Wei et al., 2019b; Chen et al., 2020]. By trivially adopting traditional elements as a negative reward, prior RL-based methods have witnessed a great improvement upon the traditional methods by a significant margin. However, these heuristic rewards are usually designed empirically and not derived from the target. Optimizing with these rewards might hamper the RL agent from improving in the correct direction.

To develop an unbiased and dense reward function for TSC, we propose the Ideal-Factual Distance Gap (IFDG) reward function. As the name suggests, IFDG characterizes the gap between the ideal distance that the vehicles could have traveled at full speed and the factual distance affected by the phase decision of the RL agent. Ideally, if vehicles are running at full speed, IFDG will be zero. We use the negative IFDG as the reward ri IFDG,d for i-th intersection received after d-th decision, and xi ∈ Xi to represent each vehicle at i-th intersection. Formally,

According to the Eq. (1), the vehicles that either stay or leave the intersection will contribute their distance gaps to the reward. And since the distance gap is sensitive to traffic congestion, ri IFDG,d can effectively reflect the difference in different degrees of traffic congestion caused by different phases. More importantly, IFDG turns out to be an equivalent substitute for travel time:

where the first equation is the sum of the IFDG of all vehicles. The ∑x∈X ∫ xl t=xe vt(x)dt in Eq. (2) is the sum of the traveling distance of all vehicles, which is a constant since each vehicle has a fixed traveled path in our task. And the right-hand side of Eq.(2) is linear correlative with the travel time. Thus, our IFDG provides dense feedback for RL improvement and guides the improvement in an unbiased direction.

4.2 Non-local Enhanced TSCAgent

Considering the Markov property in RL, the decision-making and the value estimation of an agent require the observation of RL policy to be self-contained. Due to the complex relations between intersections of TSC, it is not sufficient to take the local observation of the intersection to estimate the policy's future value. In this paper, we propose an Non-local enhanced Traffic Signal Control (NL-TSC) agent, which takes advantage of spatial-temporal augmented observation and non-local traffic information fusion to make a better decision and estimate a more accurate value for each intersection.

Spatial-temporal augmented observation. In a road network, the dynamics of traffic congestion at an intersection generally vary across different areas and different periods. For example, the traffic congestion in downtown intersections is usually heavier than that in other areas. And the traffic tends to be more crowded at the beginning of the rush hour and more smooth at the end of it. Therefore, the TSC agent requires not only the current traffic conditions observed in each intersection but also the location and the tendency of the traffic congestion of an intersection to make more comprehensive decisions. To this end, we propose a spatial-temporal augmented observation for i-th intersection by concatenating the original observation sid at time t with its position encoding pi and its previous observation sid-1 :

where concatenate{...} puts all vectors in the bracket into one vector and pi is a 2-D position encoding implemented as [Dosovitskiy et al., 2020]. Including the position information allows NL-TSC agents of each intersection to make decisions according to their locations. Moreover, augmented with the previous observation, the changes in traffic conditions can be captured to better predict future traffic dynamics.

Non-local information fusion. On the TSC task, vehicles arriving at one intersection may come from either nearby or distant intersections. In this sense, the relations between intersections are complex and entangled. To provide sufficient information to estimate long-horizontal values, we propose a non-local branch that can automatically determine what information to communicate among non-local intersections. Specifically, the NL-TSC agent first extracts features for each intersection i by a stack of multiple 1 × 1 convolutional layers (abbr. ConvNet) (Eq. (4)). After that, the non-local branch propagates the features of non-local-intersections by a Dense Communication Layers (DCL) which is parameterized by W ∈ R|I|×|I| , as shown in Fig. 1(c). The features of non-local intersections are fused and integrated into the original representations of each intersection according to Eq. (5). Then, the final non-local features hi non-local for each intersection are processed by another ConvNet, as formulated in Eq. (6).

However, as the size of the road network increases, W tends to be over-parameterized and degrades the expression ability of the extracted non-local features. To address this problem, we use two consecutive DCLs with two smaller parameters Wa ∈ R|I|×M and Wb ∈ RM×|I| to replace the original DCL with W, so that W = Wa ×Wb, as demonstrated in Fig. 1(b). The above operations (Eq. (5) and (6)) are repeated twice to better fuse the non-local information. To allow the model to automatically fuse local and non-local information, the NL-TSC agent also extracts the local features focusing on the intersection itself:

Finally, both local and non-local features are forwarded to the final ConvNet to predict the categorical action distribution or the future policy value for each intersection:

oi can be either the action logits logπθ(a|ˆ s) or the value Vϕ(ˆ s) where θ and ϕ are the networks parameters of the policy and value estimator, respectively. The structure of either policy or value estimator of the NL-TSC agent is sketched in Fig. 1(a). As for RL-optimization, we adopt the well-known and stable policy gradient algorithm, proximal policy optimization (PPO-Clip) [Schulman et al., 2017].

5 Experiments

5.1 Experiment Setting

The experimental environment is simulated in CityFlow [Zhang et al., 2019], which simulates vehicles behaviors every second, provides massive observations of the road network for the agents, and executes the TSC decisions from the agents. Following the existing studies [Wei et al., 2019a; Zhang et al., 2021], each episode is a 3600-second simulation (i.e., TTSC = 3600), and the action interval Tphase is 15 seconds, then the number of decision-making steps, i.e., D, is 3600/15 = 240. By convention, a three-second yellow signal is set when switching from a green signal to a red signal.

Datasets. Experiments use four real-world road networks. i) Jinan12: with 4 × 3 (4 rows, 3 columns) intersections in Jinan, China; ii) Hangzhou16: with 4 × 4 intersections in Hangzhou, China; iii) NewYork48: with 16 × 3 intersections in New York, USA; iv) NewYork196: with 28 × 7 intersections in New York, USA. For each road network, the corresponding traffic flow data are collected and processed from multi-sources with detailed statistics recorded in Tab. 1. To evaluate the performance of the TSC methods on various traffic conditions, we synthesize a novel dataset DJN12(f) with high average traffic volumes and high variances (which indicates the fluctuating traffic flow) by randomly re-sampling vehicles from the real traffic flow data and adding them back to the original data with re-assigned enter time.

Baselines. To evaluate the effectiveness of our method, we compare our DenseLight with the traditional and RLbased approaches. The results of the baselines are obtained by re-running their corrected open codes. The traditional methods are described in the following. FixedTime [Koonce and Rodegerdts, 2008]: Fixed-Time control consists of pre-defined phase plans that are fixed in duration; MaxPressure [Varaiya, 2013]: an adaptive policy that greedily selects the phase with the maximum pressure; Efficient-MP [Wu et al., 2021]: an adaptive policy that selects the phase with the maximum efficient pressure. Efficient pressure is the difference of average queue length between the upstream and downstream of each traffic movement; Advanced-MP [Zhang et al., 2021]: an adaptive method that defines the number of running vehicles within an effective range near the intersection as the request of the current phase, and defines the pressure as the requests of other phases. The policy selects the phase with the maximum request. And the advanced RL-based methods are as follows. FRAP [Zheng et al., 2019]: FRAP designs a network for traffic signal control that models the competition relations between different phases based on the demand prediction for each phase with queue length as the reward; MPLight [Chen et al., 2020]: MPLight integrates pressure into the observation and uses the reward and agent as FRAP [Zheng et al., 2019]; CoLight [Wei et al., 2019b]: CoLight uses a graph attention network to learn the influences from neighboring intersections and adopts the length of vehicles in entering lanes as the reward; Advanced-MPLight [Zhang et al., 2021]: Based on MPLight, Advanced-MPLight uses the current phase and the advanced traffic states (including efficient pressure and the number of vehicles within an effective range) as an intersection observation; Advanced-CoLight [Zhang et al., 2021]: Advanced-CoLight adds the advanced traffic states to the observation of CoLight; TimeLoss-FRAP [D'Almeida et al., 2021]: TimeLoss-FRAP uses the FRAP [Zheng et al., 2019] agent with the instant time loss of vehicles in the entering lanes as the observation and reward; DenseLight (Ours): The component of observation sid at the intersection i and step d is the same as Advanced-MPLight. DenseLight is optimized with IFDG reward and NL-TSC agent.

Training details. As for DenseLight, the training hyperparameters follow the defaults in PPO-Clip [Schulman et al., 2017] implemented in the open source platform [Weng et al., 2021]. In brief, the number of episodes in Bk for k ∈ [0, 500) is 2. The size of all hidden layers of neural networks is 64. Both the local branch of NL-TSC and ConvNetprocess are ConvNets with 2 1 × 1 convolutional layers. Other ConvNets in NL-TSC are single 1×1 convolutional layers. The size of the training batch is 64. The learning rate is 3e-4 at the beginning and linearly decays to 0. M of Wa and Wb is set as |I|. For DNY196, the batch size is set as 16 to reduce the memory occupancy, M is set as 0.1× |I| to prevent over-parameterization, and the 28×7 road network is divided into four 7×7 road networks to reduce the dimension of the state. These measures are adopted to ease the optimization without causing conflicts with our contributions. The dimension of position encoding is 16. Following the conventional settings in [Chen et al., 2020; Zhang et al., 2021], all results in the tables of experiments are the average of the final 10 evaluation episodes.

5.2 Results

We test each method in 60-minute simulations separately and report the average performance of the last ten episodes. We have following findings from the comparison results in Tab. 2.

Consistent improvement. As shown in Tab. 2, our DenseLight achieves consistent improvement by a clear margin across different road networks and traffic flows. Particularly, DenseLight hits a travel time of 800 seconds on DNY196 while the previous state-of-the-art result is about 1000 seconds, saving about 20% of the time. Another significant improvement can be observed under DHZ16(f) where DenseLight achieves nearly 10% improvement over the previous best result.

Generalization to various traffic flows. We observe that under two existing flow data DJN12 and DJN12(2) with the Jinan12 road network, different methods have close performances. However, on another similar road network, i.e., Hangzhou16, DenseLight can obtain a more significant improvement under DHZ16(f). The potential reason is that the traffic flow in DHZ16(f) shows a more diverse pattern than that of the Jinan12 road network. As shown in Tab. 1, the numbers of arriving vehicles in every 300 seconds in DHZ16(f) vary considerably. To further verify this point, we have designed a fluctuating flow DJN12(f) with its detailed statistics presented in Tab. 1. From the results of DJN12(f), we observe that the improvement of DenseLight becomes more evident.

5.3 Ablations and Analyses

In this part, we sequentially add modules for ablation.

Rewards

To evaluate the effect of reward, we compare our IFDG reward with other rewards. In the experiments, all the elements except for the reward are strictly controlled to be the same. The upper part of Tab. 3 shows that our IFDG obtains the best performance. As the size of the road network and the variance of flows increase, the advantage of IFDG becomes more significant. Especially on DHZ16(f) and DNY196, our IFDG can obtain the most clear improvements. The potential reason could be that IFDG not only aligns with the ultimate goal as deduced in Eq. (1), but also carries dense information about traffic congestion of different degrees. To intuitively demonstrate that our IFDG carries more dense information about the intersection than step-wise travel time, we show their associations with other dense rewards, respectively in Fig. 2a and Fig. 2b. We observe that IFDG is positively associated with the other dense rewards while step-wise travel time is not.

Spatial-Temporal Augmentation

To investigate the individual effect of position encoding and consecutive observation separately, we conduct experiments that concatenate the original observation with either position encoding (P) or consecutive observation (C). As shown in the middle part of Tab. 3, in comparison to the results of the "Ideal-Factual Distance Gap" which only uses the original observation, we find that TSC agents can obtain further improvements with either spatial or temporal information. Especially with position encoding (P), TSC agents can achieve consistent gains. However, we also discover that in the case of DNY196, the performance degrades with consecutive observation alone while using both information does not witness the same degradation. To this end, we suppose the potential reason is that in a large road network, the location of the intersection plays a critical role in predicting future traffic congestion. And the consecutive observations without the context of the location of the intersection, i.e., Consecutive Observation (C) in Tab. 3, might be a piece of misleading information. We visualize the curves of future congestion estimation losses with different components in Fig. 3. In Fig. 3b, the future traffic congestion prediction is worse with only consecutive observations. Moreover, from Fig. 3, we can observe that the patterns of estimation losses are in accordance with the results in Tab. 3, which implies that the average travel time has an association with the accuracy of value estimations. This is because, by using IFDG as a reward, the value is the expectation of the discounted accumulated ideal-factual distance gap, positively associated with the average travel time. This observation also justifies the benefit of our IFDG reward as an unbiased measurement of the average travel time.

Non-Local Fusion

As shown in Tab. 3, adding the non-local branch achieves consistent improvement on all tasks. Especially under DNY196, non-local information enhancement results in a breakthrough in that the average travel time approaches 800 seconds. To comprehensively analyze the advantages of our non-local branch, we conduct analyses from three aspects including the accuracy of future traffic conditions, communication mechanisms and dense communication layers.

The accuracy of future traffic conditions. From Fig. 3, we observe that, by adding a non-local branch, the value losses of DenseLight obtain further improvements in comparison with others. Especially on tasks DHZ16(f) and DNY196, the value losses when using non-local branch drop significantly, which is consistent with the results in Tab. 3 (the last row of "S.T. Aug." part v.s. "DenseLight"). Given the positive association between value using IFDG and future average travel time, this observation highlights the importance of aggregating non-local traffic conditions in predicting the long-horizon future traffic condition of each intersection.

Non-local communication. To highlight the advantage of learning to propagate and integrate information from nonlocal intersections against manually specified communication among neighboring intersections, we include two sets of experiments that use fixed attention weights, i.e., W1-hop and W2-hop, in the dense communication layer. W1-hop, for example, assigns non-zero 1 |I i 1-hop | to the W i,j 1-hop , ∀j ∈ I i 1-hop , where I i n-hop includes intersections that can be reached at most n steps from the intersection i. As shown in Tab. 3 ("1-hop", "2-hop" and "DenseLight"), with the learnable W, DenseLight can outperform both W1-hop and W2-hop consistently.

The effectiveness of dense communication layer. To investigate the effectiveness, we modify our DenseLight by replacing the non-local branch with a transformer encoder [Vaswani et al., 2017; Wang et al., 2022]. The transformer is a well-known architecture to capture the global relationships among language tokens/image patches. From the results in Tab. 3, DenseLight+Transformer can outperform DenseLight without a non-local branch consistently and has better results against both 1-hop and 2-hop in most cases. This observation again justifies the benefit of aggregating non-local information in solving TSC problems. However, efficient as the transformer is, it is usually overparameterized (the total size of parameters is about 2.8 MB) and computation-heavy, making RL difficult to improve. Different from the transformer, our non-local branch only uses a linear weighted sum to aggregate non-local information, resulting in a more slim model (the total size of parameters is about 0.2 MB) and an easier RL optimization process.

6 Conclusion

In this paper, we propose a novel method DenseLight for a multi-intersection traffic signal control problem (TSC) based on deep reinforcement learning (RL). Specifically, DenseLight optimizes the average travel time of vehicles in the road network under the guidance of an unbiased and dense reward named Ideal-Factual Distance Gap (IFDG) reward and further benefits the future accumulated IFDG modeling by a Non-local enhanced TSC (NL-TSC) agent through spatialtemporal augmented observation and non-local information fusion. We conduct comprehensive experiments on several real-world road networks and various traffic flows, and the results demonstrate consistent performance improvement. In the future, we will extend our method to learn policies with generalization ability over different road networks.

Acknowledgments

This work was supported in part by National Key R&D Program of China under Grant No.2021ZD0111601, and the Guangdong Basic and Applied Basic Research Foundation (Nos.2023A1515011530, 2021A1515012311, and 2020B1515020048), in part by the National Natural Science Foundation of China (Nos.62002395 and 61976250), in part by the Shenzhen Science and Technology Program (No.JCYJ20220530141211024), in part by the Fundamental Research Funds for the Central Universities under Grant 22lgqb25, and in part by the Guangzhou Science and Technology Planning Project (No.2023A04J2030).

Contribution Statement

Junfan Lin and Yuying Zhu have equal contributions. Yang Liu is the corresponding author. All of the authors have contributed to the experiments and writing of this work.