Dynamic Lane Traffic Signal Control with Group Attention and Multi-Timescale Reinforcement Learning

1 Introduction

Traffic congestion has become a hot button issue in major cities all round the world. Traffic signals and traffic lanes are two common approaches adopted to help control traffic flows and improve traffic safety. The former refers to signalling devices positioned mainly at road intersections to regulate the traffic flow; and the latter partitions a road into lanes with each used by a single line of vehicles to control and guide the movements of vehicles and to reduce traffic conflicts.

Most traffic signals are controlled by simple timers, which are simple yet effective for busy areas with consistent or predictable volume of traffic. However, they may force drivers to stop unnecessarily when no traffic is present in areas where traffic is unpredictable or sporadic. Recently, intelligent traffic lights have been widely exploited, which are expected to be able to detect cars and change the traffic lights accordingly to better manage the traffic flows.

Among many efforts to equip the traffic signals with intelligence, control algorithms using reinforcement learning have achieved good results, such as IntelliLight [Wei et al., 2018] and FRAP [Zheng et al., 2019]. They use the current traffic status of an intersection as the state, select a practicable combination of the traffic signals as the action, and consider data such as queue length and vehicle travel time as the value. With deep reinforcement learning, these methods have achieved better results than traditional methods. However, they do not capture the characteristics of traffic data well. They either simply use state observation, without considering the symmetric structure of intersection; or consider only part of data but are not able to capture global information, which makes them not generalizable to different situations. Although there are algorithms on improving the generalizability, such as [Zang et al., 2020; Zheng et al., 2019], their structures are not well-designed and suffer from slow convergence speed. Consequently, low model learning efficiency or poor generalizability stop them from learning good traffic signal control strategies or being employed at intersections with different traffic flows.

In terms of traffic lanes, we focus on dynamic lanes (a.k.a. variable lanes), a special type of traffic lanes that could change the direction of the lanes dynamically, mainly during peak hours, to improve traffic flow. The lane labelled 2 in Figure 1 is a typical dynamic lane. We want to highlight that the dynamic lanes focused in this work refer to the lanes whose directions could be changed between go straight and turn left (or turn right), but not the reversible lanes whose directions could be reversed.

Dynamic lanes have achieved big success in many cities. For example, Shanghai set 7 dynamic lanes in 2005, with all making positive impact at various locations. Take the one on Tianyaoqiao Road as an example. It helps to reduce the average number of traffic light cycles that left-turning vehicles have to wait during rush hours by 50%.

Controlling of traffic signal has been well-studied, but not that of dynamic lanes. We would like to highlight that these two problems have very different characteristics. First, traffic signal controlling algorithms typically only consider the current volume of traffic flow, as the change of traffic signal is expected to have an immediate impact on the traffic flow. On the other hand, the change of the directions of dynamic lanes does affect the traffic, but the immediate results might not be positive. However, it will generate positive results which actually take some time. In other words, an algorithm controlling traffic signal cares more about immediate impacts while that of dynamic lanes focuses on non-immediate results. Second, the change of traffic signal has a much higher frequency, as compared with that of dynamic lanes. We actually celebrate their differences in this paper as they provide opportunities to develop a controlling system that could coordinate the change of traffic signal and that of dynamic lanes for a better controlling of traffic flow.

In order to enable the coordination of the above two controlling tasks, we propose a novel algorithm named MultiTimescale Group Attention Dynamic lane traffic signal control (MT-GAD). To the best of our knowledge, this is the first attempt to study the traffic signal control with dynamic lanes. Our main contributions are listed below.

• We propose MT-GAD as the first solution to coordinate the control of traffic signal and that of dynamic lanes to better control the overall traffic flows.
• We adopt group attention to achieve better traffic signal control results and enhance the model's generalizability.
• We design a multi-timescale learning method to enable the learning of signal control and that of lane control simultaneously.
• We perform an experimental study based on real data to demonstrate the superior performance of our model.

2 RelatedWork

In this section, we review existing works on traffic control with dynamic lanes and traffic signal control.

Traffic control with dynamic lanes. Controlling traffic lanes to ease traffic congestion is common. Dynamic lanes, as an example, are used in many major cities, especially in China, to improve traffic efficiency in rush hour. [Li et al., 2009] proposes a pre-signal control method based on a fictitious stop line to control dynamic lanes. [Zhou et al., 2019] proposes a traffic priority method as a solution. [Yao et al., 2018; Li et al., 2013] propose an optimization model which minimizes the total delay.

Traffic signal control. For traffic signal control, the fixedtimer which switches traffic signal phase with pre-defined time interval is simple and easy, but can't make use of real time traffic flows. SOTL [Cools et al., 2013] switches traffic signals based on vehicle numbers of approaching lanes.

Recently, many traffic signal control methods with deep reinforcement learning have been proposed. [Li et al., 2016] uses stacked auto-encoders to approximate rewards based on queue lengths. [Gao et al., 2017] uses CNN to approximate value function. [Wei et al., 2018] uses deep reinforcement learning to solve the problem. [Zheng et al., 2019] considers the signal phase competition. [Zang et al., 2020] uses MAML to learn a model with better training speed. [Kim and Jeong, 2020] learns a policy with predicted future traffic flow. [Zhang et al., 2020] tries to control traffic signals when only part of vehicles can be detected. [Rizzo et al., 2019] uses time critic policy gradient methods to handle congestion scenarios. [Oroojlooy et al., 2020] uses two attention model to handle different road number intersections and traffic signal phases. Although all the above works and our study presented in this paper focus on single intersection, there are also many works analyzing traffic signal control with multiple intersections. [Wei et al., 2019a] proposes to use pressures as the reward to perform better coordination. [Wei et al., 2019b] uses attention to learn different weights between neighbors. [Zhu et al., 2021] learns polices which only based on local observations by meta-learning. [Yu et al., 2020] uses communication and action rectification to learn multi-agent policies.

Discussion. To our best knowledge, the control of traffic signal and that of dynamic lanes are considered as two independent tasks in the literature. However, they are interrelated. An effective learning model that could perform the two tasks simultaneously is expected to offer a better control of the traffic flow and that's the focus of this paper.

3 ProblemDefinition and Preliminaries

Following existing problem definitions on traffic signal control [Zheng et al., 2019; Wei et al., 2019b], we adjust the definitions to introduce the traffic signal control problem with dynamic lanes.

Traffic flow refers to the number of vehicles passing through a road within a time unit. Given one intersection, without loss of generality, we assume that there are four approaching roads with each having three outgoing directions, excluding rarely appeared U-turns. We differentiate approaching roads by their positions relative to the intersection, i.e., North, South, East, and West, and use {S, L, R} to name outgoing directions corresponding to go Straight, turn Left, and turn Right respectively. Consequently, there are in total 4 × 3 = 12 directional traffic flows, as defined below.

Definition 1. A directional traffic flow (DTF) at an intersection is denoted as FX,Y , where X ∈ {N, S,E,W} and Y ∈ {S,L,R}.

Traffic signals are employed to regulate the traffic flow at busy intersections. A signal will be set for every DTF to control vehicles' movements, e.g., a green light signals that it's safe to pass through the intersection and a red light stops all the vehicles. For road safety, the routes of DTFs receiving the green signal shouldn't cross each other. Note that DTF FX,Y with Y = R (i.e., the outgoing direction is to turn right) will never cross any other DTFs and it usually receives green signal all the time. For other eight DTFs, there are eight nonconflict phases, with each enabling two DTFs. We plot nonconflict phases and three DTFs in Figure 2 for illustration, FS,S and FN,L conflict with each other, while FS,L in blue conflicts with neither one.

Definition 2. A traffic signal phase at time step t, denoted as Pt, sets every DTF FX,Y (Y ̸= R) one signal sigX,Y ∈ {red, green} such that i) FX,Y s with corresponding signal being green do not cross each; and ii) the number of DTFs with signal being green is maximized. In addition, when a different phase is selected in time step t + 1, i.e. Pt ̸= Pt+1, a short all-red phase Pred is used to clear vehicles going through the intersection.

For approaching roads, each normal lane has one fixed outgoing direction, usually painted on the road; dynamic lanes could change their outgoing directions dynamically. In this paper, we set {S,L} as the only available directions for the dynamic lane, which are most common outgoing directions in countries using right-handed traffic (RHT). We also assume a dynamic lane has only one outgoing direction at any time, which is consistent with the real setting used in China.

Definition 3. The outgoing direction of a dynamic lane located on road X at time step t is denoted as V X t ∈ {S,L}. For safety purpose, when a dynamic lane changes its direction, Vred is added to stop the entering of dynamic lane for several time steps.

According to the above definitions, we formalize the problem to be studied in this paper as follows. We assume that there is only one dynamic lane at an intersection, which is most common in the real world for intersections with fourlane roads.

Problem 1. Dynamic lane traffic signal control. At each time step t, given observation ot of traffic conditions at the intersection, an agent should find an optimal policy to decide Pt+1 and Vt+1 such that the reward r (to be formalized next) is maximized.

4 Method Framework

In this section, we will first explain the reinforcement learning environment setting, and then detail the proposed algorithm MT-GAD.

4.1 Reinforcement Learning Environment Setting

Traffic signal control can be defined as a Markov Decision Process (MDP), which could be solved using reinforcement learning methods. MDP is defined by 5-tuple (S, A, P, R, γ), corresponding to state space S, action space A, state transition probability P, reward function R, and discount factor γ.

We adopt Deep Q-Learning (DQN) [Mnih et al., 2015] as the reinforcement learning structure. With given state observation st and action a = {Pt, Vt} at time step t, DQN approximates the state-action function Q(st, a; θ). DQN updates the network θ using the loss function defined in Eq. (1).

The state, the action, and the reward considered in this work are presented in the following.

State. The state for every DTF FX,Y is described by following five pieces of information. 1) wX,Y , the number of vehicles currently waiting in FX,Y ; 2) qX,Y , the number of vehicles in DTF trying to pass the intersection, i.e., considering the vehicles in FX,Y approaching the intersection and vehicles in wX,Y ; 3) sigX,Y , the current signal received by FX,Y ; 4) tX,Y , the time passed since the current signal has been set; and 5) pX,Y , the predicted number of approaching vehicles in a predefined tp time steps. In addition to the above observations related to DTFs, there are two additional pieces of information, the traffic signal phase P and dynamic lane direction V . Here, we use a simple LSTM to estimate pX,Y , which could be replaced with any modern method to get a higher accuracy. As to be reported in the experiments, our method is not sensitive to the accuracy, once it is above certain threshold.

Action. The action here is to set two signals, including AP and AV . AP selects a traffic signal phase for the next time step; AV selects the direction for the dynamic lane. As there are 8 legal traffic signal phases and 2 directions for dynamic lanes, there are in total 16 different actions.

Reward. There are many available reward metrics that could be used as the performance metrics. However, many of them are based on vehicle's movement, which might not be directly available. Consequently, we use average queue length of every approaching lane between two time steps as the reward.

We now propose our method MT-GAD, whose structure is illustrated in Figure 3.

4.2 Traffic Flow Information Extraction

State information such as qX,Y and wX,Y is based on DTFs, so the first step of MT-GAD is to extract flow information to describe a state accurately. As stated previously, traffic signal control focuses more on the current flow and its impact is expected to be immediate and last only a few minutes; while dynamic lane control aims at improving the traffic condition in the near future and one setup could last multiple hours. Consequently, they take in different information as inputs.

To better consider corresponding features and avoid learning from unimportant information, we input qX,Y , wX,Y , sigX,Y , and tX,Y to traffic signal controller and input qX,Y and pX,Y to dynamic lane controller. Thereafter, we use two fully-connected layers to generate traffic signal control embedding hPFX,Y and dynamic lane control embedding hVFX,Y . We want to highlight that we purposely design our embeddings hPFX,Y and hVFX,Y to be based on every DTF, in order to remove the impact of different input order of DTFs. As there are eight DTFs, this step is trained eight times using one input, one per DTF.

4.3 Action Embedding Generation

After generating hPFX,Y and hVFX,Y , we extract embeddings for every action based on hPFX,Y and hVFX,Y . Attention mechanism [Vaswani et al., 2017] provides a good solution to make input order of DTF states irrelevant. Attention takes query Q to combine key-value pairs 〈K, V 〉 to an output based on Q.

Attention allows us to combine DTFs together based on their importance. However, attention also introduces a large number of parameters which slow down the training process significantly and increase the risk of overfitting the model. Because action embedding contains all input information, it's common to set hidden size of action embedding larger than hFi , and the biggest parameter matrix is W Q. Consequently, we propose to calculate the attention without using W Q.

We notice that attention has many parameters as it doesn't fully utilize the characteristics of traffic flows and control actions. Take the top left phase in Figure 2 as an example. Information related to DTFs FW,L and FE,L is significantly more important than the information of other DTFs. Accordingly, we group DTFs based on their impacts on the current control phase. When calculating the k-th phase P k , we split DTFs into two groups based on their signals received in P k , where group Gg contains DTFs receiving green signal, and group Gr for DTFs receiving red signal. For example, the top-left phase in Figure 2 has FW,L and FE,L in Gg, and other DTFs in Gr. Because the DTFs in the same group share the same impact on P k , we use attention to generate one embedding for every group, with all-one vector I as Q and hPFX,Y as K and V . Then, we use predefined weights wg and wr to calculate the weighted average HPk of embeddings, i.e., HPk = wg AP k g + wrAP k r , where AP k i = Attention(I,GP k i ,GP k i ), (i ∈ g, r).

For a dynamic lane, it controls two different directions with opposite impacts on its controlling. Considering one outgoing direction Y ∈ {S,L} as the selected direction, we use FX,Y to represent the selected DTF Fsel. Accordingly, FX,Y ′ with Y ′ = {S, L}\Y indicates the opposite DTF Fopp. We further introduce the coordinate DTF Fcor to represent the DTF that has the chance to receive green signal together with Fsel. For example, given a dynamic lane VE = L, we have Fsel = FE,L, Fopp = FE,S, and Fcor = FW,L. We strongly believe that Fcor, as compared with other DTFs, is more important for the controlling of the dynamic lanes.

Consequently, we extract these three DTFs (i.e., Fsel, Fopp, and Fcor) individually but consider the other five DTFs as a group, represented by an attention. We apply the same idea to traffic signal control to calculate the weighted average HVk .

Finally, we concatenate HPk and HVk with current phase embedding that goes through one fully-connected layer, , i.e., EPk = σ(W(HPK ⊕ Embedding(P)) + b), and the same as EVk . Then, we use the Dueling DQN structure to calculate QP and QV based on EP and EV separately.

4.4 Multi-Timescale Training

The differences between traffic signal control and dynamic lane control affect not only the extraction of relevant information, but also the learning process. A multi-timescale reinforcement learning method that is able to capture both close time information and future information is preferred.

To overcome the challenge of multi-timescale learning, we propose to focus on the discount factor γ for the MDP environment. When γ is small (e.g., γ = 0.8), the environment considers short-term reward, which fits traffic signal control. Meanwhile, a small γ allows the learning algorithm to ignore the reward achieved by successful direction changes of dynamic lanes, as γ Vred is expected to be so small that is almost negligible (e.g., 0.830 ≈ 0.001 based on our parameter setups). On the other hand, when γ is large (e.g., γ = 0.95), the reward achieved by dynamic lane control becomes dominant. In other words, the control of traffic signal and that of dynamic lane use γ of different scales. When calculating QP (st, at; θ) for traffic signal control, we use small γ (e.g., 0.8); when calculating dynamic lane control QV (st, at; θ), we use bigger γ (e.g., 0.95). We then calculate the Q value for combined action aij = 〈Pi, Vj〉 as Q(st, aij ; θ) = QP (st, Pi; θ)/2 + QV (st, Vj ; θ)/2.

Instead of using the standard loss function defined in Eq. (1), we consider the one defined in Eq. (2) to take into account the difference between γP and γV .

5 Experiments

5.1 Experiment Settings

Our experimental study is based on a modified version of CityFlow [Zhang et al., 2019] to support dynamic lanes. In addition to traffic signal control available in original CityFlow, we further implement dynamic lane controller.

In our experimental study, we focus on a single intersection with one dynamic lane, which is a common setup used in many existing works [Li et al., 2009; Zhou et al., 2019]. Every road has 3 lanes, with 1 turn-left lane, and 2 go-straight lanes. We want to highlight that our model considers input with respective to the directions of the DTFs, so the number of lanes is actually irrelevant. As stated before, we assume there is only one road with a dynamic lane, and the road with a dynamic lane VX ∈ {L, S} replaces one of the two gostraight lanes it has with the dynamic lane. We set Pred = 5s, Vred = 300s, and the time interval for agent to make action is 10s. The predict length of traffic flows tp = 600s.

We run the experimental study using a personal computer with AMD ThreadRipper 2950X CPU, 2080 Ti GPU, and 128GB memory. All methods are trained with 240,000 frames.

Datasets. Existing datasets available for traffic signal control problem (e.g., [Zheng et al., 2019]) only contain traffics in 3600 seconds, which is too short to evaluate the impact of dynamic lane changes. Consequently, we develop two datasets based on real taxi trajectories in Porto and Shanghai to simulate the traffic within one whole day. Porto (PT) dataset [Moreira-Matias et al., 2015] contains close to 1 million trajectories generated by 438 taxis from 1 July 2013 to 20 March 2014. Shanghai (SH) dataset contains 11,861,593 trajectories generated by 13,767 taxis in the whole month of April, 2015. We use map-matching [Song et al., 2012] to convert GPS points to trajectories. To overcome the high variance caused by small amount of taxis compared with real traffic, we ignore the day of trajectories to make data dense, and balance the average vehicle number to simulate high-volume and low-volume traffic flows. To benchmark the average volume, we refer to the data collected by the Los Angeles government [Alcazar, 2020] and we find that the most busy intersections, such as Angeles Vista Boulevard and Slauson Avenue intersection, have over 44, 000 vehicles passed in a day, i.e. 0.51 vehicle per second on average. Consequently, we use this in our setting to simulate high-demand situation, and reduce the flow by 50% to simulate low-volume traffic.

A dynamic lane should be implemented at intersections satisfying following two rules, rule 1) they have large numbers of samples to guarantee they are busy; and rule 2) the directions of their flows do change. We select two intersections randomly from each dataset based on these two rules, denoted as SH1, SH2, PT1, and PT2, and another two intersections SH3 and PT3 that not fully satisfy two rules to showcase that our algorithm is also able to learn effective signal control methods when there is no need for dynamic lane. We simulate two different flow volumes for every intersection, denoted by subscriptions low and high, e.g., SH1low and SH1high refer to SH1 with low volume and high volume respectively.

Compared methods. To evaluate the efficiency of MTGAD, we implement the following methods as the main competitors, including FixedTime and SOTL [Cools et al., 2013], two representatives of traditional methods, and PressLight [Wei et al., 2019a], CoLight [Wei et al., 2019b] and FRAP [Zheng et al., 2019], three representatives of reinforcement learning (RL) approaches. Due to space limitation, we skip their details that have been presented in related work. We also extend these methods to control dynamic lanes, except FixedTime, which is unable to control dynamic lanes. For SOTL, it changes the direction of a dynamic lane when the vehicle number on corresponding DTFs is unbalanced; for RL based algorithms, we extend their action space from 8 to 16. Last but not the least, we include a variant of MT-GAD, labelled as MT-GAD-w/o-DL, that controls only traffic signal by fixing the dynamic lane to one direction (e.g., AVt = S all the time).

Evaluation metrics. We use vehicles' average delay D as the main performance metric. D refers to the difference between the real travel time Treal spent by a vehicle from entering the intersection until passing through the intersection and the expected travel time Texp, which is the ratio of the route length to the speed limit [Wei et al., 2019c], i.e., D = Treal - Texp. As controlling algorithms do not affect Texp, this metric is the same as vehicle's average travel time.

5.2 Evaluate Results

Overall performance. The overall performance is reported in Table 1. In general, reinforcement learning algorithms perform better than traditional algorithms, because they can learn detailed strategies. When traffic flow is low, the differences among RL algorithms are rather minor, because it's easier to learn a good strategy in low traffic and the room for further improvement is very limited. When traffic volume becomes higher, our model is able to consistently outperform all the competitors by a significant margin. When we compare a method with its variance that implements dynamic lane controller, it is observed that dynamic lane is able to improve the traffic only in some cases. This is because they all only combine traffic signal control and dynamic lane control via a simple strategy, without considering the unique features of the two controlling tasks. On the other hand, MT-GAD outperforms MT-GAD w/o DL consistently when the traffic volume is high.

Generalizability. Due to space limitation, we only report the result under SH dataset. We train the models based on SH1high, and then deploy them directly to other datasets, including SH2high, SH3high and SH1rotate that rotates all traffic flows and the dynamic lane of SH1high in clockwise by 90 degree, e.g., VW → VN, FW,S → FN,S, etc. Table 2 reports the results, with % indicating the ratio of D to D′ . Here, D refers to the performance a model could achieve if the training and testing are for the same intersection, and D′ refers to the performance achieved based on the model trained on top of SH1high. MT-GAT* is a variant of MT-GAD which replaces the group attention with standard attention. Obviously, MTGAD demonstrates a superior generalizability, significantly better than other models. MT-GAT* is observed to perform well in some situations but perform badly in others. This is because the large number of parameters could easily overfit the model. With group attention, our model achieves the best generalizability consistently.

Multi-timescale training evaluation. To demonstrate the effectiveness of our Multi-timescale training, we report the values of two parameters, DV and #D.C. in Table 3. DV is the delay of vehicles driving on L or S directions of a dynamic lane, for example, for VW ∈ {L, S}, only vehicles in DTFs FW,L and FW,S are considered. #D.C. is the average number of direction changes the dynamic lane experiences during the experiment. With the environment setting, the maximum #D.C. is 80. Because of the space limitation, we only report their values under SH1high and PT1high, while we do make similar observations in other intersections. Note that DV is also relevant to the traffic signal control efficiency. Here MTGAD-0.8 and MT-GAD-0.95 refer to two variants of MTGAD that fix γ to 0.8 and 0.95 respectively, that means, we consider the traffic signal control and dynamic lane control using same timescale, either short (0.8) or long (0.95). We can find that existing RL methods fail to learn an effective strategy to control dynamic lanes, as they either never change the lane direction (i.e., #D.C. = 0) or keep changing the direction in some datasets. MT-GAD performs much better, but not MT-GAD-0.8 or MT-GAD-0.95. The latter two fail to learn direction change strategy in PT1high. Although they do change dynamic lane directions, their DV is much longer than that of MT-GAD, because with wrong timescale input, an agent can't learn strategies effectively.

The impact of traffic flow prediction. We adopt LSTM to predict the traffic flow. However, our model is not sensitive to the prediction accuracy, as long as it reaches a minimum expectation. We conduct a simulated traffic predictor S, with the accuracy a ∈ (0, 1] as an input. S takes real traffic prediction pt from dataset, and calculates the output prediction p′t as p′t = rpt, with r ∼ N (0, (1 - a)-2 ). We observe that once a reaches 0.4, its impact on the model performance is not significant.

6 Conclusion

In this paper, we explore the traffic signal control problem with dynamic lanes. To the best of our knowledge, it's the first attempt to control dynamic lane and traffic signal together. We propose MT-GAD as a solution, that uses group attention to improve learning speed and generalizability, and adopts multi-timescale learning to find optimal strategies for both traffic signal control and dynamic lane control. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods, with better generalizability. In the near future, we plan to explore other related directions, such as control of traffic signal and reversible lanes.

Acknowledgements

This research is supported in part by the National Natural Science Foundation of China under grant 61772138, the National Key Research and Development Program of China under grant 2018YFB0505000,2019YFB1704400 and the Ministry of Education, Singapore, under its AcRF Tier 2 Funding (Grant No: MOE2019-T2-2-116).