MaCAR: Urban Traffic Light Control via Active Multi-agent Communication and Action Rectification

1 Introduction

Urban Traffic Light Control (TLC) is a critical and challenging real-world problem, which aims to maximize the traffic efficiency with limited urban road resource and avoid traffic conflict inside intersections. Finding an appropriate TLC approach can significantly mitigate traffic congestion and bring in significant economic, environmental and societal benefits [Wei et al., 2018].

∗Equal Contribution

Most of the recently proposed reinforcement learning (RL) based methods [Wei et al., 2018; Nishi et al., 2018; Casas, 2017] were focusing on independently controlling intersections in the same road network. In these works, the observable surrounding of an agent is limited to itself, which may cause action conflict between agents and deviate action value from expectation. One example of this action conflict is increasing green-light time in directions suffering heavy traffic can significantly mitigate the traffic. However, it may cause severe congestion in adjacent regions if the adjacent agents cannot adjust themselves in time. Such congestion often occurs because traffic changes caused by agent action changes can be rapid and abrupt.

To overcome this problem, some collaborative optimization based works [Van der Pol and Oliehoek, 2016; Nishi et al., 2018; Chu et al., 2019; Wei et al., 2019b] were proposed recently. These works mitigate the action conflict problem mainly by expanding the observable surroundings of agents. It can be seen as establishing an inactive communication mechanism between agents, in which agents can obtain not only the traffic state of themselves but also adjacent intersections. This mechanism is inactive because agents do not directly share decision information such as actions and agent states, but share noised responses of the road network.

There are two shortcomings of this inactive communication mechanism. First, agents cannot know each other's new actions before perceiving the traffic changes in this inactive communication mechanism. Hence, the perceived action values of agents are biased, since agents perceive traffic pattern changes always lag behind action change. Secondly, the information propagated is insufficient for collaborative optimization, because the shared information only includes the surrounding traffic states, but not includes important agent decision information such as historical actions.

In this work, we propose a novel Multi-agent Communication and Action Rectification (MaCAR) framework, which consists of two parts. The first part is a Communication Agent Network (CAN) with a Message Propagation Graph Neural Network (MPGNN) based active communication network. The purpose of CAN is to generate actions considering other agents' actions and states. By introducing the action information, agents can effectively learn the traffic pattern under the corresponding action, helping reduce the action space and improve model performance. The second part of MaCAR is a novel Traffic Forecasting Network (TFN). TFN forecast the traffic of the whole road network under agents' new actions, together with the corresponding action values. Using this predicted information, we further amend the action value of agents during training to mitigate action value deviation.

We carried out experiments on both synthetic and realworld datasets to evaluate the performance of MaCAR. Experimental results demonstrate that by introducing the active communication mechanism and taking advantage of traffic forecasting information, our proposal can achieve superior performance against the state-of-the-art methods.

We summarize the contributions of this work as follows:

1. We focus on the action conflict problem in previous independent RL methods, which can leads to severe congestion at adjacent intersections when multiple agents act synchronously.
2. In this work, we propose a novel Multi-agent Communication and Action Rectification (MaCAR) framework to control multiple agents collaboratively. MaCAR consists of two components: (1) a novel Message Propagation based Communication Agent Network (CAN), which generates coordinated actions via an active crossagent communication mechanism; (2) a novel Traffic Forecasting Network (TFN) which helps to rectify action against action value bias.
3. In experiments, we demonstrate that our proposal can achieve state-of-the-art performance on both synthetic and real-world datasets by involving the active communication mechanism and take advantage of traffic forecasting information.

2 RelatedWorks

2.1 Reinforcement Learning based Traffic Light Control

Independent Control v.s. Collaborative Control

Most recently proposed methods [Aslani et al., 2017; Wei et al., 2018; Zheng et al., 2019; Chen et al., 2020] were focusing on optimizing agent independently, in which an independent RL agent is compelled to control a certain intersection without noticing other agents' existing. However, as part of the whole road network, the impact of other agents cannot be completely blocked out in real-world applications.

To mitigate this problem, some collaborative control based methods were proposed recently [Lee et al., 2019; Van der Pol and Oliehoek, 2016; Wei et al., 2019b]. Wei et al. [2019b] proposed the CoLight, an independent TLC algorithm with inactive communication mechanism. Their inactive communication mechanism extended agent's receptive field to help make coordinated action. Agents do not share decision information directly in their work, but rather share the road network's responses to historical actions. Therefore, agents do not know the latest actions of other agents until it feels the traffic changes, resulting in a possible action conflict. Different from them, MaCAR is a multi-agent collaborative control based method with an active communication mechanism. In MaCAR, we use the traffic prediction information to actively rectify agents' future actions during training by alleviating action value bias caused by other agents' actions.

Adaptive Control v.s. Pre-defined Phase Scheme

Most recently proposed RL methods [Wei et al., 2018; Nishi et al., 2018; Wei et al., 2019a; Wei et al., 2019b; Aslani et al., 2017; Casas, 2017] were based on adaptive control strategy in which agent selects whether switch the permitted phase (lanes) per second based on traffic situation. Adaptive control provides excellent controllability since the permitted phase can be switched in seconds. However, frequent phase switching may cause traffic accidents and significantly affect the driving experience. To avoid bad driving experience and traffic accidents caused by a sudden phase switching, some pre-defined phase scheme based RL approaches [Aslani et al., 2017; Casas, 2017] were proposed. A typical phase scheme consists of a fixed phase execution cycle and the corresponding execution time list. The agent's action is to adjust the phase execution time. Similarly, MaCAR using a pre-defined phase scheme based control strategy to provides practicability in real-world applications.

2.2 Graph Neural Network

Recently proposed GNNs can be divided into two categories, spectral-based (mainly GCN based) methods [Li et al., 2018; Yu et al., 2019], and Message Propagation based methods [Battaglia et al., 2018; Wei et al., 2019c]. A typical GCN based model that trained on a specific graph could not be directly applied to a graph with a different graph [Zhou et al., 2018]. Hence, plain GCNs is inconvenient in a TLC scenario because multiple agents' actions can change the weight of each road tremendously. More recently, several Message Propagation based methods were proposed to solve this problem [Wei et al., 2019c]. Similarly, MaCAR uses Message Propagation based neural networks to model dynamic graphs from traffic.

3 Our Method

We first introduce some notations and terms. We define term period as the time length of executing all phases once. For simplicity, we set the period length of all intersections as P to ensure agents' synchronous actions. The action applied in period t is defined as a(t) , and its corresponding perceived action value is defined as v(t) . The adjacent intersection set of intersection n is defined by N(n), and it is obtained from the road network topology G.

The input traffic state q(t) is a tensor contains the queue lengths of all roads in the road network after t-th period. Our purpose is to minimize the queue length q(t) of the whole road network, which is a common practice in previous works. To achieve that, we use a concatenated tensor contains the input traffic state and the corresponding actions of the past M periods as input, and we denote this concatenated tensor as {{q, a}(t-M) , ..., {q, a}(t-1) }, where M is a training interval hyper-parameter selected by practice.

To provide good agent customization capability in realworld applications and avoid causing bad driving experience and traffic accidents, we use pre-defined phase scheme based control strategy in MaCAR. A typical action of MaCAR is to redefine the intersection's phase execution time list at the beginning of a new period. The pre-defined phases can be hand-crafted by experts according to real-world applications.

3.1 Communication Agent Network

The framework of MaCAR has shown in Figure 1, the first part is the Communication Agent Network (CAN). CAN is a multi-agent action generation network consisting of an MPGNN based central active communication network and several customized agent subnets, as shown in Figure 2. In CAN, the embedding of the previous M periods' traffic states and its corresponded actions are extracted and propagated via the active communication network, and then feed into the customized agent subnets. By introducing the decision information, we impel the model to learn the message propagation pattern while considering the impact of agents' actions.

Active Communication Network

The active communication network of CAN is formed by several linear layers and Message Propagation based Graph Neural Networks (MPGNNs) which is designed by following the strong discriminant graph theory proved by Xu et al. [2019]. MPGNN learns the spatio-temporal message propagation mechanism between agents.

MPGNN has a multi-layered network architecture, which can learn complex propagation pattern with a wide receptive field. Formally, the feature embedding of intersection n in k-th layer hkn is:

where mkn is the aggregated message received from adjacent intersections of intersection n in k-th layer. fp and fa is the propagation and aggregation function respectively, both of them are learned by using a neural network consists of two linear layers. ϵk is a learnable parameter which helps to generate discriminant graph representation. The input intersection features {{qn, an}(t-M) , ..., {qn, an}(t-1) } are defined as h0n . The output of the last layer of MPGNN hout is a graph embedding contains all aggregated features {hout n | n ∈ G}.

Based on MPGNN, we establish the active communication network to learn the message aggregation and propagation among agents under certain actions, as shown in Figure 2. We then feed the output into agent subnets to generate actions.

Customized Agent Subnet

To fit different types of intersection's traffic light settings, agents in CAN are still decentralized agents. The intersection number determines the number of customized agent subnets. Meanwhile, agent subnet is customized according to the corresponding intersection settings, such as its channelization or direction number. By doing so, MaCAR can fit most existing intersection settings and commonly used control strategies.

Each agent (subnet) of CAN learns a continuous action distribution P. We then sample action from the learned distribution using following equations:

where n is the intersection index, fn learns the mapping from the global feature embedding hout to the mean and variance of action distribution Pn.

Action sampled from the continuous distribution P is first uniformed by using a uniform multiplier α > 0, and then rescaled by using a Softmax function to generate the new phase executing time ratio a(t) n . At last, we multiply a(t) n and P to get the new execution time list.

3.2 Traffic Forecasting Network

The second part of MaCAR is the Traffic Forecasting Network (TFN), which aims to predict the future traffic and the action value of the given actions. By using this forecasting information, we amend the action value and further rectify agent action against other agents' actions.

As shown in Figure 3, TFN first extracts feature embeddings from the previous M periods' traffic using an MPGNN (k=2), and then concatenates them with the embeddings of the corresponding actions. The concatenated embeddings are then fed into two branches (Predictor and Critic). The Predictor branch is used to predict the future traffic ˜q(t) of the whole road network. The Critic branch aims to predict the action value ˜v(t) of the given actions a(t) .

As shown in Figure 3, there is a shortcut from the Predictor branch to the Critic branch. We argue that this shortcut can bring in the predicted future traffic under the given actions to help mitigate the impact of other agents' actions on ˜v(t) .

3.3 Online Training Algorithm

We devise a policy gradient based online training algorithm to train the CAN and TFN after each specific periods M. The online training algorithm has shown in Algorithm 1.

As shown in Algorithm 1, to collect the initial information for MaCAR, we first let simulator run for M periods under initial action a(0) in which all phases have the same executing time. After that, new predictions and actions are generated by using the traffic states and corresponding actions of previous M periods. We train the MaCAR network after every M periods, in which one period is equal to P time steps. During training, we first calculate the action values of the past M periods by using Eq. 5. The action values of t-th period are defined by the difference of queue lengths between (t - 1) and (t) period:

We then optimize the TFN network by minimizing the following loss function:

where || · ||ℓ1 represents ℓ1 norm. It's worth noting that both ˜v(t) and ˜q(t) are generated before generating a(t) .

Algorithm 1 Online Training algorithm

Input: Initial action a(0) , parameters θTFN and θCAN, period length P, simulation time length tmax, training interval M, simulator S

Output: Optimized θTFN and θCAN 1: Let t =M 2: run S(a(0) ) Mperiods → {{q, a}(0) , ..., {q, a}(t-1) } 3: TFN({{q, a}(0) , ..., {q, a}(t-1) }; θTFN) →˜v(t) , ˜q(t) 4: CAN({{q, a}(0) , ..., {q, a}(t-1) }, ˜q(t) ; θCAN) → a(t) 5: S(a(t) ) → {q, a}(t) 6: t = t +1 7: while (t ×P < tmax) do 8: if (t %M ̸= 0) then 9: TFN({{q, a}(t-M) , ..., {q, a}t-1 }; θTFN) →˜v(t) , ˜q(t) 10: CAN({{q, a}(t-M) , ..., {q, a}t-1 }, ˜q(t) ; θCAN) → a(t) 11: S(a(t) ) → {q, a}(t) 12: t = t +1 13: else 14: calculate {v(t-M) , ..., v(t) } via Eq. 5 15: optimize θTFN, θCAN by minimizing Eq. 6 and Eq. 7 16: end if 17: end while 18: return θTFN, θCAN

As we discussed above, other agents' newly generated actions will impact the action values and cause action value deviation between perceived and real action values. To overcome this problem, we use the difference between predicted and perceived action value to help anchoring agent action. We optimize the CAN by minimizing the following loss function:

4 Experiment

We conduct experiments in two open-source traffic simulators: CityFlow 1 and SUMO 2 . Both of them are commonly used in previous works [Wei et al., 2018; Wei et al., 2019b; Wei et al., 2019a].

4.1 Datasets

Real-world Datasets

We used three real-world datasets: DHangzhou, DJinan and DNewY ork. All three datasets are obtained from CoLight [Wei et al., 2019b]. The New York, Hangzhou and Jinan datasets have 196, 16 and 12 intersections, respectively. We used the same simulator (CityFlow) and experimental settings as CoLight to conducted fair comparison.

Synthetic Datasets

In SUMO simulator, we build a road network with 21 intersections to simulate the urban trunk roads, as shown in Figure 4. The road network is constructed by following Wei et al. [2019c]. Intersections in the middle of the road network have four directions, and intersections on the boundary have three directions. Each road is 500 meters long, except roads in the corner is 680 meters. Each road has two directions and three lanes per direction. Road speed limit is set as 16.68 m/s, and the period length is 120 seconds.

1 https://cityflow-project.github.io

2 http://sumo.dlr.de/index.html

As shown in Table 1, we built three different traffic scenarios based on the road network above to mimic the prevailing and challenging traffic conditions in the real world. In Config 1, we synthesized the daytime with heavy flat traffic. In Config 2, we synthesized the traffic switching from light flat-time traffic to heavy peak-time traffic. In Config 3, we simulated the tidal traffic in which part of traffic has a strong trend (West→East) during heavy peak-time traffic. All three kinds of configurations are widespread in metropolitan areas around the world.

The arrival rate defines the traffic density in these configurations. The traffic trend of different traffic groups (T1-T8 in Table 1) is determined by each road's probability of becoming a starting point, via point and destination from a uniform distribution. End time represents the departure time of the last vehicle. All data contain bidirectional and dynamic flows with turning traffic. Moreover, we used the vehicle rerouter algorithm provided by SUMO to synthesize vehicle rerouting cases as in the real world.

Traffic Forecasting Dataset

To evaluate the traffic forecasting performance of TFN, we conduct experiments in the persuasive real-world traffic forecasting dataset METR-LA [Li et al., 2018] comparing with several state-of-the-art methods.

4.2 ComparedMethods

We compare MaCAR with several baseline methods, including several state-of-the-art methods.

In synthetic experiments, we compare with four different baseline methods, including Fixed-time Control (FT), Random Adjustment (RA), Actuated Control (SOTL [Cools et al., 2013]), and Q-Learning Traffic Light Optimization within Multiple Intersections Traffic Network (QLTSO) [Chin et al., 2012]. Each baseline represents a type of commonly used method in real-world applications.

We then conduct experiments in three real-world datasets, comparing with several state-of-the-art methods [Wei et al., 2019b; Wei et al., 2018; Chu et al., 2019; Nishi et al., 2018; Arel et al., 2010; Van der Pol and Oliehoek, 2016]. The state-of-the-art method CoLight [Wei et al., 2019b] is a collaborative optimization based method, in which information exchange between agents is carried out by expanding other agents' observable surrounding.

4.3 Experiments on Synthetic Datasets

We first compare our proposal with several baselines under three traffic scenarios, experimental results have shown in Table 2-4. We can notice that MaCAR can effectively improve traffic efficiency under all three different traffic conditions. Meanwhile, our proposal outperforms all baselines significantly in Avg.Waiting in Config 2 and Config 3. It demonstrates that when traffic shifts, our method still can effectively change the phase plans to mitigate traffic congestion.

In Figure 5 (a) and (b), we show that MaCAR can significantly reduce the total queue length and increase the avg. speed. In Figure 5 (b), we can notice the curve of MaCAR is flatter than other methods and has an upward trend even when the traffic starts to increase from 60 periods. Moreover, the upward trend of MaCAR's curve is more significant than MaCAR-noMPTF. These experiments validate our idea of taking advantage of prediction information is feasible while demonstrating the effectiveness of the proposed crossagent communication mechanism.

4.4 Experiments on Real-world Datasets

We then compare our proposal with several state-of-the-art methods in real-world datasets. We followed the experimental setup used by CoLight to compare our method with previous works fairly. Results have shown in Table 5, MaCAR achieves consistent performance improvements over state-ofthe-art methods on all three real-world datasets: the average improvement is 2.78% compare with CoLight. The performance improvements of our method are attributed to the unique design of our proposal.

Our method also outperforms the joint-action modeling method CGRL. To achieve cooperation, CGRL establishes one model to determine the joint actions of two adjacent intersections, and then conducts centralized coordination of the global joint actions. It needs to search a large operating space, and may faces scalability issues. Compare with CGRL, our method has a smaller search space since we still use independent agents.

In experiments on the New York dataset, we show that MaCAR can outperforms state-of-the-arts in a 196 intersections road network. These experiments also demonstrate that MaCAR has good scalability in practice.

4.5 Experiments on Traffic Forecasting Dataset

We then conduct traffic forecasting experiments on METRLA dataset to demonstrate the traffic forecasting performance of TFN comparing with state-of-the-art traffic forecasting methods. Results have shown in Table 6, TFN can outperforms all the state-of-the-art methods. We can notice that the state-of-the-art methods may perform properly for short-term forecasting, but their long-term predictions are not accurate due to the error accumulation.

4.6 Ablation Study

We also compared the performance of variants of our proposed method, as shown in Table 2 - 4. By removing TFN, we notice that the performance of MaCAR-noTFN is reduced significantly, but still can outperforms baseline methods in most metrics. These experiments show that our idea of combining traffic forecasting with traffic light control can effectively improve traffic efficiency.

5 Conclusion

In this work, we address the traffic light control problem by proposing a novel Multi-agent Communication and Action Rectification (MaCAR) framework. We conduct extensive experiments on both synthetic and real-world datasets to demonstrate the superior performance of our proposed method over baseline and state-of-the-art methods. Besides, we show in-depth case studies and observations to understand how the proposed method overcomes two shortcomings of previous collaborative optimization methods. Several future directions are worth exploring, global optimization methods that consider local importance, and causal inference based methods.

Acknowledgements

This work was supported in part by The National Key Research and Development Program of China (Grant Nos: 2018AAA0101400), in part by The National Nature Science Foundation of China (Grant Nos: 61936006, U1909203), in part by the Alibaba-Zhejiang University Joint Institute of Frontier Technologies.