GPLight: Grouped Multi-agent Reinforcement Learning for Large-scale Traffic Signal Control

1 Introduction

In recent years, there has been an unprecedented trend in coordinating and controlling traffic lights. This trend has been shown to be effective in improving the efficiency and robustness of road networks [Jiang et al., 2021]. With the development of AI technology and the availability of largevolume traffic data, learning-based control approaches have shown great potential in solving traffic signal control (TSC) problems. In particular, multi-agent reinforcement learning (MARL) has shown great potential as a promising solution [Luo et al., 2020], as it enables coordinated control and global optimization of large-scale traffic lights without the need for manual intervention.

∗Corresponding author.

In the past few years, there are many representative research achievements in TSC. IntelliLight [Wei et al., 2018] highlights the importance of features and emphasizes that different agents in the same environment should give greater weight to important features when making decisions. To avoid the need for heuristic design of reinforcement learning (RL) parameters, PressLight [Wei et al., 2019a] maps the rewards in the multi-agent system directly to the pressure values defined in traffic. FRAP [Zheng et al., 2019] recognizes the spatial symmetry of the same agent model at different times and improves the generalization ability of the model. CoLight [Wei et al., 2019b] introduces the graph attention mechanism (GAT [Velickovic et al., 2017]) in the multi-agent system, considering the interaction of the surrounding agents for the first time. In addition, in the latest study, EMVLight [Su et al., 2022] provides a good solution for emergency vehicles through TSC. However, none of the above methods have been applied in large-scale scenarios. MPLight [Chen et al., 2020] and OAM [Liang et al., 2022] are applied to large-scale traffic scenarios, but the heterogeneity of each intersection as

well as the influence relationship between intersections are

not considered.

We investigated the number of intersections discussed in the TSC field (approximately 150 studies published from

1994 to 2020), which is shown in Figure 1. The results showed that 96% of the studies were conducted in the scenario with less than 100 intersections. Therefore, large-scale TSC is still an immature field requiring special attention. Coordinating large-scale traffic lights using MARL is a practical requirement, but it is a challenging problem to solve. On the one hand, it is difficult for all agents to use only a single network representation and strategy [Smith, 1937] since largescale road intersections have complex and abundant patterns, which requires an extremely large and deep neural network. On the other hand, if each agent applies a different neural network, there would be an extremely large number of parameters, which has exceedingly low efficiency and high complexity. Furthermore, intersections that are far apart could have strong spatial-temporal correlation and dynamics, which increase the connection complexity between agents.

To this end, we introduce GPLight, which extracts spatialtemporal features from intersections in real-time and clusters them into different groups. Agents in each group share the same neural network. We first consider real-time traffic flow and static fine-grained road topologies to dynamically divide intersections into different groups by mining the similarity between environments. Grouping agents can reduce the scale of the multi-agent system and improve training efficiency. We use GCN [Kipf and Welling, 2016] network to extract features, and then introduce two loss functions to carry out finegrained partitioning of multi-agent systems. MI Loss aims to keep agents changing smoothly, while Gather Loss aims to let agents find their partners in the same group. The grouping results will be transmitted into the improved QMIX [Rashid et al., 2018] network for training. By mining the similarities among intersections in large-scale intersections, intersections with high similarity can be grouped for cooperation, even if they are far away, thereby reducing the complexity of the system. At the same time, differences between different groups are preserved, allowing GPLight to strike a balance between accuracy and complexity. This process breaks through the traditional TSC method, which only considers the mutual influence of adjacent intersections.

We evaluate GPLight using both synthetic and real-world datasets with up to 1,089 intersections. The experimental results demonstrate that our proposed multi-agent grouping approach, which incorporates dynamic features, enables agents to share policies more effectively and dynamically. The traffic system scheduled by GPLight achieves better efficiency in coordinating large-scale traffic lights compared to existing methods.

The contributions of this work are threefold:

• We comprehensively extract both dynamic and static features of each agent to create its embedding. By considering the real-time dynamic traffic flow and the real road topology, the similarity between intersections can be better mined.
• We propose a MARL algorithm for large-scale traffic light intersections, which divides the multi-agent system into different groups to reduce complexity. Different from the traditional method which only considers the interaction between adjacent intersections, it allows in-
• tersections that show similarities in the whole region to cooperate even though they are far apart.
• We conduct extensive experiments on large-scale multiple scenarios with up to 1,089 intersections, including both synthetic and real-world datasets. Experimental results show that GPLight can achieve better performance in terms of total travel time than other advanced TSC methods in large-scale scenarios.

2 RelatedWork

Intelligent transportation is an important direction of future urban construction [Luo et al., 2022a; Luo et al., 2022b; Luo et al., 2023]. TSC has been studied in the field of transportation for many years. In recent years, there has been a growing interest in combining TSC with MARL. In this approach, the road network is treated as the observation and the signal phase combination as the action set. The signal phase is defined as a set of permissible traffic movements [Zheng et al., 2019]. For example, at an intersection shown in Figure 2, there are eight signal phases combinations to choose from.

In the past, there are many well-known studies of TSC combined with MARL. Intellilight [Wei et al., 2018] uses deep Q-Network (DQN), which states the queue length of each lane, the total number of vehicles at the intersection, the updated wait time, the image of each vehicle's position at the intersection, the current agent's action and the next action. Presslight [Wei et al., 2019a] proposes the use of max pressure as an input feature and reward to maximize throughput in the traffic network. A study in 2020 [Jamil et al., 2020] proposed to integrate rewards obtained by different methods into the training process. Each reward has an independent network to learn Q value, and then votes to get the final action and interact with the environment. GeneraLight [Zhang et al., 2020] designed a traffic flow generator based on Wasserstein generation adversarial network, which improves the adaptability of MARL model to dynamic traffic flow and enhances the generalization ability of MARL model. FedLight [Ye et al., 2021] considers collaborative optimization between intersections and proposes a combination of federated learning and RL. Jiang et al. [Jiang et al., 2021] used multi-time scale model training to learn appropriate strategies for optimal control of traffic signals and dynamic lanes. MACAR [Yu et al., 2021] realizes active communication between agents by considering the effect of the synchronization of agents. It consists of an active communication agent network (CAN) involving a message propagation graph neural network (MPGNN) and a traffic prediction network (TFN). By using predictive information, action value bias during the training process is mitigated to help correct the agent's future actions.

To sum up, as a multi-agent problem, each intersection can be considered as an agent in TSC. Considering the large number of intersections in a city, solving this problem becomes exceptionally challenging. While creating a model for each agent would ensure accuracy, it would require significant resources and difficult training. Therefore, we propose grouping the multi-agent system to identify similar intersections in the entire area for collaboration, thereby reducing the complexity of the training process.

3 Preliminary

GPLight treats TSC as a multi-agent systems, by grouping agents into clusters for achieving better accuracy-complexity tradeoff in coordinating large-scale traffic lights. This section introduces the MARL in TSC.

GPLight considers TSC as a multi-agent task that can be modeled by a distributed multi-agent partially observable Markov decision process (Dec-POMDP) [Oliehoek and Amato, 2016] L = 〈 N, S, A, P, O, O, R, n, γ 〉, where N = { 1,2,· · · , n } is the set of n agents. S is a finite set of states. Each agent i ∈ N can observe only partial environment oit at each time step t, where oit is part of the state st ∈ S. In our scenario, oit includes both the real-time traffic flow as well as road network topology. A is the set of joint actions, where the action ait of each agent i at time step t involves the signal phase combinations that can be selected at the current intersection. Take Figure 2 as an example, for a four-way intersection, there is a total of eight actions that can be selected [Chen et al., 2020]. For an agent i, it will choose an appropriate action ait ∈ A based on the observation oit at time t. The agent will keep this action until the next decision is made. P is the transition probability function. γ is the discount factor whose value space is [0,1). Each intersection is controlled by an RL agent. We consider the system partially observable, which means agent i can only derive an observation oi ∈ O from the observation O(s,i). Given the traffic situation and current traffic signal phase, the goal of the agent is to take an optimal action a ∈ A to maximize the cumulative reward R at each time step t. Our goal is to expect the overall traffic situation to become more unimpeded. Therefore, we consider combining each agent's reward with the length of the queue at the intersection. The reward R of each agent at time t is obtained by the reward function S × A1 × ... × An → R. Here, for a certain intersection i, we assume that zi,l t is the queue length of vehicles in the approaching lane l at time t. We define its reward as Rit=-∑l zi,l t . Each agent has history phases τi. The joint strategy π generates a joint action-value function:

4 The ProposedModel: GPLight

In this section, we propose a grouped multi-agent reinforcement learning method GPLight. As shown in Figure 3, our model mainly includes three parts: Feature Extraction, Group Cohesion and Q-Learning.

GPLight first mines the similarity of agents considering both real-time traffic flow and static road topology, and then maintains a learnable and dynamic clustering to group agents. Since the road topology information is non-euclidean data, we apply the GCN network to extract features for each intersection. As previous research [Kipf and Welling, 2016] suggests that GCN embedding (even with random weights) can automatically cluster when extracting features, the feature extraction process enables a coarse-grained clustering. However, such a clustering is not meticulously designed to guarantee the best performance for subsequent MARL tasks. To address this issue, we propose two loss functions, namely Mutual Information (MI) Loss and Gather Loss, to supervise the GCN network for fine-grained clustering. While MI Loss is used to ensure steady changes of the agents, Gather Loss is applied to maximize the separability between clusters. Furthermore, the clustering is also supervised by the task (MARL Loss), i.e., traffic lights coordination performance. These three losses act as supervisory signals to guide the GCN network to extract ample features, which ultimately lead to the best grouping result for the MARL task. Finally, agents in the same group will share the same network parameters to make decisions.

4.1 Feature Extraction

We model the traffic network in the multi-agent scenario as a graph G = (V ,E), where V is the set intersections and E means the road connections in between. Each intersection is treated as an agent. Each agent i ∈ V has a partial observation oit at time step t. oit includes 1) static features, such as the number of lanes, length, speed limits, type of roads, as well as the local road topology; and 2) dynamic features, such as the real-time traffic flow as well as the current signal phase. We concatenate the state and dynamic features as vector xi ∈ RM , which represents the partial observation oit , where M is the feature dimension. All nodes' features can be represented by a matrix Xn×M, where n represents the number of nodes. The input of GCN at each layer is the adjacency matrix Z and node feature H, where H0 = X. The final layer feature propagation formula improved by GCN is as follows:

where ˜ C is a matrix introduced to normalize Z.

For the feature extraction model, we construct a GCN network with several layers, and the activation function adopts ReLU and Softmax respectively, so the overall forward propagation formula is as follows:

Through the above, static and dynamic features are extracted and processed by the GCN embedding layer. The result obtained by the embedding will be input to the next part for grouping cohesion.

1. Feature Extraction Through GCN Embedding 2. Group Cohesion

4.2 Group Cohesion

In this section, we introduce two loss functions for better group cohesion as follow.

MI Loss: Stability Maintenance

Given the features extracted by GCN, each agent i has an embedding ρi. In order to adapt to the dynamic environment and avoid rapid change that leads to learning instability, we propose to ensure slow changes of agents by maximizing I(τi;ρi | oi), the conditional mutual information between the individual actions and the group given the current observation. However, estimating and maximizing mutual information can often be challenging. To address this, we refer to the work of ROMA[Lhaksmana et al., 2018], which introduced a variational posterior estimate to derive a tractable lower bound for mutual information targets [Wainwright and Jordan, 2008; Alemi et al., 2016]:

where τ t-1 i = (o0i , a0i , · · · , ot-1 i , at-1 i ). qξ is the variational estimator parameterised with ξ , and we call it an encoder, which uses a GRU [Cho et al., 2014] to encode the history of the agent's observation and behavior. The first loss function we use is rewritten from the lower bound of Eq. 4 as follows:

where parameters θρ are conditioned on ρi, B is a replay buffer and BKL[·‖·] is the KL divergence operator.

Gather Loss: Separate Different Groups

Furthermore, we have to separate different groups in order to 1) make agents in the same multi-agent group have more similar features to ensure the accuracy of the shared decision, and 2) maximize differentiation of different multi-agent groups to ensure that the grouping is reasonable. To achieve distinguishable groups, we have the following formula to minimize the similarity between agent i and agent j [Lhaksmana et al., 2018]:

where matrix Uφ = (uij), uij = uφ(τi, τj) is used to measure the difference in the distribution of agent i and agent j by historical local states comparison. The meaning of subscript (i.e., 2,0) is the Frobenius norm. I(ρi; τj) represents the mutual information between agent i and agent j. The values of u and I are both in [0, 1]. We want to minimize the non-zero elements in matrix U while maximizing the sum of I and u. The purpose of this is that we expect to maximize I preferentially, that is, to enhance compactness within multi-agent groups. In this way, the value of u will be high only when the mutual information I of the two agents is low, which means their difference is large. Thus, the multi-agent group becomes compact and the distinction between groups becomes more obvious.

Similarly, we construct an upper bound as the second loss function we will use:

where F represents Frobenius norm, τ t-1 joint distribution and ot is the joint observation.

4.3 Q-Learning

QMIX [Rashid et al., 2018] is a multi-agent reinforcement learning algorithm, which is suitable for Dec-POMDP [Oliehoek and Amato, 2016]. Through group cohesion in the previous step, it is possible for us to combine centralized training with the multi-agent group. As a result, the same group of multi-agents will exhibit a high degree of similarity in decision-making. Conversely, different types of multiagents will be distinguished by variations in their network parameters. Additionally, the group membership of individual agents will continuously change over time.

To better utilize the system's state information, St is incorporated into Qtot (the sum of q values of each agent through linear transformation) using the hyper network, rather than solely as an input to the mixing network. During the updating process, we employ the traditional Deep Q-Network approach and sample data from the replay buffer. This allows us to obtain the final loss function for the Q-learning process as follows [Rashid et al., 2018]:

where θ- are periodically updated parameters for the target network.

From what has been discussed above, the final learning goal of our GPLight framework consists of three Loss functions:

where λI and λU are scaling factors.

5 Experiments

5.1 Settings

We run our experiments on CityFlow [Zhang et al., 2019], a traffic simulator. Compared to SUMO [Lopez et al., 2018], CityFlow is a highly concurrent multi-threaded system with significantly faster simulation. In our experiments, each car has its own set of parameters, e.g., acceleration, maximum speed, which greatly improves the realism of the traffic simulation environment.

As each car makes its way from the start position to the destination. GPLight schedules the traffic light of all intersections, which would influence the moving speed of all vehicles since Vehicles follow the traffic rules. According to the traditional setting, each green signal is followed by three seconds of yellow light and two-second all red time.

5.2 Dataset

Synthetic Data. In the synthetic dataset, we will use two

kinds of maps. They are made up of different numbers of intersections. Synthetic maps are generated via Cityflow and include road attributes such as the number of lanes and road speed limits. Each road at the intersection has three lanes with 3 meters in width, lanes between two intersections are different in length.

• Grid10×10-Uni. In this type of map, traffic flows in one direction. We uniformly set it to run west to east and north to south. The west→east traffic flow is 300 vehicles/lane/hour, and the north→south traffic flow is 90 vehicles/lane/hour.
• Grid33×33-Bi. In this type of map, there are 1,089 intersections, and traffic flows in both directions. Vehicles moving in from east, west, north, south. The

east↔west traffic flow is 300 vehicles/lane/hour, and the north↔south traffic flow is 90 vehicles/lane/hour. The structure of the road network is heterogeneous, thus the setting is more realistic.

Real-worldData. We also experiment with real traffic data. For the convenience of subsequent comparative experiments, we continue to use the real maps of Hangzhou, Jinan in China, and New York in the USA. Their road network structure can be imported from OpenStreetMap, as shown in Figure 4. A detailed comparison of the three real-world datasets is shown in Table 1.

(d) Manhattan, NewYork, USA

Figure 4: (a) is Synthetic Map with 1,089 intersections. (b)-(d) are Real-world Maps with 16, 12, and 196 intersections. The green areas on the maps are the ones we use. The intersections within the yellow circles will be used in the experiment.

5.3 Baseline

Our experiments mainly compare two types of methods, traditional traffic signal control methods as well as deep reinforcement learning-based signal control methods. Here are the details:

• Fixedtime [Koonce and Rodegerdts, 2008]. In the method of Fixedtime, intersection traffic signals are in

accordance with the pre-set timing scheme. Traffic signal light changes periodically.

• MaxPressure [Varaiya, 2013]. In MaxPressure, the purpose of traffic signal control is to minimize the pressure at the intersection and balance the length of vehicle queue on the lanes connected with the intersection.
• IntelliLight [Wei et al., 2018]. It essentially uses a deep Q-learning network (DQN). The agent at each intersection is completely independent, regardless of adjacency and parameter sharing. The reward value is set as the weighted result of the six evaluation indexes.
• CoLight [Wei et al., 2019b]. Graph neural network is introduced in CoLight. It takes into account the influence of surrounding intersections on the current intersection by introducing graph attention network and some multihead calculations.
• MPLight [Chen et al., 2020]. MPLight combined with MARL to conduct experiments at large-scale intersections. It sets up a DQN at each intersection.

5.4 EvaluationMetric

The main purpose of controlling the traffic signal lights at the intersection is to make the vehicles pass through the intersection more efficiently. In order to achieve this goal, we usually set some indicators to evaluate the efficiency [Wei et al., 2019c]. In our experiment, we choose Travel Time to evaluate the performance of the signal control algorithm. It is defined as the average time taken by all vehicles during their journey.

5.5 Effect Verification of Group Cohesion

In this section, we visualize the results of Group Cohesion to demonstrate the feasibility of the method.

We process the data of Group Cohesion in GPLight, hoping to see its effect intuitively. It is worth noting that since we want to visualize the results, we compress the embedding of GCN into two dimensions.

We run the experiment on the synthetic map. Figure 5 shows the clustering results of Group Cohesion. It can be seen that GPLight can divide groups effectively and the results change in real-time with different traffic conditions at different times. Figure 6 shows the results of Group Cohesion combined with the road network. Different from traditional TSC in which only the influence of adjacent intersections is considered, we can see from the figure that in GPLight, it is possible to cooperate even when intersections are far apart. This is because the topological structure and traffic features are likely to be highly similar even if intersections are far apart. GPLight processes the comprehensive features of the intersections to mine these similarities.

5.6 Performance Comparison

This section presents the performance of GPLight and compares it with conventional transportation methods and RL methods.

Overall Analysis

Table 2 shows GPLight's comparison to the other five approaches, including two traditional TSC approaches and three advanced TSC approaches in MARL. According to the experimental result, GRLight has an average improvement of

21.6% compared with the two traditional methods (Fixedtime and MaxPressure) on the synthetic datasets. On realworld datasets, GPLight has improved 53.3% over Fixedtime and 31.4% improvement over MaxPressure on average. This is because conventional traffic light controls do not apply to traffic conditions that change over time. The traffic signal control methods which are adjusted according to the real-time traffic flow are more suitable for our life.

We also compare GPLight with three advanced MARLbased TSC methods (IntelliLight, CoLight and MPLight). As we can see from the table, the performance of GPLight is significantly better. GRLight achieves an average 13.2% improvement over the other three methods on synthetic datasets. In addition, GPLight averages 30.9% improvement over IntelliLight, 15.1% improvement over CoLight and 21% improvement over MPLight on real-world datasets, which proves its superior performance. We can see that the RL-based approaches are significantly superior to the traditional TSC approaches. This is because the TSC methods based on RL can flexibly make judgements derived from the current states of the intersections, which makes a great contribution to the changing traffic situation at every moment.

Furthermore, we conducted an experiment on a large and irregular road network. As shown in Table 3, GPLight demonstrates its superiority more prominently on Grid33×33Bi maps than on the others, highlighting the effectiveness of our approach in large-scale TSC. The reason for this is that we have mined intersections with high similarity among largescale intersections and grouped them for cooperation. The experimental results demonstrate that this cooperation model significantly improves efficiency.

In conclusion, GPLight groups multiple agents, which not only ensures the diversity between different groups, but also reduces the difficulty of training within the same group. The experimental results prove that GPLight can effectively group multi-agent systems and achieve superior performance, which is particularly evident in large-scale TSC.

Convergence Analysis

In Figure 7, we compare GPLight with IntelliLight, CoLight and MPLight's convergence rate during training. The metric used is the average travel time of vehicles evaluated at each episode. CoLight's convergence trend is similar to GPLight's, but GPLight performs best compared to the other three advanced RL-based TSC approaches. This is reflected in three aspects, respectively, initial performance after the first episode, learning time to achieve a pre-expected goal, and the final learning result. From this, we can conclude that our model GPLight learned the best way to make decisions and achieved good results in overall average travel time while maintaining excellent convergence rates.

5.7 Ablation Experiments

In the above experiments, we can see that GPLight has shown excellent results. As GPLight is mainly composed of three modules: GCN Embedding, Group Cohesion and QMIX. To prove the effectiveness of each module, we conduct ablation experiments. It can be seen from Table 4 that multi-agent reinforcement learning effects without GCN embedding or Group cohesion will become worse, which proves that both GCN Embedding and Group Cohesion have played indispensable roles in GPLight.

6 Conclusion

This paper presents GPLight, a MARL TSC method that effectively balances accuracy and complexity in large-scale TSC by grouping agents with a high degree of similarity. Experimental results demonstrate its superiority. In future research, we will focus on the following aspects: 1) addressing heterogeneous intersections and 2) developing MARL algorithms specifically designed for grouped agents. By dividing agents into groups, we enable collaboration within each group to enhance training efficiency and reduce complexity, while agents from different groups cooperate to optimize traffic efficiency. Taking these interactions into account, we aim to propose novel MARL methods that improve training efficiency and stability.

Acknowledgments

This paper is supported in part by the National Key Research and Development Program of China under Grant 2022YFB4300403, the Natural Science Foundation of China under Grant 62102041, Grant 62272053, and in part by the Young Elite Scientists Sponsorship Program by China Association for Science and Technology (CAST) under Grant 2022QNRC001.