InitLight: Initial Model Generation for Traffic Signal Control Using Adversarial Inverse Reinforcement Learning

1 Introduction

Along with the rapid growth of urbanization, we are witnessing an increasing number of traffic jams on road networks worldwide, inevitably resulting in both serious environmental pollution (e.g., carbon emissions) and tremendous loss in economic and time costs. To reduce such traffic congestion, various Traffic Signal Control (TSC) methods [Rizzo et al., 2019; Chen et al., 2020b; Zheng et al., 2021] have been extensively studied. So far, the most widely used TSC approaches in practice can be classified into two categories: i) rule-based methods (e.g., FixedTime [Koonce and Rodegerdts, 2008], GreenWave [T¨or¨ok and Kert´esz, 1996], SCOOT [Hunt et al., 1982], and SCATS [PR, 1992]) that make control decisions based on the pre-defined rules of traffic plans; and ii) adaptive methods (e.g., MaxPressure [Varaiya, 2013], MaxQueue [Zhang et al., 2021], and SOTL [Cools et al., 2013]) that control the traffic in a heuristic manner.

Thanks to the prosperity of Artificial Intelligence (AI), Reinforcement Learning (RL) has become more and more popular in designing intelligent policies to further improve TSC performance. Unlike traditional TSC methods, RL-based TSC equips each intersection with an agent, which gradually trains its control policy based on repetitive "trial-and-error" style interactions with the corresponding intersection traffic. Although such methods can achieve much better TSC performance than their traditional counterparts, due to the increasing simulation complexity of modern traffics, the training time for agents of RL-based TSC is skyrocketing. Worse still, existing RL-based TSC methods are typically traffic environment-specific, i.e., a set of correlated RL-based agents are trained together within a specific traffic environment. As a result, it is hard for such agents to either learn from or benefit from the training in other complex traffic environments. Therefore, how to effectively improve both the learning efficiency and generalization ability of RL agents is becoming a

major challenge in the design of RL-based TSC.

Inspired by the concept of Adversarial Inverse Reinforcement Learning (AIRL), we propose a novel agent pre-training method named InitLight, which can quickly generate an initial model with superior generalization ability to other complex traffic environments. As shown in Figure 1, InitLight is built on top of an adversarial framework consisting of two main components, i.e., generator and discriminator. Unlike traditional RL-based TSC methods that train a large set of correlated agents simultaneously for a specific multiintersection environment, the pre-training made by InitLight is based on a variety of single-intersection environments together with their expert trajectories. To form a pre-trained initial model, InitLight conducts the training on a Proximal Policy Optimization (PPO) [Schulman et al., 2017] agent guided by the discriminator, which iteratively distinguishes the expert trajectories (obtained from a given TSC method) from agent trajectories (derived from real-time agent-environment interactions). Eventually, the discriminator learns a reward function that can recover the ground-truth reward function at optimality for TSC of any traffic environment. Therefore, the pre-trained agent by InitLight can be deployed at intersections of any complex traffic environments as the initial models to accelerate their overall global RL training. This paper makes the following three major contributions:

• ·We propose a novel AIRL-based framework that can quickly derive a general but effective initial model for various multi-intersection environments to accelerate their overall RL-training performance.
• ·We introduce an adversarial learning mechanism that supports simultaneous learning of the RL model and reward function, where the discriminator can recover the ground-truth reward at optimality.
• ·We implement InitLight on top of the traffic simulator Cityflow [Zhang et al., 2019], and conduct comprehensive experiments on both complex real-world and synthetic multi-intersection datasets to show the effectiveness of our approach.

2 RelatedWork

Various RL-based methods [Chen et al., 2022] have been proposed to improve the performance of TSC. For example, inspired by the Max Pressure (MP) control theory [Varaiya, 2013], PressLight [Wei et al., 2019a], MPLight [Chen et al., 2020a], and MetaLight [Zang et al., 2020] conducted RLbased TSC optimization by encoding both system states and reward functions based on pressure. Similarly, Zhao et al. [2022] created a new concept named intensity, which takes extra vehicle dynamics (i.e., speed and position) besides pressure into account. Based on the notion of intensity, Ye et al. [2022] presented the fair index, which takes the travel quality of individual vehicles into account. To further improve TSC performance, Ye et al. [2021] developed FedLight, which adopts federated reinforcement learning to reduce the overall RL training costs. Jiang et al. [2022] proposed a universal communication form to enable high-quality TSC based on cooperation among intersections. Moreover, RL-based methods have been increasingly investigated in specific TSC problems, e.g., multi-agent-based TSC [de Almeida et al., 2022; Koohy et al., 2022], TSC with dynamic lanes [Jiang et al., 2021], and resource-constrained TSC [Xing et al., 2022]. Although the above methods are promising, most of them adopt randomly initialized models, where TSC agents need a long training time to achieve near-optimal control performance.

Under the guidance of expert trajectories, Inverse Reinforcement Learning (IRL) [Russell, 1998; Ng et al., 2000] has been acknowledged as a promising way to optimize RL

policy learning. However, existing IRL methods (e.g., maximum margin approaches [Abbeel and Ng, 2004; Ratliff et al., 2006] and probabilistic approaches [Ziebart et al., 2008; Ramachandran and Amir, 2007; Boularias et al., 2011; Choi and Kim, 2013]) are not good at dealing with complex RL tasks. Although various adversarial IRL methods [Ho and Ermon, 2016; Finn et al., 2016a; Finn et al., 2016b] have been proposed to tackle such a problem, most of them: i) do not support reward function recovery; or ii) have to learn from complete trajectories. Therefore, they are unsuitable for RL scenarios with complex settings or uncertain environments. To enable smooth transfer between environments with notable variations, Fu et al. [2018] proposed an effective adversarial IRL method named AIRL based on a novel adversarial reward learning formulation, whose discriminator can accurately recover reward functions with superior adaptability to various dynamic environments. Inspired by AIRL, we propose a novel pre-training method that can derive initial models for TSC agents to accelerate the overall RL performance of complex traffic environments. Unlike existing RLbased TSC methods [Xiong et al., 2019; Zhang et al., 2020; Zhu et al., 2021] that consider generalization performance, our work can quickly pre-train a general but effective initial model from single-intersection environments, which can be deployed to various kinds of multiple-intersection environments to accelerate their RL training processes.

To the best of our knowledge, InitLight is the first adversarial IRL framework for TSC to enable the pre-training of an initial model for the RL agent, which can be deployed to arbitrary complex traffic environments to accelerate their RL training processes.

3 Our InitLight Approach

To enable training an effective initial RL model with outstanding adaptability to various traffic environments, we design InitLight based on an adversarial architecture, where the discriminator learns a reward function from expert trajectories to guide the RL training. Figure 2 details the major components and workflow of InitLight. In our approach, the adversarial framework consists of a generator, a discriminator, and three buffers. Through interactions between the PPO agent and the corresponding traffic environment, the generator generates agent trajectory data stored in the agent trajectory buffer to confuse the discriminator. By distinguishing between expert and agent trajectories, the discriminator can learn a reward function to guide the learning process of the PPO agent. Based on the adversarial learning framework above, InitLight can improve both the learning efficiency and generalization ability of RL models. The following subsections will give the details of our approach.

3.1 Generator Design

As shown on the left part of Figure 2, the generator consists of a PPO agent and a single-intersection environment. The PPO agent interacts with the environment to generate agent trajectory data in the same way as classic reinforcement learning. We use solid-line arrows in the generator part to show this process. First, the agent obtains the current traffic state s Generator Agent Trajectory Buffer from the intersection environment. Second, a best action a is chosen by the agent based on s. Finally, after applying a in traffic lights, the traffic environment will enter the next state s′ . The trajectory data ⟨s, a, r, s′ ⟩ will be saved in the agent replay buffer for future learning by using mini-batch samples B⟨s,a,r,s′ ⟩, where reward r is obtained from the discriminator based on ⟨s, a, s′ ⟩.

Figure 2: Architecture and workflow of InitLight.

Intersection Modeling. The right part of the generator in Figure 2 shows an intersection example. The intersection consists of a set of arrival lanes La = {l1 i , l2 i , · · · , l12 i } and a set of departure lanes Ld = {l1 o , l2 o , · · · , l12 o }. Each road is represented by a 2-tuple (li, lo), where li ∈ La and lo ∈ Ld are incoming and outgoing lanes, respectively. Based on direction marks on the end of each arrival lane, we define a directed road as (li, lo) if lo is the departure lane indicated by the direction mark on the ground of li. For example, (l10 i , l10 o ) and (l11 i , l11 o ) are two directed roads. According to common sense, the vehicles turning right are not restricted by traffic signals. Therefore, there is a set of eight feasible control phases P = {p1, · · · , p8} indicating the rights-of-way signaled to vehicles by traffic lights. The intersection example shows a scenario with control phase p5 enabled, where the vehicles on lane l10 i can turn left to enter lane l10 o and the vehicles on lane l11 i can go straight to enter lane l11 o . Note that, when the number of lanes increases, the number of control phases will not change.

Pressure. Inspired by the MP control theory [Varaiya, 2013], we use the concept of pressure for the design of RL agents.

Definition 1. (Pressure of a Directed Road). For a directed road (li, lo), the pressure of (li, lo) is defined as

where N(li) and N(lo) are the numbers of vehicles on the arrival lane li and departure lane lo, respectively.

For example, in Figure 2, the pressure of directed road (l2 i , l2 0 ) is P(l2 i ,l2 0 ) = 2 - 1 = 1.

Definition 2. (Pressure of an Intersection). The pressure of an intersection I is calculated as:

In the example of Figure 2, we have PI = |0+2+1+2+ 2 + 1 + 2 + 1 + 1 + 2 + 2 + 2 - 0 - 1 - 1 - 1 - 1 - 2 - 1 - 1 - 1 - 2 - 1 - 1| = 5.

Intuitively, the pressures of roads and intersections indicate the degrees of disequilibrium between the numbers of arrival and departure vehicles. In general, we can maximize the throughput of an intersection by minimizing the pressure, PI, of the intersection. This is because larger PI implies a more unbalanced distribution of vehicles.

Agent Design. Based on the pressure definition, we design the key elements of PPO agent using the following settings:

• State: The PPO agent captures part of the intersection information as the state. For a standard intersection as shown in Figure 2, the state includes the pressure of all the directed roads (i.e., P(l1 i ,l1o ), P(l2 i ,l2o ), · · · , P(l12 i ,l12 o )) and the current control phase pcur (i.e., p5).
• Action: Based on the observed state, the PPO agent needs to choose one best control phase to maximize the throughput of the intersection. In this paper, the PPO agent has 8 permissible phases (i.e., p1, · · · , p8) by default as shown in Figure 2.
• Reward: When an action finishes, the environment will return a reward as r = -PI to reflect the effect of the action, where PI is the intersection pressure. Note that, in single-intersection environments, we use a discriminator to learn a reward function from expert trajectories and generate the reward of each agent trajectory sample, which is helpful to pre-train a robust initial agent.

In our approach, the PPO agent has two neural networks, namely, Actor and Critic models, with their own model parameters, Aθ and Cθ, respectively. The Actor model is used to learn which action to take under the current state, whereas the Critic model learns to evaluate whether or not the action (chosen by the Actor model) can result in a better state of the

Expert Trajectory Buffer traffic environment. Meanwhile, the feedback from the Critic model is used to optimize the Actor model. On the left part of Figure 2, the dashed-line arrows show the learning process of the PPO agent. To calculate the loss value for optimization, a mini-batch of sequential trajectory samples is collected from the agent replay buffer. Note that, the trajectory samples of the mini-batch must be continuous.

Critic Model. The loss function of the Critic model is:

where E is an operator to calculate empirical average over a mini-batch of samples. By using the Temporal-Difference (TD) algorithm [Tesauro and others, 1995] to estimate the target value, Cθ(st)target is calculated by Cθ(st)target = rt+1+γ ·C(st+1). The parameters of the Critic model can be updated by the stochastic gradient descent algorithm Adam [Kingma and Ba, 2014]:

where ∇LC is the gradient of Critic's loss, θC is model parameters of C, θ′C denotes the updated parameters, and ηC is the learning rate for the optimization of the Critic model.

Actor Model. Since PPO is a policy gradient-based RL algorithm, the original loss function of the Actor model LA is:

where At is an estimated value of the advantage function at time step t. By using the Generalized Advantage Estimator (GAE) [Schulman et al., 2015], At is calculated as follows:

where γ ∈ [0, 1] is the discount factor of future rewards, λ ∈ [0, 1] is the GAE parameter, |B| is the batch size of the sampled mini-batch B, and δt = rt + γCθ(st+1) - Cθ(st).

In our PPO agent, instead of using the original loss function in Equation 5, we use the importance sampling to obtain the expectation of samples under the new Actor model Aθ we need to update. Note that, the samples are gathered from an old Actor model Aold θ . The loss function of the Actor model by using the importance sampling is:

By using Kullback-Leibler divergence [Kullback, 1997] with a small value ϵ, we can optimize LA subject to the constraint on the amount of the Actor model optimization:

To simplify the implementation and improve the sampling efficiency, the loss function LA can be given by a clipped surrogate objective function:

where Rt = Aθ(at|st) Aold θ (at|st) and σ is the clipping parameter that restricts the upper/lower bounds in the clip(·) function to stabilize the updating process. By using the clipped objective function, the PPO agent can quickly avoid actions with negative advantage values and does not greedily choose actions with positive advantage values.

3.2 Discriminator Design

Inspired by AIRL, we design a discriminator to learn a reward function from expert trajectories, which is robust to changes in dynamics and can guide the pre-training process of the agent. On the right part of Figure 2, the discriminator has two neural networks R and V . The solid-line arrows show the reward generation based on the agent trajectory ⟨s, a, s′ ⟩ and dashed-line arrows show the learned reward function optimization from the expert trajectory ⟨se, ae, s′e ⟩.

In a trajectory-centric formulation proposed by [Finn et al., 2016a], the discriminator Dθ is defined as follows:

where τ is a sequence of states and actions induced by a policy and dynamics, fθ is a learned function with its own parameter θ, π is the policy of agent trained to maximize the learned reward R(τ) = log(1 -Dθ(τ)) - logDθ(τ). Note that, updating the discriminator can be viewed as updating the reward function, and updating the policy can be viewed as improving the sampling distribution used to estimate the partition function. If trained to optimality, it has been proved that an optimal reward function can be extracted from the optimal discriminator as f ∗ (τ ) = R∗ (τ )+c and the policy π can recover the optimal policy, where c is a constant value. However, using the entire trajectory τ can result in high-variance estimates compared to using single state-action pairs.

In order to solve this issue, we convert Equation 10 into a single state and action case:

Fu et al. [Fu et al., 2018] proved that, at optimality, f ∗ θ (s, a) is equal to the advantage function of the optimal policy A∗ (s, a):

Although this change results in an efficient algorithm for imitation learning, it is less desirable for the purpose of reward learning. This is because the advantage function is a heavily entangled reward and it supervises each action based on the action of the optimal policy, which is not robust to changes in environmental dynamics.

To decouple the reward function from the advantage, we modify the discriminator with the form:

where the policy π is equivalent to the Actor model Aθ, fRθ,Vθ is restricted to a reward approximator Rθ and a shaping term Sθ with their own parameters θ:

The additional shaping term Vθ helps mitigate the effects of unwanted shaping on our reward approximator Rθ. The advantage of this approach is that Rθ(s) can be parameterized as a sole function of the state, which allows us to extract disentangled rewards from the dynamics of the environment. At optimality, it can recover a state-only ground-truth reward in this restricted case:

where R∗ (·) is the true reward function and V ∗ (·) is the optimal value function. This is because, by Equation 12, f ∗Rθ,Vθ must recover to the advantage function:

Algorithm 1 The Training Procedure for InitLight

Input: i) the number, E, of episodes, ii) the number, S, of episode steps; iii) expert trajectory buffer Be; iv) agent trajectory buffer Ba; v) agent replay buffer Br; vi) Actor model Aθ; vii) Critic model Cθ; viii) R model Rθ; ix) V model Vθ;

x) discount factor γ, and; xi) batch size N. Output: i) Cθ; ii) Aθ.

• 1: initialize Ba and Br to empty;

2: for episode = 1, 2, · · · ,E do 3: for step = 1, 2, · · · , S do 4: obtain the current state s; 5: choose the action a based on s; 6: execute action a in environment; 7: observe the next state s′ ; 8: store agent trajectory ⟨s, a, s′ ⟩ in Ba; 9: compute f ← Rθ(s, a) + γVθ(s′ ) - Vθ(s); 10: compute d ← exp(f )/(exp(f ) + Aθ (a|s)); 11: compute reward r ← log(d) - log(1 - d); 12: store trajectory ⟨s, a, s′ , r⟩ in Br; 13: if Br.size() ≥ N then 14: sample a mini-batch be of size N from Be; 15: sample a mini-batch ba of size N from Ba; 16: for samplee in be and samplea in ba do 17: compute Le and La by Equation 17; 18: compute LD ← Le + La; 19: update model parameters of Rθ and Vθ; 20: end for 21: sample a mini-batch br of size N from Br; 22: for sampler in br do 23: compute LC using Equation 3; 24: compute LA using Equation 9; 25: update model parameters of Cθ and Aθ; 26: end for 27: end if 28: end for 29: end for 30: return trained models Cθ and Aθ;

where f(s, a, s′ ) can be considered as a single-sample estimate of A∗ (s, a) in stochastic environments and Q(·) is the Q-function. Therefore, Rθ and Vθ can recover to the groundtruth reward and optimal value function, respectively.

Since the objective of the discriminator is to classify expert and agent trajectories, we use a binary cross-entropy loss to train the discriminator. The loss function of the discriminator is LD = Le + La, where Le and La are defined as follows:

where xei and xai are expert and agent trajectory samples, respectively, yei and yai are labels that indicate whether the corresponding sample is from agent or expert trajectories, respectively, and N is the number of samples.

3.3 InitLight Implementation

Algorithm 1 details the training process of InitLight. In lines 4-12, the PPO agent interacts with the environment and stores the trajectory, where the reward is generated by the discriminator. Lines 14-20 show the optimization of the discriminator. After the discriminator update, lines 21-26 update the parameters of the PPO agent. Finally, line 30 returns a trained PPO agent as the initial model to be applied in multi-intersection traffic scenarios.

4 Performance Evaluation

To evaluate the effectiveness of our approach, we conducted experiments on an Ubuntu server equipped with 3.7GHz Intel CPU, 32GB memory, and NVIDIA RTX 3080 GPU. We used the open-source traffic simulator Cityflow [Zhang et al., 2019] to simulate both single- and multi-intersection environments. During the simulation of traffic scenarios, similar to the work in [Wei et al., 2019a; Ye et al., 2021], we set the phase duration of each signal light to 10 seconds. In InitLight, the PPO agent contains two neural networks, i.e., the Actor model with three layers (13, 32, and 8 neurons on each layer, respectively) and the Critic model with three layers (13, 64, and 1 neurons on each layer, respectively). The R model and V model of the discriminator have the same structure, i.e., three layers with 13, 32, and 1 neurons on each layer, respectively. We used the Adam optimizer for parameter updating and set the learning rate η to 0.0003 for all models. The discount factor γ is 0.99, and the GAE parameter λ is 0.95. The batch size N of the data sampled for training is 20, and the clipping parameter ϵ is 0.2. We designed comprehensive experiments to answer the following three research questions.

RQ1 (Effectiveness): Can InitLight quickly learn from expert trajectories of single-intersection environments and form a corresponding high-quality TSC policy?

RQ2 (Generalizability): Can the agent pre-trained in RQ1 be deployed in arbitrary complex multi-intersection environments to benefit their RL training processes?

RQ3 (Benefits): Why can our proposed InitLight substantially improve the learning performance and generalization ability of RL models?

4.1 Experimental Settings

Baselines. Since the agent design of our InitLight approach is based on the concept of the pressure, for fair comparison, we chose six representative pressure-based baseline methods, including two classic Non-RL methods and four RL-based methods as follows: i) FixedTime [Koonce and Rodegerdts, 2008], a non-RL control method that cyclically selects control phases based on a predefined phase sequence; ii) MaxPressure [Varaiya, 2013], a heuristic MP-based control method that greedily selects the phase with the maximum pressure; iii) PressLight [Wei et al., 2019a], a deep RL-based control method controlling traffic lights intelligently based on the MP control theory; iv) A2C [Ye et al., 2021], a TSC method that controls one intersection by an individual Advantage ActorCritic (A2C) agent; v) FedLight [Ye et al., 2021], a state-ofthe-art federated reinforcement learning method that enables the sharing of the knowledge among RL agents; and vi) PPO [Schulman et al., 2017], a deep RL-based method that uses the PPO algorithm to control traffic lights.

Datasets. We tested twenty public datasets (i.e., environments) provided by [Wei et al., 2019b], including eleven real-world single-intersection datasets (i.e., S1-S11), four synthetic multi-intersection datasets (i.e., Syn1-Syn4), and five real-world multi-intersection datasets (i.e., Hangzhou1, Hangzhou2, and Jinan1-Jinan3). For all the investigated datasets, each intersection of all the road networks has four incoming roads and four outgoing roads, where each road has three lanes, i.e., turning left, going straight, and turning right. The details of the datasets are as follows:

• Real-world single-intersection datasets: We considered 11 real-world single-intersection datasets (i.e., S1S11), which are captured by cameras deployed in the city of Hangzhou. Based on the collected taxi statistics in Hangzhou, we set the turning ratios of these datasets to 10%, 60%, and 30% for turning left, going straight, and turning right, respectively.
• Synthetic multi-intersection datasets: Synthetic

datasets, Syn1-Syn4, contain 1 × 3, 2 × 2, 3 × 3, and 4×4 intersections, respectively. The vehicle arrival rates of each dataset follow a Gaussian distribution with an average of 500 vehicles per hour for each entry lane.

• Real-world multi-intersection datasets: We investigated five real-world datasets (i.e., Hangzhou1, Hangzhou2, Jinan1, Jinan2, and Jinan3), which are collected by the cameras deployed in the cities of Hangzhou (Gudang sub-district) and Jinan (Dongfeng sub-district). Note that each multi-intersection dataset collected from Hangzhou has 12 (3×4) intersections, and each multiintersection dataset collected from Jinan has 16 (4×4) intersections.

Since the diversity of different datasets is crucial for our experiments of generalization ability, we used eleven single-intersection datasets, S1-S11, to pre-train the initial PPO agent and nine multi-intersection datasets, Syn1-Syn4, Hangzhou1-Hangzhou2, and Jinan1-Jinan3, to evaluate the transfer performance of the initial model generated by our InitLight approach.

4.2 Performance of Imitation Learning (RQ1)

To answer RQ1, we considered InitLight as an imitation learning method and compared it against the six baseline methods on single-intersection datasets. For each RL method, we trained its model on the eleven single-intersection datasets (i.e., S1-S11) in a round-robin fashion, where the training lasted 20 times. Therefore, the whole training process involved 220 episodes in total. We collected the expert trajectories for each single-intersection dataset, which were generated by the TSC method MaxPressure [Wei et al., 2019a]. The reason why we chose MaxPressure here is that MaxPressure can achieve competitive results compared with state-ofthe-art pressure-based methods.

Table 1 shows the comparison results for different control methods in terms of average travel time over S1-S11. For each dataset, the TSC methods with the best or second-best performance are highlighted in bold. From this table, we can find that InitLight achieves the best performance, since it achieves top-2 best average time in 7 out of 11 datasets. Since InitLight adopts the same RL model as PPO, due to the guidance of expert trajectories, we can find that InitLight significantly outperforms PPO by more than 10% in four datasets (i.e., S1, S2, S4, and S8). Note that FedLight and A2C have the same results. This is because the clients of FedLight are implemented on top of A2C models, where FedLight is equivalent to A2C when dealing with only one single intersection.

Figure 3 compares the convergence performance of all the RL-based methods on datasets S1-S11. Since the training is in a round-robin fashion, we can observe periodic fluctuations along with the convergence curves. Since FedLight and A2C are the same for single-intersections, we omit the result for FedLight here. From this figure, we can find that, compared with the three baselines, our InitLight approach can achieve the quickest convergence and lowest average travel time for the given 11 single-intersection datasets. According to the convergence criterion used by [Zhao et al., 2022; Fang et al., 2022], the three baselines need more training episodes to converge (e.g., PressLight converges at the 88th episode, whereas InitLight needs only 22 episodes to converge) and have more significant fluctuations (without the guidance of expert trajectories).

4.3 Performance of Transfer Learning (RQ2)

To evaluate the generalization ability of InitLight, we considered nine transfer tasks that adopt the pre-trained PPO agent in various multi-intersection datasets, where each intersection is equipped with one pre-trained PPO agent. For a specific multi-intersection dataset, we trained each RL with 200 episodes. Table 2 presents the TSC performance of all the investigated TSC methods on multi-intersection datasets. Similar to Table 1, we highlight the best and second-best results in bold for each dataset. From this table, we can find that RL-based methods can achieve much better performance than non-RL methods. For example, although MaxPressure and PressLight are based on MP control theory, PressLight outperforms MaxPressure on all datasets as it adopts RL to control traffic signals more intelligently. Compared with the four RL-based baselines, our InitLight method can achieve top-2 performance on all nine datasets.

Figure 4 evaluates the convergence performance of the RLbased methods on the nine multi-intersection datasets. Note that here only InitLight adopted the same pre-trained model at intersections for all the multi-intersection datasets, while all the baselines adopt randomly initialized models at intersections. Compared to all the four baseline methods, our InitLight approach achieves almost the lowest average travel time at the beginning of RL training for all the datasets with much fewer fluctuations. Note that, for all the investigated datasets, InitLight can converge within 5 episodes, which is much quicker than all the baselines. Moreover, InitLight can always achieve the top-2 lowest average travel time for all the multi-intersection datasets eventually. The above facts evidently reveal the generalization ability of the pre-trained agent by InitLight.

To better understand the merits of InitLight, Table 3 shows the detailed convergence information of different RL-based TSC methods from the perspective of jumpstart performance (i.e., average travel time after the first episode) and the starting episode of the convergence, where the best and secondbest results are marked in bold. Here, we used the criterion of convergence following [Zhao et al., 2022; Fang et al., 2022]. From this table, we can find that InitLight achieves the best jumpstart performance and the fastest convergence on all datasets. After the first episode, the average travel time of InitLight over all the datasets approximates the final results as shown in Table 2. It means that InitLight can almost converge in the first episode, which again confirms the generalization ability of pre-trained agents.

4.4 Quality of Learned Reward Function (RQ3)

To understand why InitLight can improve both the effectiveness and generalization ability of RL models, we compared the rewards learned by our discriminator and the ground-truth rewards calculated by r = -PI according to the MP control theory. Figure 5 compares the ground-truth rewards with the learned rewards for single-intersection datasets in the last round (i.e., the 20th round) of pre-training. Note that here the horizontal axis indicates the step indices, where each episode (with one-hour simulation) of a dataset involves 360 steps, since a reward is calculated at the end of each control phase (with 10 seconds). From this figure, we can find that the learned reward trend approximates the ground-truth reward trend. The reason for such consistency is mainly because InitLight can learn an accurate reward function from expert trajectories to guide the pre-training process of the agent.

Figure 6 compares the average accumulative learned rewards with the average accumulative ground-truth rewards of all the intersections obtained within each episode. Note that, for a dataset, a reward of an episode is the summation of 360 rewards calculated within the same episode for all the dataset intersections. From this figure, we can find that the learned reward trends are similar to the ground-truth reward trends for all the datasets. It means that the discriminator learned by InitLight can recover the ground-truth reward function from expert trajectories, which is robust to various kinds of complex environments.

5 Conclusions

To address the problems of slow convergence and poor model generalization ability existing in Reinforcement Learning (RL)-based Traffic Signal Control (TSC), this paper proposed a novel AIRL-based method named InitLight, which enables the pre-training of an initial model to accelerate the global training process of RL-based TSC methods. Since InitLight trains the initial model only on a limited set of singleintersection environments, the generation time of the initial model is negligible. Meanwhile, since the reward function learned by InitLight can recover the ground-truth TSC rewards at optimality for different intersections, the obtained agent by InitLight can be deployed on different kinds of complex traffic environments to accelerate their overall RL processes. Comprehensive experimental results validate the effectiveness of InitLight.

Acknowledgments

This work was supported by the Natural Science Foundation of China (62272170), Shanghai Trusted Industry Internet Software Collaborative Innovation Center, "Digital Silk Road" Shanghai International Joint Lab of Trustworthy Intelligent Software (22510750100), and Natural Science Foundation (NSF CCF-2217104). Mingsong Chen is the corresponding author (mschen@sei.ecnu.edu.cn).