Full Bayesian Significance Testing for Neural Networks in Traffic Forecasting

1 Introduction

With substantial amounts of daily traffic data, including flow, volume, and speed, collected via city sensors, Intelligent Transportation Systems (ITS) [Mori et al., 2015] have come to the forefront in meeting the mounting challenges presented by ever-increasing transportation network demands. Traffic forecasting, a core constituent of ITS, aims to extrapolate future traffic conditions based on historical data and existing road networks. Its crucial role spans across traffic management, planning, and control functionalities.

In traffic forecasting, earlier methods primarily depend on statistical models [Ahmed and Cook, 1979; Min and Wynter, 2011; Cressie and Wikle, 2015], albeit these are limited in deployment due to stringent data assumptions and constrained abilities to capture intricate non-linear correlations. In recent years, the advances of deep learning have significantly enhanced the task completion of current methods. They benefit from the powerful capabilities of Graph Neural Networks (GNNs) to extract spatial dependencies from graphs [Kipf and Welling, 2017; Jiang and Luo, 2022; Song et al., 2020; Xu et al., 2020] and the sequence learning techniques to capture temporal dependencies [Gehring et al., 2017; Wu et al., 2019; Bai et al., 2020; Li et al., 2018].

∗Corresponding author

Despite the considerable success achieved by existing methods in prediction performance, the considerations of interpretability and uncertainty in traffic forecasting have been consistently disregarded. The bulk of these methods merely provide deterministic predictions, failing to account for the essential factor of uncertainty. However, the spatial-temporal relationships found in traffic data exhibit a high degree of complexity and diversity. It frequently leads models to incorporate unstable correlations and noise information in data, subsequently resulting in erratic model performance. Therefore, understanding the inherent evolution schema in traffic forecasting and identifying the pivotal traffic factors become imperative. Furthermore, this comprehension forms the foundation for effectively optimizing the modeling of true relationships in traffic forecasting. Despite the growing concerns regarding the uncertainty of prediction results [Liu et al., 2023; Qian et al., 2023; Wu et al., 2021], the interpretability of traffic forecasting remains an unresolved challenge.

Significance testing is a powerful tool in statistics to tackle the problem. It aims to determine whether a proposition about the population distribution 1 is true or false given observations, which is widely used in many scientific fields. However, traditional significance testing is restricted by assumptions about the true relationship and the derivation of complicated, even intractable, theoretical distributions, thus imposing strict limitations on the model. Hence, significance testing in spatial-temporal forecasting predominantly relies on shallow models, such as ARIMA. Recently, [Horel and Giesecke, 2020] and [Liu et al., 2024] have successfully introduced neural networks into significance testing, yielding impressive results. However, there is a dearth of research on significance testing for deep learning in traffic forecasting.

To fill this gap, we propose to conduct Full Bayesian Significance Testing for Neural Networks in Traffic Forecasting, namely ST-nFBST. It is a Bayesian framework incorporating spatial-temporal modeling, uncertainty quantification, and significance testing. First, we approach traffic forecasting through Bayesian modeling and employ a Bayesian neural network to capture complicated traffic relationships. Then, the framework is optimized in the context of heteroscedastic aleatoric uncertainty and epistemic uncertainty, decomposing the prediction risks derived from the model and data, respectively. Thereby, a quantitative analysis of both uncertainties can be implemented. Finally, a delicate grad-based Bayesian evidence is calculated based on the posterior distribution of parameters, and the testing results would contribute to reconstructing input graph signals. Through the analysis of experimental results with real data, we have successfully identified the inherent schema that impacts traffic forecasting on marginal and central nodes from both temporal and spatial dimensions. Building upon this discovery, we optimize the traffic prediction model, significantly reducing both model uncertainty and prediction errors.

1 https://online.stat.psu.edu/stat462/node/249/

The main contributions can be summarized as follows:

• To the best of our knowledge, we are the first to capture the inherent evolution schema in traffic forecasting through significance testing for neural networks.
• We propose ST-nFBST, a Bayesian framework incorporating spatial-temporal modeling, uncertainties quantification, and significance testing. It effectively improves predictive performance while reducing uncertainty.
• ST-nFBST can effectively detect changes in traffic conditions through uncertainty analysis, identify the inherent traffic schema and provide guidance for reconstructing the input graph signals through significance testing.
• Extensive experiments are conducted to verify the advantages of our approach in terms of traffic forecasting and significance testing.

2 Preliminaries

In this section, we first introduce basic notations in this paper. Then, we formalize the problem of traffic forecasting.

2.1 Notations and Definitions

Definition 1 (Sensor Network). We represent the sensor network as a weighted directed graph G = (V, E,W), where V = {v1, . . . , v|V|} is a set of |V| nodes, each node vi representing a sensor, E ⊆ V × V is a set of edges, each ei,j = (vi, vj) representing the correlation between sensors vi and vj, and W ∈ R|V|×|V| is a weighted adjacency matrix representing the nodes proximity (e.g., a function of their road network distance).

Definition 2 (Traffic States). Denote the traffic states observed on G as a graph signal x = (x1, . . . , x|V|) ∈ R|V| , such as velocity, volume, and flow. Traffic states can be regarded as multivariate time series. In this context, x(t) denotes the graph signal observed at time t, representing the values obtained from all sensors in the sensor network at time t; x(t) i represents the value obtained from the i-th sensor in the sensor network at time t.

2.2 Problem Statement

Problem 1 (Traffic Forecasting). The goal of traffic forecasting is to predict the future traffic state given previously observed traffic states from |V| correlated sensors in the sensor network. Let X<t = {x(t-τ1+1) , . . . , x(t) } ∈ Rτ1×|V| be the corresponding historic input sequence with τ1 steps. Similarly, X>t = {x(t+1) , . . . , x(t+τ2) } ∈ Rτ2×|V| represents the prediction sequence, where τ2 denotes the prediction horizon. The traffic forecasting problem aims to learn a function f(·) that maps τ1 historical graph signals X<t to future τ2 graph signals X>t, given a graph G:

For convenience, we will simplify X<t as Xt and X>t as Y t in the following discussion.

3 Methodology

In this section, we introduce the proposed ST-nFBST framework (Figure 1). We start with the modeling of spatial and temporal dependencies in a Bayesian perspective. Then, we introduce the aleatoric uncertainty and epistemic uncertainty and formulate the optimization problem in the context of both uncertainties. Finally, we display how to conduct significance testing through a delicate grad-based evidence value.

3.1 Bayesian Spatial-Temporal Modeling

Instead of regarding traffic forecasting as deterministic, we model it from a Bayesian perspective. Given a dataset of n samples D = {(X1, Y 1), . . . , (Xn, Y n)}, we use a Bayesian neural network (BNN) [Denker and LeCun, 1990; MacKay, 1992; Neal, 1995], whose parameters θ follow a distribution rather than deterministic values, to represent the underlying function f0 : Rτ1×|V| → Rτ2×|V| . For a pair of observations (Xt, Y t) at time point t, the regression process can be modeled as corrupted with Gaussian random noise:

The Gaussian assumption is common and computationally stable for regression tasks [Lakshminarayanan et al., 2017; Kendall and Gal, 2017; Gal and Ghahramani, 2016]. We assume the observation noise can vary with inputs Xt at different time points, namely heteroscedastic. In traffic forecasting, the traffic state highly depends on complicated contexts, such as congestion, weather conditions, and unexpected traffic events, resulting in substantial variability. Consequently, heteroscedastic modeling is more aligned with real scenarios. Given a new case Xt, the prediction made by the Bayesian neural network is the weighted average of an ensemble

where Θ is the whole parameter space.

As shown in Figure 1, the architecture includes an encoder and two independent decoders, respectively modeling spatialtemporal dependencies and observation noise. We model the spatial dependency by relating traffic flow to a diffusion process, which explicitly captures the stochastic nature of traffic dynamics. The diffusion convolution operation over a graph signal x and a filter fθ is defined as [Li et al., 2018]

where k is the diffusion step, θ ∈ RK×2 is the parameter for the filter, and DOW,D-1 I W ⊤ represent the transition matrices of the diffusion process and the reverse one. Then, we leverage the Diffusion Convolutional Gated Recurrent Unit (DCGRU) [Li et al., 2018] to model the temporal dependency

Based on DCGRU and Sequence to Sequence architecture, we build an encoder and two independent decoders. They all adhere to BNNs and the random outputs of the model f are

where θ ∼ P(θ|D). Besides, we add a ReLU-like clip layer after the Sigma-Decoder to ensure that ˆσ0(Xt) is always positive. Specifically, ˆσ0(Xt) = max(ˆσ0(Xt), τ), where τ is the threshold to control the precision.

3.2 Optimization with Uncertainties

In Bayesian modeling, there exist two primary categories of uncertainty, i.e., aleatoric and epistemic. The former represents data uncertainty, while the latter represents model uncertainty. Our framework incorporates both uncertainties into the optimization of traffic forecasting.

Aleatoric Uncertainty. Aleatoric uncertainty is caused by the intrinsic randomness of data, which cannot be reduced even if more data were to be collected. For example, it could be caused by sensor noise inherent in the observations. Under the assumption of Eq (2), the aleatoric uncertainty is exactly equal to the variance of the noise

Epistemic Uncertainty. Epistemic uncertainty accounts for uncertainty in the model parameters, which arises from that lack of data or model misspecification. Fortunately, this uncertainty can be explained away given enough data. In traffic forecasting, the epistemic uncertainty can be captured by the predictive variance:

Before optimization, a prior distribution is assigned to model parameters θ as an initial belief π(θ) according to experience. This belief is gradually adjusted to fit data D by using the Bayesian rule. The final belief is presented as the posterior distribution

The main challenge of optimization lies in the difficulty of solving the integral in Eq (9) in practice. A popular way, known as Variational Inference (VI), entails approximating the real but intractable posterior distribution with a tractable distribution called variational distribution [Blei et al., 2017]. Therefore, Eq (9) could be efficiently approximated. Formally, variational family Q = {qϑ : ϑ ∈ Γ} is a predefined family of tractable distributions on model parameter space Θ, where ϑ is the parameter of variational distribution and Γ is the range of ϑ. The optimal variational distribution qϑ∗ is chosen from Q such that

KL divergence describes the "distance" between two distributions. We set diagonal Gaussian distributions as the prior and variational families of parameter θ. This assumption is common in many works [Blundell et al., 2015; Kendall and Gal, 2017]. Under this assumption, Eq (10) can be further simplified as (excluding the constant logP(D))

The first term is related to data, which is equivalent to Mean Squared Error (MSE) with a scaling factor 1 2σ2 0 under the assumption Eq (2). The second term is only related to parameters ϑ. The detailed derivation is shown in the Appendix.

In the end, we finish approximating the posterior distribution of parameters P(θ|D) with variational distribution qϑ∗(θ). Using Monte Carlo integration, the aleatoric uncertainty and epistemic uncertainty are approximated as follows:

where ˆσ0(Xt), ˆ Y t are the outputs of two decoders obtained by sampling m times drawn from θ ∼ qϑ∗(θ), respectively.

3.3 Significance Testing

To capture the traffic evolution schema, we propose to conduct significance testing in traffic forecasting. However, there are two main defects in classical significance testing.

• First, the effectiveness of classical significance testing is based on reasonable assumptions about f0, such as stationarity and invertibility (ARIMA). However, it is difficult to find such precise and suitable assumptions when the data distribution is complicated such as traffic forecasting.
• Second, some models, such as deep learning, excel in accurately fitting complex data distributions. However, the more complex the assumption of f0, the more computational derivation of theoretical distribution, which is inevitable for classical significance testing.

Fortunately, nFBST replaces complicated theoretical derivation by fitting distributions in a Bayesian way, and the neural network serves as a good estimator of f0 without assuming specific forms [Liu et al., 2024].

In traffic forecasting, to test the significance of input features Xt, we first need a reasonable measure as the testing statistic to represent the true relationship between features and targets. In this paper, we conduct the Bayesian significance testing using Grad-nFBST, which adopts the gradient as the instance-wise testing statistic

where η(Xt, i, k1, j, k2) ∈ R represents the testing statistic for the velocity of the k1-th sensor at future time t + i based on the velocity of the k2-th sensor at historical time t - j (abbreviated as η). The testing problem is formulated as:

We denote the whole space of η as Ψ such that η ∈ Ψ. In Section 3.2, we have approximated the posterior distribution of parameters P(θ|D) with variational distribution qϑ∗(θ). Based on Eq (13), we can obtain the approximated distribution of the testing statistic η further. We denote p(η|D) as its probability density and define the region whose probability greater than p(η = 0|D) according to the following formula:

where p(η = 0|D) should be the maximum of the posterior density under the null hypothesis H0. A valid Bayesian evidence for the null hypothesis H0 can be calculated as follows [De Braganc¸a Pereira and Stern, 1999; Liu et al., 2024]:

where 1(·) is the indicator function. Using Monte Carlo integration, the above formula can be further simplified to

where ηi is obtained by sampling m times based on the posterior probability density p(η|D). The result of Eq (17) is called Bayesian evidence, whose value is between 0 and 1. The closer the Bayesian evidence to 1, the more likely to accept H0. The closer the Bayesian evidence to 0, the more likely to reject H0.

Significance testing helps explore knowledge hidden behind the underlying relationships between features and targets in a rigorous manner. When integrated with uncertainty analysis, it facilitates a more precise identification of the pivotal factors influencing heightened or diminished predictive uncertainty. This, in turn, facilitates a systematic refinement of our models through a reconstructed input graph signals X′t .

4 Experiments

In this section, we perform extensive experiments on two realworld large-scale datasets. We provide detailed analysis in terms of traffic forecasting and significance testing.

4.1 Datasets

METR-LA and PEMS-BAY are two standard benchmark datasets. In both datasets, the traffic speed data are aggregated into 5-minute intervals, and both the historical input window and the prediction window are set to 1 hour. The datasets are split into three parts, with a ratio of 7:2:1 for training, validation, and testing, respectively. We use METRLA as the default dataset in some analyses, and the results in PEMS-BAY are provided in the Appendix.

Based on the distance between sensors and the quantity of neighboring sensors, we categorize sensors into two types. The first type is referred to as "marginal sensors", positioned at the periphery of the sensor network or in the middle of a spacious road. The second type is "central sensors", surrounded by numerous neighboring sensors. In our experiment, the division hinges on whether the number of sensors within 4,000 miles exceeds 4, as depicted in the Appendix.

4.2 Baselines for Comparison

To compare the performance of ST-nFBST, we adopt the following widely used time series models as the baselines:

• Non-deep learning methods: Historical Average (HA), Auto-Regressive Integrated Moving Average model with Kalman filter (ARIMA), Vector Auto-Regression (VAR), and Linear Support Vector Regression (SVR).
• LSTM [Sutskever et al., 2014]: The encoder-decoder framework incorporating LSTM and FC layers.
• DCRNN [Li et al., 2018]: Diffusion Convolutional Recurrent Neural Network captures temporal dependencies through graph convolutions formalized by the diffusion process and captures spatial dependencies using an encoder-decoder framework.
• STGCN [Yu et al., 2018]: Spatial-Temporal Graph Convolutional Network integrates gated temporal and graph convolution to capture spatial-temporal correlations.
• STTN [Xu et al., 2020]: Spatial-Temporal Transformer Network employs a Transformer structure for temporal and spatial modeling in traffic prediction.
• AGCRN [Bai et al., 2020]: Adaptive Graph Convolutional Recurrent Network adaptively learns node-specific parameters for graph convolution.
• CCRNN [Ye et al., 2021]: Coupled Layer-wise Convolutional Recurrent Neural Network captures multi-scale spatial and temporal dependencies.
• DeepSTUQ [Qian et al., 2023]: Deep Spatio-Temporal Uncertainty Quantification unifies uncertainty quantification and traffic prediction using Monte Carlo Dropout [Gal and Ghahramani, 2016].

4.3 Traffic Forecasting Results

We first provide a performance comparison of ST-nFBST with baselines on two datasets, METR-LA and PEMS-BAY. Subsequently, we conduct a more detailed analysis of the prediction results using uncertainty analysis.

Performance Comparison. We evaluate models by predicting traffic speeds ahead on multi-steps, at intervals of 15 minutes, 30 minutes, and 1 hour. The performance is evaluated by three commonly used metrics: Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and Root Mean Squared Error (RMSE). Missing values are excluded in calculating these metrics. Table 1 shows the comparison results with different baselines on two datasets. We have the following observations:

• Deep learning methods outperform non-deep learning methods due to the powerful representing capability of neural networks. The poor performance of LSTM compared with other deep learning models indicates the limitation of employing temporal correlations only.
• ST-nFBST nearly outperforms all other methods across all metrics and all forecasting horizons on both datasets, validating its effectiveness.
• As the prediction horizon expands, the improvement of STnFBST compared to methods that do not model uncertainty (excluding DeepSTUQ) becomes more apparent, validating the effectiveness of modeling uncertainty. For example, compared to CCRNN, the improvement of MAE at 15 minutes, 30 minutes, and 1 hour is 0.14, 0.15, and 0.21 respectively on the METR-LA dataset.

Uncertainty Analysis. In traffic forecasting, most existing methods only provide deterministic traffic predictions without uncertainty. Through the quantitative and visual analysis of uncertainty derived from ST-nFBST, we have uncovered more insights. Figure 2 illustrates the trends of MAE and uncertainties under different prediction horizons. Sensors 0 and 26 represent central and marginal types, respectively. We have the following observations:

• The fluctuation of MAE closely aligns with the variations in uncertainty across diverse forecasting horizons. This implies that our method accurately captures uncertainties. Considering that we thoroughly leverage both uncertainties in the optimization process by incorporating them into

• (d) Predict 15min after for sensor 26 (e) Predict 30min after for sensor 26 (f) Predict 60min after for sensor 26

Figure 2: Comparison between uncertainties and |error| for different forecasting horizons on the METR-LA dataset.

the loss function, this further explains why our approach outperforms other state-of-the-art models.

• In most cases, the aleatoric uncertainty is much larger than the epistemic uncertainty, accounting for the majority of uncertainty. This implies that the traffic uncertainty is mainly data-related. Thanks to the powerful representation capability of neural networks, we can train models effectively to reduce epistemic uncertainty.
• There are some cases where error and uncertainties manifest inconsistently, specifically, with low error but high uncertainties. Moreover, epistemic uncertainty is high in such cases. This occurs when changes in road conditions at intersections, such as traffic congestion or unexpected accidents, result in a sharp decrease in the speed of other sensors. This implies that uncertainty, compared to error, is more effective in monitoring changes in road conditions. A more detailed case study is presented in the Appendix.
• As the forecasting horizon expands, both error and uncertainties of central sensors increase correspondingly, while those of marginal sensors remain relatively stable. However, the predictions of marginal sensors fluctuate significantly at different times. This aligns with the definition of two types of sensors. The marginal sensors exhibit fewer impacts from other sensors while utilizing less information from other sensors. However, the central sensors are more susceptible to the influence exerted by other sensors, rendering predictions more intricate.

4.4 Significance Testing Results

In order to capture the traffic evolution schema, we conduct significance testing from two dimensions: Temporal and Spatial. From Section 2.2, we know that the model's input consists of two parts: a sensor network and a historical sequence composed of observations. Since the sensor network is established once the task is determined (the computation of the weighted adjacency matrix W is calculated before training), our focus is primarily on the testing of Xt.

Testing in the Temporal Dimension. In this experiment, we aim to identify the impact of observations at different times ahead in the input sequence on the prediction. By conducting significance testing on the historical moments of a sensor, we observe that recent historical data (within 30min) generally exhibits significance, yet this may not hold for data over a longer time. Then, we remove different sizes of the historical inputs to reconstruct the input graph signals and retrain using the same network structure. The results are illustrated in Table 2. We have the following observations:

• For short-term forecasting, the historical observations of the past 15 minutes are crucial. The removal of these observations significantly degrades performance and uncertainty, with a worsening effect as the removal window increases. This indicates that short-term forecasting highly depends on historical data of its own.
• For middle-term forecasting, the performance of marginal sensors decreases, while the performance of central sensors remains unchanged or decreases within an acceptable range. This indicates that middle-term forecasting integrates information from both its own and surrounding sensors. When excluding its own historical data, the central sensor can utilize more information from other sensors.
• For long-term forecasting, removing a sensor's historical data demonstrates negligible alterations in performance, even a slight improvement. This indicates that long-term forecasting does not focus on the historical data of itself

but instead utilizes information about the surroundings.

Testing in the Spatial Dimension. In this experiment, we aim to identify the impact of sensors at different locations on the prediction. The testing results indicate that short-term traffic forecasting is more concerned with the road conditions where their sensors are located, but do not pay attention to the situation at intersections. However, the changes at intersections take time to propagate to other road segments and, hence may have a significant impact on long-term forecasting. A more detailed analysis is shown in the Appendix.

Based on the above conclusion, we selectively remove the insignificant sensors to reconstruct the input graph signals and retrain using the same network structure. As illustrated in Table 2, the model's predictive performance and uncertainty significantly improve after retraining on the reconstructed graph signals. Furthermore, due to the rigorous modeling of true relationships and the exclusion of noise that may interfere with the modeling process, the performance is more stable. This validates the efficacy of using significance testing to guide a more deliberate refinement of traffic forecasting.

5 RelatedWork

Traffic forecasting has always been a challenging task due to its complex spatial and temporal dependencies [Wang et al., 2018; Wang et al., 2019b; Ji et al., 2020; Wang et al., 2021; Wang et al., 2023a; Wang et al., 2023b]. Early models in traffic forecasting primarily rely on statistical models [Cressie and Wikle, 2015], such as ARIMA [Ahmed and Cook, 1979; Min and Wynter, 2011] and Bayesian models [Wang et al., 2014]. Recent years have witnessed the emergence of deep learning models to model spatial and temporal dependencies. To extract spatial dependency, some works attempt to use CNN by converting road networks to regular grids [Zhang et al., 2017; Zhang et al., 2019; Ma et al., 2017], while others use GNN and its variants (such as GCN and GAT) by representing road networks as graphs [Jiang and Luo, 2022; Atwood and Towsley, 2016; Kipf and Welling, 2017; Song et al., 2020; Bai et al., 2020; Li et al., 2018; Yu et al., 2018; Li and Zhu, 2021; Wu et al., 2020b; Jiang et al., 2023c; Jiang et al., 2023b; Wu et al., 2020a; Ji et al., 2023]. To extract temporal dependency, some works use RNN and its variants (such as LSTM and GRU) [Ma et al., 2015], while others integrate with convolutional sequence learning or Transformer [Jiang et al., 2023a; Gehring et al., 2017; Ye et al., 2021; Wu et al., 2019; Xu et al., 2020].

In recent years, there has been an increasing focus on exploring the uncertainty in traffic forecasting [Qian et al., 2023; Zhou et al., 2021; Wang et al., 2019a; Liu et al., 2023; Wu et al., 2021]. There are also numerous researches on significance testing for neural networks from parametric [White, 1989b; White, 1989a; Olden and Jackson, 2002] or non-parametric [Lavergne and Vuong, 1996; Lavergne and Vuong, 2000; Fan and Li, 1996; Racine, 1997]. However, testing is limited by the non-identifiability of neural networks or certain assumptions. [Horel and Giesecke, 2020; Liu et al., 2024] have yielded impressive results from Frequentist and Bayesian perspectives respectively. Nevertheless, a notable gap persists in the realm of traffic forecasting.

6 Conclusion

In this paper, we propose to conduct the Full Bayesian Significance Testing for neural networks in Traffic Forecasting, called ST-nFBST. It is a Bayesian framework incorporating spatial-temporal modeling, uncertainty quantification, and significance testing. To the best of our knowledge, we are the first to capture the inherent evolution schema in traffic forecasting through significance testing for neural networks. Moreover, it can effectively detect changes in traffic conditions, identify the inherent traffic schema, and provide guidance for reconstructing the input graph signals.

Acknowledgments

This work was supported by the National Key R&D Program of China (2021ZD0111200). Prof. Wang's work was supported by the National Natural Science Foundation of China (No. 72171013, 72222022, 72242101). Dr. He's work was supported by the China National Postdoctoral Program for Innovative Talents (BX20230195).