Spatio-Temporal Field Neural Networks for Air Quality Inference

1 Introduction

Real-time monitoring of air quality, such as PM2.5, PM10, and NO2 concentrations, is crucial for air pollution control and protecting human health, with air pollution contributing to seven million deaths annually according to the WHO [Vallero, 2014]. However, the deployment of air quality stations in urban areas is limited due to high costs, requiring around 200,000 USD for construction and 30,000 USD annually for maintenance [Zheng et al., 2013]. Additionally, these stations need significant land and dedicated personnel for upkeep, further limiting their prevalence in cities.

In the past decade, substantial research endeavors have been directed towards air quality inference [Han et al., 2023], seeking to infer real-time air quality in locations devoid of monitoring stations by leveraging data gleaned from existing sites, as shown in Figure 2(b)-(c). With recent advancements in deep learning, Graph Neural Networks (GNN) [Kipf and Welling, 2016a] have become dominant for non-Euclidean data representation, particularly in learning complex spatial correlations among air quality monitoring stations. Integrating GNNs with temporal learning modules (e.g., RNN [Graves, 2013], TCN [Bai et al., 2018], ODE [Liang et al., 2022]) has led to the development of Spatio-Temporal Graph Neural Networks (STGNN)[Wang et al., 2020; Jin et al., 2023], addressing the dynamic nature of air quality data across spatial and temporal dimension. STGNNs, exemplified in studies like [Han et al., 2021; Hu et al., 2023b] offer superior representation extraction and flexibility in cross-domain data fusion.

∗Corresponding author. Email: yuxliang@outlook.com

Though promising, STGNNs simply treat air quality data as a Spatio-Temporal Graph (STG), as shown in Figure 1(a). However, these models overlook a crucial property - continuity, which manifests across both spatial and temporal dimensions. In reality, air quality readings of stations are sampled from a continuous Euclidean space and cannot be fully encapsulated by a discrete graph structure using GNNs. Meanwhile, the temporal modules (e.g., RNN, TCN) in STGNNs exhibit the discrete nature as well, rendering them incapable of capturing continuous-time dynamics within data. To better represent the continuous and evolving nature of real-world air quality phenomena, a more powerful approach is needed, surpassing the discrete representation of STGNNs.

In this paper, we draw inspiration from Field Theory [McMullin, 2002] and innovatively formulate air quality inference from a field perspective, where air quality data is a physical quantity that can be conceptualized by a new concept called Spatio-Temporal Fields (STF), as depicted in Figure 1(b). These fields encompass three dimensions (i.e., latitude, longitude, time), assigning a distinct value to each point in spacetime. In contrast to STGs, STFs are characterized by being regular, continuous, and unified 1 , offering a representation more aligned with reality. Under this perspective, we can transform the air quality inference problem to reconstruct STFs from available readings using coordinate-based neural networks, particularly Implicit Neural Representations (INR) [Sitzmann et al., 2020; Xie et al., 2022].

While INRs effectively handle the continuity property of air quality data, they inevitably confront two primary challenges. Firstly, the generation process of air quality data is extremely complex and influenced by various factors (such as humidity and wind speed/direction), which poses a challenge for reconstructing the underlying STFs through neural representation methods. Secondly, empirical studies [Xu et al., 2019; Sitzmann et al., 2020] verify that INRs always exhibit a bias towards learning low-frequency functions, which will disregard locally varying high-frequency information and higher-order derivatives even with dense supervision.

To this end, we for the first time present Spatio-Temporal Field Neural Networks (STFNN), opening new avenues for modeling spatio-temporal fields and achieving state-of-theart performance in nationwide air quality inference in the Chinese Mainland. Targeting the first challenge, we pivot our focus from reconstructing the value of each entry in STFs to learning the derivative (i.e., gradient) of each entry. This strategic shift is inspired by learning the residual is often easier than learning the original value directly, as exemplified in ResNet [He et al., 2016]. Such vector field can not only show how the pollutant concentration varies across time and space but also the direction of diffusion. To tackle the second challenge, we endeavor to augment our STFNN with local context knowledge during air quality inference at a specific location. Specifically, we combine the STGNN's capability to capture local spatio-temporal dependencies with STFNN's ability to learn global spaito-temporal unified representations. This integration results in what we term Pyramid Inference, a hybrid framework that leverages the strengths of both models to achieve a more comprehensive inference of air quality dynamics with both high-frequency and low-frequency components. Overall, our contributions lie in three aspects:

1 It implies that the field representation accounts for variations not only across different locations in space but also over different points in time, emphasizing the comprehensive treatment of both spatial and temporal aspects within the unified framework.

• A Field Perspective. We formulate air quality as spatiotemporal fields with the first shot. Compared to STGs, our STFs not only adeptly capture the continuity and Euclidean structure of air quality data, but also achieves a unified representation across both space and time.
• Spatio-Temporal Field Neural Networks. We propose a groundbreaking network called STFNN to model STF data. STFNN pioneers an implicit representation of the STF's gradient, deviating from conventional direct estimation approaches. Moreover, it preserves high-frequency information via Pyramid Inference.
• Empirical Evidence. We conduct extensive experiments to evaluate the effectiveness of our STFNN. The results valiadte that STFNN outperforms prior arts by a significant margin and exhibits compelling properties. A system in Figure 2 has been deployed to show its practicality in the Chinese Mainland.

2 Preliminary

Definition 1 (Air Quality Reading) We use xit ∈ RD and yit ∈ R to denote the air quality readings and the concentration of PM2.5 from the i-th monitoring stations at time t separately. Here D encompasses various measurements, such as concentrations of other air pollutants (e.g., PM10, NO2), and meteorological properties (e.g. humidity, weather and wind speed). Xt = (x1t , x2t , . . . , xNt ) ∈ RN×D denotes the observations of all stations at a specified time t. X = (X1,X2, . . . ,XT ) ∈ RT×N×D denotes the observations of all stations at all time. Similar definitions apply to Yt and Y, mirroring Xt and X, respectively.

Definition 2 (Coordinates) A coordinate c = [lng, lat, t] ∈ R3 is used to represent the spatial and temporal properties of an air quality reading or a location, including longitude, latitude, and timestamp. These coordinates are categorized into two types: source coordinate csrc , associated with readings or locations with existing air quality monitoring stations, while target coordinate ctar , corresponding to unobserved locations requiring inference. Notably, cit represents the coordinate of the corresponding xit and yit . Parallel definitions apply to Ct and C in relation to Xt and X, respectively, which are not reiterated here.

Problem Definition The air quality inference problem addresses the utilization of historical data and real-time readings from a limited number of air quality monitoring stations to infer the real-time air quality anywhere, especially unobserved location. Traditional strategies [Hou et al., 2022; Hu et al., 2023a] employ graphs to illustrate the relationship between stations and locations, and the task is translated into a recovery task for the masked nodes (target locations), as shown in Figure 3 (a). In this paper, we revisit the problem from the field perspective, as shown in Figure 3 (b). Specifically, our goal is to reconstruct a spatio-temporal field G for air quality that is capable of mapping any arbitrary coordinate, especially ctar , to the corresponding concentration of PM2.5 ytar . Additional parameters, such as X, are allowed to enhance the inference process.

3 Methodology

3.1 Global View: Spatio-Temporal Field

A Spatio-Temporal Field (STF) is a global modeling of air quality that encompasses all stations and observation times. The STF function, denoted as f(·) : c ↦-→ q, assigns a unique physical quantity q to each coordinate. When q is a scalar, f(·) represents a scalar field. Conversely, if q is a vector, with magnitude and direction, it denotes a vector field.

Specifically, our focus lies on a scalar field G : R3 → R for air quality inference, where G maps the coordinates to the corresponding PM2.5 concentration. This representation facilitates a continuous and unified spacetime perspective, allowing for the inference of air quality at any location and time by inputting the coordinates.

Directly modeling G is challenging due to its intricate complexity and nonlinearity. Alternatively, it is often more feasible to learn its derivative, which refers to the gradient field of G in spacetime. We denote the gradient field as F ≜ ∇G which is a vector field. Notably, given a specific F, an array of G solutions exists unless an initial value is specified. We use ysrc and ytar to denote the PM2.5 concentration on csrc and ctar , respectively. Our primary focus lies in inferring ytar since the true value of ysrc is known and recorded while ytar remains undisclosed. To infer ytar , we utilize a ysrc as the initial value and assume l is a piecewise smooth curve in R3 that point from csrc to ctar , then we have

where · is the dot product, and r : [a, b] → l represents the position vector. The endpoints of l are given by r(a) and r(b), with a < b. Seeking an Implicit Neural Representation (INR) becomes the objective to fit F as F is usually intricate to the extent that it cannot be explicitly formulated.

3.2 Local View: Spatio-Temporal Graph

The formulation presented in Eq. (1) ensures the recoverability of ctar across arbitrary coordinates through curve integration, utilizing solely a single initial value. However, this approach yields an excessively coarse representation of the entire STF, resulting in the loss of numerous local details and high-frequency components The impact is particularly pronounced when ysrc is situated at a considerable distance from ytar as the increase in the length of l introduces a significant cumulative error. In response to this limitation, we leverage the potent learning capabilities of STGNN to capture local spatio-temporal correlations effectively. We employ the local spatio-temporal graph (STG) to model the spatio-temporal dependencies of the given coordinates and their neighboring air monitoring stations with their histories. The corresponding design can be found in the foundational work [Song et al., 2020], and it is not necessary to reiterate it here.

3.3 Hybrid Framework: Pyramidal Inference

We intend to integrate the continuous and uniform global modeling of spacetime provided by STF with the local detailing capabilities of STG, thereby establishing a hybrid framework that leverages the strengths of both approaches. Within the local STG, the estimation of ytar is achieved by leveraging information from its neighboring nodes through Eq. (1). By calculating estimates of ytar from these neighbors and assigning a learnable weight wi to each estimation (∑Ki=1 wi = 1), we enhance the precision of the inference result for a specific coordinate. This operation can be formulated as

where ˆytar is the joint estimation of ytar by neighbors. wi and ysrc i represent the weight and PM2.5 concentration of the ith neighbor, respectively. li is the integral path in R3 that points from the coordinates of the ith neighbor to the target coordinate.

We call the inference strategy represented by Eq. (2) Pyramidal Inference, which is the framework of the STFNN we propose. To demonstrate its sophistication, we deconstruct Eq. (2) into two steps:

and

where ˆyi is the estimate of ytar by the ith neighbor, W and ˆYtar represent the vector form of wi and ˆyi, respectively. These two operations can be viewed as follows. In Eq. (3), a curve integral is used over the gradient field to estimate ytar from each neighbor in a continuous and spatio-temporally uniform way, which takes advantage of the STF. In Eq. (4), the information from the neighbors is aggregated throughW, which considers the spatio-temporal dependencies between the nodes and takes advantage of the STG. In this way, the Pyramidal Inference framework combines the two different spacetime perspectives into a distinctive new paradigm.

4 Implementation

Upon introducing the formulation of Pyramidal Inference, we proceed to its implementation through a meticulously designed model architecture, depicted in Figure 4. The model comprises three pivotal components:

• Spatio-Temporal Encoding. This component transforms coordinates into coded vectors endowed with representational meaning, enhancing the network's ability to comprehend and leverage the spatio-temporal characteristics of coordinates.
• Ring Estimation: Implementation of Eq. (3), mapping the encoded vector to the gradient of the STF. This process yields each neighbor's estimate of the PM2.5 concentration for the target coordinates through a path integral.
• Neighbor Aggregation: Implementation of Eq. (4), utilizing the coded vectors of neighbors and target coordinates as inputs to derive the estimated weights for each neighbor concerning the target coordinates.

In the following parts, we will provide a detailed exposition of each module, elucidating their functionalities step by step.

4.1 Spatio-Temporal Encoding

We revisit the previously introduced local STG, which amalgamates nodes across different time steps into a unified graph, potentially obscuring the inherent temporal properties of individual nodes. In essence, this local STG places nodes from diverse time steps into a shared environment without discerning their temporal distinctions. However, this issue can be mitigated through meticulous positional coding of nodes [Gehring et al., 2017; Song et al., 2020].

We use p ∈ R10 to denote the coding vector, essential for accurately describing the spatio-temporal characteristics of a node or coordinate. This vector is expressed as the concatenation p = [pS, pT ], where pS represents spatial coding and pT represents temporal coding. In the spatial dimension, a node's properties can be captured by its absolute position, represented by z-normalized longitude lngz and latitude latz, forming pS = [lngz, latz]. For encoding temporal information pT , sinusoidal functions with different periods are employed, reflecting the periodic nature of temporal phenomena. We utilize a set of periods T = {1a, 7a, 30.5a, 365a}, with a as the scaling index, to represent days, weeks, months, and years. The temporal coding pT is then represented as

where p(T,i) denotes the value of the ith dimension (1 ≤ i ≤ 8) of pT , int(i/2) denotes dividing i by 2 and rounding down, and Tint(i/2)+1 is the int(i/2) + 1 period of T.

4.2 Ring Estimation

Motivation. We employ the continuous approach of curve integration over the gradient to determine the PM2.5 concentration at the target coordinate in Eq. (3). However, due to inherent limitations in numerical accuracy within computing systems, achieving true continuity in STF becomes unattainable. Therefore, we have adopted an incremental approach. We set the integral path to be a straight line from the neighbors' coordinate csrc to the target coordinate ctar for convenience, then the unit direction vector ⃗r in the path can be written as ⃗r ≜ (ctar -csrc ) / (∥ctar -csrc ∥). After that, we replace the integral operation with a summation operation and modify Eq. (3) to a discrete form

where m represents the step size of the summation and csrc i is the coordinate of the ith node in the local STG. Di,j ∈ R3 represents the difference at the jth step of the ith node, which is the discrete approximation of the gradient. Our objective is to build a module for estimating Di,j, which is the only unknown in Eq. (6).

Overview. Towards this objective, we introduce a pivotal module named Ring Estimation, designed for the joint estimation of the differences [D·,j]:K = [D1,j, · · · ,DK,j] ∈ RK×3 at the jth step of all neighbors. We posit that simultaneous estimation of [D·,j] : K enhances inference efficiency and captures correlations between them, thereby reducing estimation errors compared to individually estimating Di,j for a single neighbor K times. Specifically, the Ring Estimation module divides the polygon surrounded by K neighbors around the target coordinate into m ring zones from the outermost to the innermost, with total mK of transition nodes (coordinates) uniformly spaced along the inference path. The inner edge of the jth ring zone serves as the outer edge for the (j + 1)th zone. Like the target coordinate, the transition nodes lack features and labels (PM2.5 concentration). They serve as the intermediary states and springboards in the process of estimating ˆytar . By increasing the value of m, the Ring Estimation block facilitates the inference in an approximately continuous manner.

4.3 Neighbor Aggregation

It is advisable to assign varying weights W = [w1, . . . , wK] to the estimations of the target nodes based on the different spatio-temporal scenarios in which they are situated. To this end, we present the Neighbor Aggregation module, which takes into account the coding of the coordinates of the neighbors and the target coordinate and employs end-to-end learning to compute the estimation weights wi of each neighbor on the target node. To obtainW, we first multiplied the output by WN ∈ R10×1 to transform it into a Logit score. Then, we applied the Softmax operation to ensure that the weights sum up to one. In this end, the formulation of the Neighbor Aggregation can be written as

5 Experiments

In this section, we delve into our experimental methodology aimed at evaluating the performance and validating the efficacy of the STFNN. Specifically, our experiments are designed to explore the following research questions, elucidating key aspects of our approach and its applicability in realworld scenarios:

• RQ1: How does STFNN's approach, focusing on inferring concentration gradients for indirect concentration value inference, outperform traditional methods in terms of accuracy and effectiveness?
• RQ2: What specific contributions do the individual components of STFNN make to its effectiveness in inferring air pollutant concentrations?
• RQ3: How do variations in each hyperparameter impact the overall performance of STFNN?
• RQ4: Do the three-dimensional hidden states learned by the model accurately represents the gradient of the spatio-temporal field?
• RQ5: Can our model demonstrate proficient performance in inferring concentrations of various air pollutants, including NO2?

5.1 Experimental Settings

Datasets

The study obtained a nationwide air quality dataset [Liang et al., 2023] from January 1st, 2018, to December 31st, 2018. This dataset includes air quality and meteorological data. The input data can be divided into two classes: continuous and categorical data. Continuous data includes critical parameters such as air pollutant concentrations (e.g., PM2.5, CO), temperature, wind speed, and others. Categorical data encompasses weather, wind direction, and time. In the event of an unanticipated occurrence, such as a power outage, some data may be unavailable.

5.2 Baselines for Comparison

We compare our STFNN with the following baselines that belong to the following four categories:

• Statistical models: KNN [Guo et al., 2003] utilizes nonparametric, instance-based learning, inferring air quality by considering data from the nearest neighbors. Random Forest (RF) [Fawagreh et al., 2014] aggregates interpolation from diverse decision trees, each trained on different dataset subsets, providing robust results.
• Neural Network based models: MCAM [Han et al., 2021] introduces multi-channel attention blocks capturing static and dynamic correlations.
• Neural Processes based models: SGNP, a modification of Sequential Neural Processes (SNP) [Singh et al., 2019], incorporates a cross-set graph network before aggregation, enhancing air quality inference. STGNP [Hu et al., 2023a] employing a Bayesian graph aggregator for context aggregation considering uncertainties and graph structure.

• AutoEncoder based models: VAE [Kingma and Welling, 2022] applies variational inference to air quality inference, utilizing reconstruction for target node inference. GAE [Kipf and Welling, 2016b] reconstructs node features within a graph structure, while GraphMAE [Hou et al., 2022] introduces a masking strategy for innovative node feature reconstruction.

5.3 Hyperparameters & Setting

To mitigate their impact, instances exceeding a 50% threshold of missing data at any given time were prudently omitted from our analysis. Our dataset was carefully partitioned into three segments: a 60% training set, a 20% validation set, and a 10% test set. During training, in each epoch, we randomly select stations with ratio α, mask their features and historical information, and let them act as the target node. We ignore all locations where PM2.5 (or NO2 in the case of RQ5) is missing. The model is trained with an Adam optimizer, starting with a learning rate of 1E-3, reduced by half every 40 epochs during the 200 training epochs. The batch size for training is set to 32. The hidden dimension of MLP and all the Transformer-Decoder networks is fixed at 64. For Ring Estimation, the neighbor number is set to 6, incorporating the past 6 timesteps, and the iteration step m is defined as 16.

5.4 Model Comparison (RQ1)

In addressing RQ1, we conduct a meticulous comparative analysis among models based on the evaluation metrics of MAE, RMSE, and MAPE. The empirical outcomes derived from this analysis are systematically presented across the expansive spectrum of the nationwide air quality dataset, meticulously documented within Table 1.

The results indicate that STFNN consistently demonstrates enhanced efficacy across various evaluation metrics, surpassing existing baseline models. Table 1 shows that our approach reduces MAE under three different mask ratios (25%, 50%, and 75%) in comparison to GraphMAE, establishing a new State-of-the-Art (SOTA) in nationwide PM2.5 concentration inference in the Chinese Mainland. In our view, there are three main reasons for this. First, the gradient field is a better representation of reality. Second, the spatial and temporal modules of STFNN capture both types of information together, avoiding bias or information loss. Finally, our Pyramidal Inference framework captures global and local spatiotemporal properties, which helps us model the pollutant concentration field more accurately.

5.5 Ablation Study (RQ2)

To assess the contributions of individual components to the performance of our model and address RQ2, we conducted ablation studies. The findings from these studies are presented in Figure 5 (a).

Effects of meteorological features. To analyze the impact of meteorological features on the accuracy of the final model, we removed them from the raw data. Therefore, the gradient was obtained solely from the spatio-temporal coordinates of neighboring stations fed into the gradient encoder. The figure demonstrates that removing the meteorological features resulted in some improvement in the model's mean absolute error (MAE), which still outperformed all baseline models.

Effects of Neighbor Aggregation. To investigate the impact of a dynamic and learnable implicit graph structure on the model, we substituted the model's Neighbor Aggregation module with IDW and SES, a non-parametric approach inspired by Zheng et al [Zheng et al., 2013]. This approach employs implicit graph relations that are static. The results depicted in Figure 5 (a) demonstrate that utilizing the Neighbor Aggregation module results in a significant decrease in MAE.

5.6 Hyperparameters Study (RQ3)

In this section, we comprehensively explore the effects of various hyperparameters on the model's performance, thereby addressing RQ3.

Effects of Hidden & FFD Dimension. We adjusted the hidden layer dimension of the Spatio-Temporal Encoding module and the forward propagation of the Transformer-Decoder structure used by the Ring Estimation and Neighbor Aggregation modules from 16 to 64. The results in Figure 6 (a-b) show that adjusting the hidden layer dimension has little effect on the absolute values of MAE and RMSE, but it significantly decreases MAPE.

Effects of Step Size. We vary the value of the accumulation step size of the Ring Estimation in {2, 4, 8, 16}. The result is shown in Figure 6 (c). It has been observed that as the step size increases, the model's performance initially declines before improving. Additionally, when comparing 2 steps to 16 steps, we note that the model's training time per round is approximately 50% longer for 16 steps.

Effects of Neighbors Number. We vary the number of neighbors from 2 to 8, and the result is shown in Figure 6 (d). We observe that the performance of our model improved as the number of neighbors increased. Notably, our proposed STFNN exhibited excellent performance even with a small number of neighbors. Due to its ability to learn global spatiotemporal patterns, the model can use global information for inference even in scenarios where there are only a few neighbors present.

5.7 Interpretability (RQ4)

To confirm that the network's learned vector is the gradient of the spatio-temporal field, we calculate the curl variation with the number of training epochs. We use Yang et al.'s method [Yang et al., 2023] to quantify the curl and present the experimental results in Figure 5 (b). It is evident that the curl of the vector field obtained by the network decreases as the training progresses, indicating successful learning of the gradient field.

5.8 Generalizability (RQ5)

Our model not only excels in inferring PM2.5 but also establishes a new benchmark, achieving the SOTA in inferring the concentration of NO2, as shown in Table 2. This noteworthy outcome underscores the versatility of our model across distinct air quality parameters. In comparison to the baseline, our model demonstrates a significant advantage, showcasing its capability to handle diverse pollutants effectively and outperforming established methods in inference for NO2.

6 RelatedWorks

Traditional methods [Hasenfratz et al., 2014; Jumaah et al., 2019] rely on linear spatial assumptions. However, these models only consider simple spatial relationships and do not adapt to complex changes in air quality. In recent times, there's been a growing interest in studying SpatioTemporal Graph to understand the intricate relationship that involves both spatial and temporal for air quality inference. STGNNs [Jiang et al., 2021; Salim and Haque, 2015; Wang et al., 2020; Sun et al., 2020; Wang et al., 2021], which integrate the strengths of GNNs, have emerged as the leading approach for uncovering intricate relationships in STG data. Some follow-ups [Li et al., 2017; Yu et al., 2017; Geng et al., 2019] introduce temporal components such as Recurrent Neural Networks (RNN) [Graves, 2013] and Temporal Convolutional Networks (TCN) [Bai et al., 2018] to better address the spatio-temporal dependencies. However, The limitation of STGNNs lies in their lack of consideration for contiguity and Euclidean spatial structures.

7 Conclusion

In this work, we introduced a novel perspective for air quality inference, framing it as a problem of reconstructing SpatioTemporal Fields (STFs) to better capture the continuous and unified nature of air quality data. Our proposed SpatioTemporal Field Neural Network (STFNN) breaks away from the limitations of Spatio-Temporal Graph Neural Networks (STGNNs) by focusing on implicit representations of gradients, offering a more faithful representation of the dynamic evolution of air quality phenomena.

Acknowledgements

This work is supported by a grant from State Key Laboratory of Resources and Environmental Information System. This study is also funded by the Guangzhou-HKUST(GZ) Joint Funding Program (No. 2024A03J0620).