Multi-Modality Spatio-Temporal Forecasting via Self-Supervised Learning

1 Introduction

Multi-modality spatio-temporal (MoST) data is ubiquitous in real-world applications. Robust MoST prediction is crucial for various monitoring systems in smart city, particularly for sensitive scenarios that require subsequent decisionmaking. Compared to spatio-temporal (ST) data, MoST integrates information from an additional modality domain beyond the spatio-temporal domains. For example, urban public transportation management involves data from different time periods, locations, and transportation modalities, i.e., {Bike, Taxi}×{Inflow, Outflow} [Zhang et al., 2023; Deng et al., 2023a]. Although MoST data enables a more comprehensive and diverse feature acquisition, it is nontrivial to model them due to its significant heterogeneity and dynamics. Specifically, modeling MoST data presents two critical challenges. The first challenge lies in precisely obtaining variations and correlations of different modalities

∗Corresponding author.

两种模式在上午8点都表现出办公区域的增加，但出租车的流入量大大超过自行车的流入量； 晚上7点，居民区的出租车流量达到峰值，而食品区的出租车流量保持稳定。 across time and space. While using one modality to supervise others [Huang et al., 2019; Wu et al., 2020a] is common practice, this results in late-fusion models ignoring inter-modality correlations or, in early-fusion models, a trade-off between inter-modality and intra-modality features. The second challenge lies in accurately quantifying heterogeneous components to protect the model from erroneous pattern variations. Taking NYC traffic demand data as an example, in Figure 1, we depict observations of two modalities (i.e., Bike/Taxi Inflow) across different locations over a period and the corresponding distributions. Findings indicate that both modalities exhibit an increase in the Food area at 8 am, yet Taxi Inflow significantly surpasses Bike Inflow; by 7 pm, Taxi Inflow peaks in the Residential zone while remaining steady in the Food area. Such heterogeneities across space, time, and modality amplify uncertainties and errors in MoST modeling.

For more effective time series modeling, the existing methods have made significant strides by modeling latent spatial correlations among sensors and temporal auto-correlation within time series. Mainstream models utilize GCN-based [Wu et al., 2020b; Shao et al., 2022] modules for spatial relations and sequence models such as RNN [Lai et al., 2018], TCN [Wu et al., 2020b], and attention mechanisms [Wu et al., 2021; Zheng et al., 2020] for temporal relations. These techniques focus more on lower-level hierarchical information and overlook valuable insights about spatio-temporal relationships provided by multiple modalities. A new line of research increasingly extracts evolution from other modalities as auxiliary inputs, adding supplemental information to leverage similar historical patterns [Fang et al., 2021; Han et al., 2021]. However, these modality-aware practices predominantly emphasize benefiting spatio-temporal aspects, posing a challenge in preserving interactions within the high-dimensional latent space in MoST due to the parameter space shared among modalities. Recently, motivated by the success of self-supervised learning [Hendrycks et al., 2019], especially contrastive learning achieved in text, image, and audio domain [Giorgi et al., 2021], researchers apply SSL principles to model time series data [Deldari et al., 2022; Ji et al., 2023; Gao et al., 2023]. But these approaches primarily concentrate on modeling domain-specific heterogeneity, be it spatio-temporal or modality-specific, without fully encompassing multiple domains. MoST modeling requires comprehensive consideration of interactions and heterogeneity across space, time, and modality to obtain a more thorough and accurate representation. Thus far, despite systematic exploration of spatio-temporal regularities, modality interactions are adequately unaddressed, let alone signals entangled in MoST.

Different modalities of spatio-temporal data record the observations from different perspectives, promising to improve the forecasting performance by mining the latent patterns and dynamic heterogeneity deeply and sufficiently. Therefore, we are motivated to propose a novel MoST learning framework via Self-Supervised Learning, namely MoSSL. It comprises the following steps: (i) implementing a MoST Encoder to model the information from the space, time, and modality; (ii) introducing a Multi-modality Data Augmentation to understand pattern correlations governed by modality rules and integrate MoST domain information; (iii) devising a Global Self-Supervised Learning (GSSL) to discern diverse pattern changes among different perspectives; (iv) developing a Modality Self-Supervised Learning (MSSL) to further strengthen learning representations of inter-modality and intra-modality features in high-dimensional feature space. The contributions of our work are summarized as follows:

• We propose a novel MoST learning framework, called MoSSL, to explicitly uncover latent patterns and disentangle the heterogeneity from temporal, spatial, and modality perspectives.
• We design a Multi-modality Data Augmentation, coupled with two auxiliary self-supervised paradigms, i.e., GSSL and MSSL, which empower the model to discern and quantify dynamic heterogeneity in MoST.
• We conduct extensive experiments on two real-world MoST datasets (i.e., the traffic dataset with four transportation modalities and the air quality dataset with three pollutants). The results demonstrate that our model outperforms multiple state-of-the-art baselines.

2 Preliminaries

In this section, we formally describe the multi-modality spatio-temporal forecasting problem.

Problem Statement. The time series information of all spatial units (i.e., nodes) N and modalities M at tth time slot is denoted as Xt ∈ RN×M . Given T consecutive time steps, the multi-modality spatio-temporal forecasting problem aims to forecast the next O steps as ˆ Y ∈ RO×N×M , which can be expressed by the following conditional distribution:

3 Methodology

In this section, we will introduce the technical details of the proposed MoST learning framework, namely MoSSL, as demonstrated in Figure 2.

3.1 Multi-modality Spatio-Temporal Encoder

We design a Multi-modality Spatio-Temporal (MoST) Encoder that collectively preserves context in MoST data, allowing the modeling of sequential patterns across different time periods, geographical correlations between spatial nodes, and shared knowledge among modalities. For convenience, we define a non-linear projection function with ReLU activation function as f(x) = relu(x · w + b), where w and b are learnable parameters. Specifically, the MoST Encoder takes H(in) = f(X) ∈ RT×N×M×dz as input.

Modality Attention. Inspired by [Zheng et al., 2020], we design a Modality Attention (MA) based on the attention mechanism to encode the modality-wise correlations. For modality m at time step t and node n, MA works as follows:

where ⟨·⟩ denotes the inner product operator, um,m′ indicates the relevance between modality m and m′ , and αm,m′ is the attention score that indicates the importance of modality m′ to m. Thus, MA can adaptively capture inter-modality correlations and generate weighted modality representation H(ma) ∈ RT×N×M×dz .

Spatial Attention. We employ a Spatial Attention (SA), akin to MA, to model dynamic spatial relationships among nodes. For node n at time step t in modality m, the spatial correlation representation H(sa) ∈ RT×N×M×dz can be calculated by:

where sn,n′ indicates the relevance between node n and n′ , and βn,n′ is the attention score that indicates the importance of node n′ to n.

Temporal Convolution. We implement a Temporal Convolution (TC) with a tangent filter and a sigmoid gate [van den Oord et al., 2016] to extract high-level temporal features. Formally, given the representation ˆH = [H(in) ,H(ma) ,H(sa) ] ∈ RT×N×M×3dz , it works as follows:

where ∗ denotes the dilated causal convolution, Wfilter and Wgate are learnable weights, and ⊙ indicates element-wise multiplication. fconv(·) is a 1×1 convolution function.

We construct our MoST Encoder with a multi-layer structure, incorporating MA and SA in conjunction with TC. This architecture serves as the foundation for representing the spatio-temporal-modality relationships and yields the original representation H ∈ RT×N×M×dz as its output.

3.2 Multi-modality Data Augmentation

Here, we introduce two data augmentation strategies,i.e., Modality-aware Augmentation and MoST Embedding.

Modality-aware Augmentation. We devise a Modalityaware Augmentation operator to provide tailored modality dependencies for each instance. Different from the common method for spatial graph augmentation [Jiang et al., 2023b; Ji et al., 2023], our operator learns the heterogeneity ruled by modalities based on the original representation H:

where ϕt,n,m is the aggregation weight indicating the modality relevance, w0 ∈ Rdz is a learnable vector. With ϕt,n,m available, we can derive the mask probability from the Bernoulli distribution, i.e., pt,n,m ∼ Bern(1 - ϕt,n,m), and conduct the augmentation on the input tensor X. Higher pt,n,m signifies stronger modality heterogeneity (lower correlation), increasing the likelihood of masking the data xt,n,m. MoST Embedding. Since the evolving patterns of MoST are confined by space, time, and modality constraints, we develop an adaptive MoST Embedding to incorporate available domain information into the model. Specifically, we create spatial and modality embeddings to encode respective features into en ∈ Rdz and em ∈ Rdz , capturing static information. To enrich dynamic correlations, we generate temporal embedding to encode each historical time step into et ∈ Rdz . Then, we obtain the adaptive MoST embedding et,n,m = et + en + em, denoted as E ∈ RT×N×M×dz .

After obtaining two augmentations, we concatenate them as ˜X, and transform it by using a non-linear projection ˜H(in) = f( ˜X), which is fed into the shared MoST Encoder to generate the augmented representation ˜H ∈ RT×N×M×dz .

3.3 Global Self-Supervised Learning

Recently, cluster-based learning has disrupted the independence between samples by leveraging latent clustering information. By tapping into additional prior knowledge derived from clustering, the aim is to retain higher-level heterogeneous information within the learned representations. However, existing time series modeling methods rely on clustering partitions that might not necessarily indicate the optimal solution. Additionally, some approaches [Zhu et al., 2021; Duan et al., 2022; Ji et al., 2023] assume temporal constancy between representations and cluster memberships, which are not applicable for intricate and dynamic MoST data. We introduce a novel Global Self-Supervised Learning (GSSL) to address the above limitations, which could explicitly capture the heterogeneous patterns across spatial, temporal, and modality domains. Furthermore, we expect to preserve the dynamic attributes based on the underlying context, showcasing both independence and complementarity. For instance, some clusters correspond to modality properties, remaining consistent across space over time, while others correspond to temporal identities, possibly having different change frequencies over time.

Specifically, with an augmented representation ˜H derived from an augmented input ˜X, GSSL seeks to model a mixture of Gaussian distributions over ˜ht,n,m within ˜H. Technically, the probability distribution is conditioned on the augmented input, but we omit this condition in the subsequent formula for simplicity. Formally, given the augmented representation ˜H, we first infer the probability distribution of ˜ht,n,m ∈ Rdz , which can be decomposed as follows:

where K is the number of clusters (heterogeneous components). γk(˜ht,n,m) indicates the membership score associated with ˜ht,n,m belonging to the kth cluster, satisfying 0 ≤ γk(˜ht,n,m) ≤ 1 and ∑k γk(˜ht,n,m) = 1. Then, we assign γk(˜ht,n,m) via a linear mapping parameterized with learnable weight W γ k ∈ RTLSdz :

Without loss of generality, we assume P(˜ht,n,m|γk(˜ht,n,m)) follows a Gaussian distribution N(˜ht,n,m|µk, σ2 k ), and the mean µk ∈ Rdz and variance σ2 k ∈ Rdz are generated by:

where W µ k,d , W σ k,d ∈ RTLS , bµk,d , bσk,d ∈ R are learnable weights and biases, d ∈ [1, ..., dz] is the hidden channel index. exp(·) ensures that σ2 k remains greater than zero.

After obtaining the distribution of the augmented representation, i.e., N(˜ht,n,m|µk, σ2 k ), we treat it as a selfsupervised signal. It aims to align the distribution of the original representation ht,n,m with that of ˜ht,n,m, performed by N (ht,n,m|µk , σ2 k ). This practice leverages data augmentation to enrich the original representation h, enhancing its understanding of heterogeneous components. From the probability perspective, we replace P(˜ht,n,m|γk(˜ht,n,m)) (in Eq. 6) with P(ht,n,m|γk(˜ht,n,m)), and apply the Negative LogLikelihood (NLL) to optimize the self-supervised task:

By aligning the distributions of H and ˜H, the original representation H can reflect the global heterogeneity across time, space, and modality, aiding the model forecasting.

3.4 Modality Self-Supervised Learning

Recent self-supervised contrastive learning methods often compare representations from two distorted versions of the same input instance (i.e., original and augmented views)

[Grill et al., 2020] or across historical time intervals [Ji et al., 2023]. Differing from these methods, we devised a Modality Self-Supervised Learning (MSSL) paradigm to further enhance the model's understanding of cross-modality heterogeneity, mitigating the loss of information due to modality ignorance or time sensitivity. This brings two advantages: (i) aiding the model in better utilizing modality information and finer learning of spatio-temporal features across different modalities; (ii) reducing interference among different modalities, enabling a more precise capture of the spatio-temporal dependencies within each modality.

Specifically, given the representations H and ˜H from the original and augmented views, we first fuse them with learnable weights w1, w2 ∈ Rdz , and provide the unified modality representation cM through aggregating spatio-temporal features as follows:

Then, the representation ¯ht,n,m = (rt,n,m, cM) is used for positive and negative pairs sampling. Within our sampling strategy, the modality-aligned instances (intra-modality) are considered positive pairs, and the instances across all available modalities (inter-modality) are considered negative pairs. Here, the idea is to obtain the modality-specific patterns and encourage the discovery of heterogeneity across different modalities. Following the convention, we derive the objective by Binary Cross Entropy (BCE) with a sigmoid function for numerical stability:

where G(¯ht,n,m′) = σ(r⊤ t,n,m′w3cM) is the transforming function, m and m′ indicate two different modalities, and w3 ∈ Rdz×dz is the learnable parameter shared by positive and negative pairs.

3.5 Model Optimization

For the MoST forecasting task, our predictor comprises two fully connected layers with ReLU as the activation function, adapting the dimension of the hidden channel to the desired output dimension. It takes the representation H from the original view as input:

where Wo1 ∈ Rdk×dk , Wo2 ∈ Rdk×O , bo1 ∈ Rdk , and bo2 ∈ RO are learnable weights and biases. relu(·) denotes ReLU function that is applied element-wise. ˆ Y ∈ RO×N×M indicates the forecasting result of the next O steps for N nodes and M modalities.

In the learning phase, we incorporate two self-supervised learning losses (i.e., Eq. 9 and Eq. 11) into the objective criterion (i.e., MSE) as the overall optimization:

4 Experiments

In this section, to evaluate the performance of our MoSSL, we develop a group of research questions (RQ) and conduct a series of experiments correspondingly:

RQ1: How does MoSSL perform compared with the stateof-the-art baselines? (Section 4.2)

RQ2: How does MoSSL perform compared with its model variants? (Section 4.3)

RQ3: How is the interpretability of MoSSL? (Section 4.4) RQ4: What are the effects of hyper-parameters in MoSSL? (Section 4.5)

4.1 Experiment Setup

Datasets. We conduct experiments on two real-world MoST datasets, namely NYC Traffic Demand and BJ Air Quality, as listed in Table 1. NYC Traffic Demand dataset is collected from the New York City, which consists of 98 nodes and four transportation modalities: Bike Inflow, Bike Outflow, Taxi Inflow, and Taxi Outflow. BJ Air Quality dataset is collected from the Beijing Municipal Environmental Monitoring Center, which contains 10 nodes and three pollutant modalities: PM2.5, PM10, and SO2.

Settings. We implement the network with the Pytorch toolkit. For the model, the layers of MoST Encoder is 4, where the kernel size of each dilated causal convolution component is 2, and the related expansion rate is {2, 4, 8, 16} in each layer. This enables our model to handle the 16 input steps. The number of cluster components K and the dimension of hidden channels dz are set to 4 and 48. The training phase is performed using the Adam optimizer, and the batch size is 16. In addition, the inputs are normalized by Z-Score.

Baselines. To evaluate our model's prediction accuracy quantitatively, We implement eight baselines that fall into two categories for comparison. Non-MoST forecasting models: 1) LSTNet [Lai et al., 2018] is a classical time series prediction model; 2) MTGNN [Wu et al., 2020b], 3) Graph Wavenet (GWN) [Wu et al., 2019], and 4) AGCRN [Bai et al., 2020], the most representative deep models for spatio-temporal forecasting, embed adaptive graph leaning into temporal convolution; 5) ST-Norm [Deng et al., 2021] is a normalization-based approach that refines the temporal and spatial components. MoST forecasting models: 6) STtrans [Wu et al., 2020a], a transformer-based model that integrates spatial, temporal and semantic dependencies; 7) MiST [Huang et al., 2019]

is a classic co-predictive model that incorporates the spatiotemporal and categorical embeddings; 8) COCOA [Deldari et al., 2022] is a modality-matching contrastive framework that minimizes the similarity between irrelevant modalities.

4.2 Overall Performance

We compare our MoSSL with the mentioned baselines, and report the average results of five repeated experiments. Through Table 2, we can find our MoSSL outperforms stateof-the-arts in all cases (modality/horizon/metric) on the NYC Traffic Demand dataset. Among the non-MoST forecasting baselines, MTGNN and GWN perform relatively well based on the WaveNet [van den Oord et al., 2016] backbone. Models like AGCRN and ST-Norm, designed for spatio-temporal dependency modeling, exhibit a worse performance due to modality-neglected dependencies and heterogeneity, making them unsuitable for MoST forecasting tasks. COCOA shows significant performance variation among MoST forecasting baselines due to the lack of spatial modeling. While MiST maintains relatively stable performance, it falls short in fully capturing the dependencies and heterogeneity within MoST as it requires referencing modalities as auxiliary inputs to predict the target modality. Table 3 demonstrates MoSSL's superiority in almost all scenarios. The BJ Air Quality dataset only has 10 locations and less corrections, and many weatherrelated features are unavailable. While our MoSSL struggles with this data insufficiency, it maximizes available feature correlations. Despite the limitations, the baseline models still fall short compared to the performance of MoSSL. Regarding the efficiency as shown in Figure 3, our MoSSL is slightly time-consuming (compared with LSTNet, ST-Norm, and COCOA) but has the smallest RMSE, indicating MoSSL strikes a balance between total training time and performance.

4.3 Ablation Study

To analyze the effects of key-modules in our MoSSL framework, we perform ablation studies with several variants on the NYC Traffic Demand dataset as follows: (1) w/o AV. It removes the augmented view in the joint framework, only keeping the MSSL paradigm and using the original representation H as input, with no change in sampling strategy; (2) w/o MG. It removes the Multi-Modality Data Augmentation and GSSL, using a MoST encoder that doesn't share parameters with the original view and retaining the MSSL paradigm; (3) w/o GSSL. It removes the GSSL from the framework; (4) w/o MSSL. It removes the MSSL from the framework. The results are shown in Figure 4. Eliminating data augmentation, GSSL, or MSSL that relied solely on original data/representation had varying negative impacts on model performance. Overall, the complete MoSSL consistently outperforms its variations across all modalities, underscoring its holistic and indivisible nature.

4.4 Case Study

Modality Augmentation. We verify the Modality-aware Augmentation on the NYC Traffic Demand dataset in Figure 5, where Figure 5a depict heatmaps of the augmented data (red indicates masked instances), and Figure 5b show the corresponding variations in node 1 and node 3 of the original data. Modalities exhibit distinctly different evolutions in the purple area (masked instances), while in the yellow area (retained instances), Bike Inflow and Bike Outflow display similar pattern changes. In this way, our MoSSL can adaptively eliminate biases in less correlated modality connections while preserving dependencies between modalities.

Heterogeneity Disentanglement. We qualitatively evaluate the quality of the global heterogeneity representation learned from GSSL by visualizing them in a low-dimensional space with t-SNE, as shown in Figure 6a. We can observe that instances in the same cluster are compact, while instances in different clusters are clearly separated. Figure 6b illustrates the population distribution of representation ˜H and its decomposed sub-distributions. The former, with multiple peaks, directly modeling it might cause information loss or confusion. The latter precisely quantifies heterogeneous components, enhancing the model's adaptability to MoST. We further plot the cluster variations in Figure 6c. Cluster 3 maintains relatively stable behavior over time due to its spatial dominance. Clusters 0 and 1 show intertwined evolution curves because the visualized instances stem from the same modality (Bike Inflow), and overlapping features correspond to several identical nodes. Clearly, MoSSL recognizes heterogeneities in MoST data, transferring information between similar clusters and thus aiding predictions.

We further investigate the explanations provided by our MoSSL in identifying crucial dynamic heterogeneity and analyzing their patterns. Figure 7 presents node correlation heatmaps, node relations, and corresponding evolutions for selected nodes at different modalities and times, computed using cosine similarity. In the node relation diagram, the thicker edge represents a higher correlation, and the bigger node size means a larger weighted outdegree. For Bike Inflow, as shown in Figure 7a, MoSSL identifies that at 8 am, the correlation between node 1 and node 4 is stronger than with node 6, while at 12 am, the correlation weakens with node 4 and strengthens with node 6. This trend is evident in

(a) t-sne可视化：

h -> [b, tns, c] -> [btns, c] -> t-sne -> [btns, 2]

Lables -> [b, tns] -> [btns] -> colors

(b) 可视化每个cluster的演变曲线

the time series diagram, where node 1 has comparable valleys with node 4 at 8 am, and a similar changing trend with node 6 at 12 am. In Taxi Inflow, as depicted in Figure 7b, node 1 is more correlated with node 4 at 8 am, similar to its performance in Bike Inflow. However, at 12 am, node 1 maintains a strong correlation with node 4 and a weak correlation with node 6, contrasting with its behavior in Bike Inflow. These findings indicate that MoSSL successfully decouples heterogeneity while preserving similar patterns, avoiding blind borrowing of information from other modality/spatial/temporal trends, which facilitates predictions.

4.5 Hyper-parameter Study

We examine the impact of key hyper-parameters in Figure 8, i.e., the number of cluster components K and the hidden channel dimension dz. We observe a trend where the performance of MoSSL decreased initially and then increased with the increment of K and dz. This phenomenon aligns with our understanding that improper settings of cluster components and channel dimensions can lead to underfitting or overfitting of the model. The model achieved comparable performance when K and dz are set to 4 and 48, respectively.

5 RelatedWork

Many studies have been devoted to developing deep learningbased time series forecasting. As the canonical cases of spatio-temporal forecasting, graph neural network is developed to model the spatial relation [Li et al., 2017; Oreshkin et al., 2021]; attention mechanisms [Vaswani et al., 2017] and its variants include spatial attention [Fang et al., 2019; Zheng et al., 2020], temporal attention [Wu et al., 2021; Zheng et al., 2020], and adaptive embedding [Liu et al., 2023]; convolution operators are utilized in spatial convolution [Deng et al., 2022; Guo et al., 2021], temporal convolution [Wu et al., 2020b; Wu et al., 2019], spatio-temporal convolution [Guo et al., 2019; Yang et al., 2021] and adaptive convolution [Pan et al., 2019]. [Jiang et al., 2021] establishes a benchmark for prevalent deep learning models. [Zhu et al., 2021] is proposed for micro-cluster analysis and prediction in macro time series. [Fan et al., 2023] models the inter- and intra-space distribution drift and [Fan et al., 2022] learns the periodicity of the time series. Traditional spatio-temporal modeling approaches might not entirely fit MoST because of the dynamic and heterogeneity arising from the additional modality domain. Some recent methods extend spatio-temporal modeling by considering additional domains (i.e., modality or categorical domain) [Ye et al., 2019; Huang et al., 2019; Han et al., 2021; Wang et al., 2021] or data sources [Deng et al., 2023a; Jiang et al., 2023a; Deng et al., 2023b], extracting insights from different modalities as auxiliary inputs to leverage similar historical patterns. However, they struggle to fully explore heterogeneity or interactions from different modalities. [Ji et al., 2023; Deldari et al., 2022] model time series with self-supervised learning but lack modality sensitivity (shared information among modalities) and spatial sensitivity (ignoring spatial characteristics). Our approach stands out by integrating multi-modality data augmentation and two self-supervised learning paradigms, explicitly capturing similar patterns while quantifying heterogeneity to better adapt to MoST data.

6 Conclusion

In this study, we propose a self-supervised learning framework MoSSL for multi-modality spatio-temporal forecasting. Specifically, we integrate Multi-Modality Data Augmentation and two self-supervised learning paradigms, i.e., GSSL for comprehending diverse pattern changes in space, time, and modality, and MSSL for strengthening the learned representations of inter- and intra-modality features. Extensive experiments on two real-world MoST datasets verify the superiority of MoSSL. In future work, we will refine the model to be more lightweight and universally applicable.

Acknowledgments

This work was partially supported by the grants of National Key Research and Development Project (2021YFB1714400) of China and Jilin Provincial International Cooperation Key Laboratory for Super Smart City.