Long-term Wind Power Forecasting with Hierarchical Spatial-Temporal Transformer

1 Introduction

The UN Sustainable Development Goals 7 [Vinuesa et al., 2020] (SDG 7) aims to ensure access to affordable, reliable, sustainable, and modern energy for all. Wind Power Forecasting (WPF), which focuses on accurately predicting the wind power generation of turbines in a wind farm for future time intervals, can contribute to the realization of SGD 7 by building more efficient low-carbon systems. It was reported that in 2021, the proportion of global wind and solar power generation had reached one-tenth, and the total energy output exceeds 2837 TWh [Jones et al., 2022]. However, due to the chaotic nature of the earth's atmosphere, wind power generation is always associated with non-stationary uncertainties. Therefore, how to integrate wind energy into the power grid with high stability and security is of great significance.

∗Corresponding author: Lingbo Liu (lingbo.liu@polyu.edu.hk).

Fortunately, these uncertainties in power systems operations can be mitigated to a certain degree via accurate WPF methods, which are becoming the most promising solutions for integrating a large amount of wind energy into power grids [Tastu et al., 2013]. Wind Power Forecasting has been extensively investigated over the past decades [Deng et al., 2020; Wang et al., 2021], and the existing research can be coarsely divided into four categories: physics-based methods [Chen et al., 2013; Shao et al., 2016], statistical methods [Zeng and Qiao, 2011; Hu et al., 2015], hybrid intelligent methods [Ghoushchi et al., 2021; De Caro et al., 2021], and deep learning-based methods [Ahmad and Zhang, 2022; Zhu et al., 2019]. However, most of the existing works still suffer from several limitations, which restricts their applications in real world:

• Physics-based and statistical methods usually take too much calculation costs and are sensitive to the errors introduced by the initial condition [Deng et al., 2020]. They can not perform well when dealing with nonlinear and non-stationary traits in wind power due to their shallow learning models [Wang et al., 2017].
• Most of them are designed for short-term predictions and can not achieve satisfactory results under longterm wind power forecasting [Shao et al., 2016; He and Wang, 2021], while accurate long-term predictions are even more critical in system dispatch planning and ramp events (large wind power fluctuation) prediction [Ouyang et al., 2019].
• Existing methods lack an effective design for spatiotemporal modeling. They either consider WPF as a simple times series forecasting problem ignoring spatial information or extract spatial dependencies in a local and static manner [Yu et al., 2020; Zhu et al., 2019]. Actually, indispensable information may be revealed by modeling spatial correlations because wind characteristics at a site resemble those nearby or share the same meteoro-

logical conditions [Ding, 2019]. It is nontrivial to capture global and comprehensive spatial correlations.

The long-term WPF aims to understand the correlation between data in each different time step. However, 1) a single point does not have semantic meaning like a word in a sentence and may have limited influence on predicting the future [Nie et al., 2022]. What is more, 2) the inherent high intermittent of wind power and the record noises caused by some external reasons bring in plenty of uncertainties, which makes the long-term prediction worse. 3) the fine-grained long-term prediction leads to high computational and space complexities when applying the point-wise selfattention mechanism. In contrast, sparse and localized contextual information is essential in analyzing their semantic connections [Du et al., 2023], e.g., the airflow in a period, like midnight, may show intense fluctuations but blows much heavier than that at noon. Thereby, the aggregated and coarsegrained temporal representations are nontrivial and complementary to the fine-grained temporal features for precise forecasting.

Inspired by the ideas and problems mentioned above, we propose a novel end-to-end deep learning-based framework termed Hierarchical Spatial-Temporal Transformer Network (HSTTN), which well addressed the long-term predictions with the hourglass-shaped network and effectively modeled the spatiotemporal contextual information and correlations. In particular, the HSTTN consists of four main modules: hourglass-shaped encoder-decoder architecture, residual spatiotemporal encoder/decoder layers, and Contextual Fusion Blocks (CFBs).

Different from the standard transformer architecture, in encoder, temporal pooling operations are inserted between several cascaded residual spatiotemporal encoder layers to generate hierarchical temporal scale features from fine-grain to coarse-grain. Symmetrically, in decoder, we gradually recover the fine-grained predictions from coarse-grained representations with upsampling operations inserted between the residual spatiotemporal decoder layers. Aggregating time steps to coarse-grained scale not only provides comprehensive semantic representations that are complementary to finer scales, but also reduces the amount of calculation since the sequence length is smaller. Besides, via inter-scale skipconnections, the outputs of each residual spatiotemporal encoder layer are directly concatenated to the outputs of each residual spatiotemporal decoder layer with the same temporal scales, which aggregates the rich fine-grained information from hierarchical encoder layers to facilitate the decoder to make more precise predictions. It is noteworthy that vanilla Transformer [Vaswani et al., 2017] is designed for the machine translation task which follows a seq2seq paradigm, while wind power records are spatiotemporal structured. So we first decouple the topological data into temporal-wise feature vectors and spatial-wise feature vectors, then feed them into the residual spatiotemporal encoder layers in parallel, which consists of temporal and spatial Transformer skeletons and contextual fusion blocks, to capture the hierarchical longrange temporal dependencies and spatial global correlations with the multi-head self-attention mechanism. The CFBs are designed for better spatiotemporal feature fusion and are inserted between each temporal encoder layer and spatial encoder layer. In specific, the latent representations from both layers are firstly rearranged into original spatiotemporalshape and concatenated along feature dimension. Then a Convolutional Neural Network is set to fuse and learn their contextual feature representations. The enhanced features which carry more comprehensive information are then fed into the next residual encoder layer for sparser scale but higher-level representation learning. The residual spatiotemporal decoder layers follow the same structure while the inputs are only future time spots and turbines locations without knowing the meteorological data and turbine internal status. The main contributions of this work are as follows:

• We propose a Transformer-based framework to predict wind power generations, which well addresses the longterm forecasting problem due to its impactful capacity to capture long-range and global dependencies by selfattention mechanism. To the best of our knowledge, this is the first attempt to apply Transformer architecture to long-term wind power forecasting tasks.
• A well-designed hourglass-shaped hierarchical architecture with lower time and space complexity is introduced to improve the long-term predictions by aggregating complementary multiscale temporal representations.
• Extensive experiments are conducted on two real-world wind power datasets with diverse dynamic context factors (meteorological data and turbine internal status), which demonstrate the effectiveness of the proposed framework for wind power forecasting.

2 RelatedWork

Wind Power Forecasting: Existing forecasting models can be grouped into four types based on differences in modeling theory: physics-based methods [Lobo and Sanchez, 2012; Shao et al., 2016], statistical methods [Zeng and Qiao, 2011; Hu et al., 2015], hybrid intelligent methods [He and Wang, 2021; Shahid et al., 2021] and deep learning based methods [Deng et al., 2020; Wang et al., 2021; Wang et al., 2017]. In physical models, numerical weather predictions (NWP) or weather researcher forecasting (WRF) are usually performed to predict weather conditions and then the weather condition predictions are fed into physics-based models to generate wind power forecasting. A non-negligible drawback of these two-stage prediction frameworks is that the errors in the first stage will accumulate and magnify the final prediction errors. To avoid the limitations of a single model, hybrid intelligent method also attracted much attention, which is a weighted sum of several models or the combination of compensatory models. [Shahid et al., 2021] developed a hybrid framework comprising of long short term memory (LSTM) and genetic algorithm (GA). The global optimization strategy of GA was exploited to optimize hyperparameters in LSTM layers. The deep learning based methods have drawn increasing attention in recent years due to its capacity of modeling intricate and non-linear relations. For instance, [Yu et al., 2020] proposed a hybrid neural network to capture spatial-temporal characteristics, in which the spatial features were extracted by a 2DCNN and the temporal features were extracted by an LSTM.

CNN is efficient in local feature extraction whereas in this work we prefer global spatial correlations as illustrated in the above section.

Transformers: Transformer is an advanced attention-based neural network block, which is originally proposed to tackle the machine translation task then widely used in natural language processing due to its superior performance in capturing long-range dependency by global self-attention mechanism [Vaswani et al., 2017; Devlin et al., 2019]. Recently, researchers have applied transformers to more artificial intelligence tasks such as visual understanding [Li et al., 2021; Wu et al., 2022; Liu et al., 2022b; Li et al., 2022; Zhang et al., 2023; Jiang et al., 2023a], time series analysis [Zhou et al., 2021; Wu et al., 2021; Jiang et al., 2023b; Luo et al., 2023] and spatial-temporal modeling [Xu et al., 2020; Liu et al., 2021; Liu et al., 2022a; Geng et al., 2022]. For long-term forecasting, [Zhou et al., 2021] proposed a transformer-based model named Informer to predict long sequences and designed the ProbSparse self-attention mechanism and distilling operation which drastically improved the inference speed of long-sequence predictions. Others explore to divide time steps into segments and capture segment-wise correlations [Nie et al., 2022; Du et al., 2023]. Inspired by these amazing works, we propose a Transformer-based spatial-temporal learning framework to hierarchically capture the contextual information interaction between complementary spatial and temporal features. To the best of our knowledge, this work is the first attempt to apply Transformer architecture to long-term wind power forecasting.

3 Problem Specification

In this section, we provide basic notations and the definition of wind power forecasting. The objective of WPF is to accurately estimate the wind power supply of a wind farm at different time steps by characterising the intricate relations between historical records and future wind power generations. In practice, wind is deflected by the blades of a wind turbine, and then generates electricity through rotations and generators, indicating the wind power is not only related to wind speed, but also to other meteorological data like wind direction, external temperature and essential turbine internal status. Given a wind farm consisting of N turbines, each of them generates wind power time series and corresponding dynamic context factors (meteorological data, turbine internal status, etc.). The dynamic context records of all turbines in a time window with T timestamps is formulated as X = {X1,X2, . . . ,Xn, . . . ,XN} ∈ RN×T×C ,where C is the number of feature channels including the target variable Patv. For the n-th turbine, we denote Xn = {X1n ,X2n , . . . ,X T n } ∈ RT×C as the context records of all timestamps for the n-th turbine. Symmetrically, the context records of the whole farm at timestamp t is denoted as Xt = {Xt1 ,Xt2 , . . . ,X tN } ∈ RN×C .

Definition 1. Wind Power Forecasting. Assuming that the current timestamp is t, the wind power forecasting problem is to predict all turbines' power generations of future F timestamps utilizing the historical dynamic context factors of previous H timestamps, which is also a spatio-temporal data prediction problem. Mathematically, the predictions ˆ Yt+F = {Y 1 1 , Y 2 1 , . . . , Y 1 2 , Y 2 2 , . . . , Y F N } ∈ RN×F×1 , where 1 represents the target variable Patv, is obtained:

where f(·) represents the model for wind power forecasting, Xt-H denotes the historical dynamic context records and Φ represents the parameters in our model.

4 Method

The overall architecture of the proposed framework HSTTN is shown in Figure 1, which is composed of the hourglassshaped encoder-decoder architecture, the residual spatiotemporal encoder/decoder layer, the Contextual Fusion Block and a wind power regression module. The hierarchical residual spatiotemporal encoder/decoder layers (RSTEL/RSTDL) with pooling and up-convolution operations capture multiscale temporal dependencies and global spatial correlations from the embedded temporal-wise features and spatial-wise features respectively. Skip-connections between encoder and decoder will help enhance finer predictions by recovering localized coarse temporal information. Meanwhile, the CFBs inserted in residual spatiotemporal layers take the outputs of each temporal sublayer and spatial sublayer to capture the contextual information interaction between spatial and temporal features and propagate the enhanced representations carrying both spatial and temporal information. The encoderdecoder modeling paradigm generates output sequence one element at a time.

4.1 Hourglass-shaped Encoder-decoder

This is the main body of our framework. To tackle the longterm time series forecasting problem, global self-attention mechanisms are always preferred for modeling dependencies without regard to their distance in the sequences. Different from previous local modeling works [Yu et al., 2020; Zhu et al., 2019], we try to capture the global spatial correlations among different locations, as wind characteristics are similar in a local area or distant areas with similar climatic conditions. Besides, turbines that are not close to each other but sharing the same working status will also perform in an analogous way. So we employ the transformer architecture to model both long-range temporal dependencies and spatial correlations. Noted that the inputs of the first decoder layer Xd t+H are different from encoder inputs Xe t-H , which contain only future time spots and turbine locations without knowing the meteorological data and turbine internal status. The unknown factors in decoder's inputs are padded with zero.

As introduced in Section 3, the raw wind power context records are spatiotemporal structured data. As is depicted in Figure 2, a 1×1 convolutional block is firstly adopted to learn a high-dimension latent feature embedding from raw inputs. Without padding and stride, this step will generate a 2-D feature map F Conv ∈ RN×T×dmodel , where dmodel is the number of convolution kernels, i.e., the embedded features dimension:

Skip-connection

Figure 1: The architecture of our proposed Hierarchical Spatial-Temporal Transformer Network (HSTTN). RSTEL/RSTDL represent the

residual spatiotemoral encoder/decoder layers, respectively. ×L means each encoder/decoder layer could repeat multiple times for deeper

semantics. Key and V al indicate the outputs of the encoder layers are mapped to latent spaces and transferred to their inter-scale decoder

layers for attention calculations.

temporal-wise feature vectors

× 𝑁

× 𝑇

spatial-wise feature vectors

Duplicate & Decouple

where F Convi denotes the i-th channel in the final feature map F Conv , ReLU(·) is the activation function, ⋆ represents the convolution operation, W i ∈ R1×1×C and bi ∈ RC are the learned weights of the i-th kernel and bias, respectively. Then F Conv is duplicated and decoupled into N temporalwise feature vectors F tem ∈ RT×dmodel and T spatial-wise feature vectors F spa ∈ RN×dmodel as the inputs of residual encoder layers.

Significantly, different from the general transformer architecture, we propose to successively downsample the longterm fine-grained temporal scale to coarse-grained scale in the encoder and recovering it with upsamplings in the decoder, which forms a hierarchical structure with different temporal scales. The cascaded residual spatiotemporal encoder/decoder layers which can self-repeat multiple times form the hourglass-shaped network and the details of RSTEL is depicted in Figure 3. The input features F tem and F spa are firstly fed to the corresponding temporal encoder layer and spatial encoder layer, which apply the standard Multi-head Self-Attention (MSA) mechanism [Vaswani et al., 2017]. Taking temporal encoder layer for example, the

i-th temporal encoder layer's input F tem i is first fed into three linear layers to generate query, key and value embedding: Qtem i = F tem i W q i , Ktem i = F tem i W k i , V tem i = F tem i W v i , where W q i ∈ Rdmodel×dk , W k i ∈ Rdmodel×dk , and W v i ∈

𝐹𝑡𝑒𝑚

𝐹𝑠𝑝𝑎

Rdmodel×dv are the projection parameter matrices. we can obtain the latent representation:

where Attntem ∈ RT×dmodel . To be exact, for each embedded F Conv , there are N encoded vectors Attntem. Then, both temporal and spatial features are delivered into the contextual fusion block, which generates the enhanced representation. At last the origin inputs are directly added to the fusion output in a residual manner, which can help to reduce overfitting and gradient vanishing. Therefore the output of the i-th residual spatiotemporal encoder layer can be written as follows:

where FUSE(·) represents the contextual fusion in Section 4.2. After the maxpooling, we have the inputs for the next RSTEL F tem i+1 ∈ R Tp ×dmodel , where p is the pooling factor. RSTDL follows a similar structure except for an additional multi-head attention over the outputs of its corresponding RSTEL. The inter-scale skip connections between encoder and decoder layers further facilitate the model to make precise predictions. Specifically, the outputs of each RSTEL are directly concatenated to the outputs of their corresponding RSTDL, followed by a up convolution to recover the finegrained details.

4.2 Contextual Fusion Block

The temporal dependencies and spatial correlations are characterized in their own branches thus lack of contextual information interactions. Here we utilize a simple, computationally efficient but effective module named Contextual Fusion Block (CFB) to capture the contextual joint features and propagate them among temporal and spatial representations. Figure 4 shows the structure of CFB.

We take the feature fusion in encoder for example since the decoder shares the same mechanism. As described in Section 4.1, the original input features are decoupled into multiple sequences of vectors. After aggregating information from the whole sequence, the outputs of each temporal encoder layer are Attntemn ∈ RT×dmodel , n =

1, 2, . . . ,N and the outputs of the corresponding spatial encoder layer are Attnspat ∈ RN×dmodel , t = 1, 2, . . . , T. Both of them are firstly stacked to the original 2-D shape feature maps and then concatenated along the channels: Ospa = Concat(Attnspa1 , . . . ,AttnspaT ), Otem = Concat(Attntem1 , . . . ,AttntemN ), Osp = Concat(OspaT ,Otem), where Concat(·) denotes a concatenation operation, Ospa ∈ RT×N×dmodel and Otem ∈ RN×T×dmodel are the stacked spatial encoder layer output and temporal encoder layer output respectively and Osp ∈ RN×T×dmodel·2 is the learned representation including both spatial and temporal characteristics. Then, in order to remove redundant information and reduce feature dimensions, a 1×1 convolution is employed to capture the contextual spatiotemporal correlations and generate the enhanced representation Ofuse ∈ RN×T×dmodel :

where Ofusei is the i-th channel in Ofuse . Finally, the enhanced informative features carrying both spatial and temporal information are duplicated and decoupled into multiple temporal-wise and spatial-wise sequence vectors as described in Section 4.1, which are then fed to their corresponding next encoder layers for higher-level representation learning.

4.3 Wind Power RegressionModule

The outputs of the original scale residual spatiotemporal encoder and decoder layers are concatenated to make the final predictions. The concatenated outputs Oorign ∈ RN×F×dmodel·2 , are fed into a fully-connected layer to predict wind power generations of the next F timestamps ˆ Yt+F ∈ RN×F×1 :

where Drop is the dropout operation, W Y ∈ R2·dmodel×1 are the learnable parameters and bY is the bias.

To evaluate the difference between our prediction and the ground truth, we utilize mean square error (MSE) as our loss function, which is defined below:

where m is the number of samples.

5 Experiments

5.1 Experimental Setup

Datasets: We conduct experiments on two challenging realworld datasets: i) SDWPF [Zhou et al., 2022] is obtained from the real-world data from Longyuan Power Group Corp. Ltd. This dataset contains 4,727,520 records sampled every 10 minutes and are collected from a wind farm with 134 wind turbines in 245 days. Each record contains 13 attributes including critical external features (such as wind speed, wind direction and temperature) and essential internal features (such as inside temperature, nacelle direction and so on). ii) Engie 1 is obtained from ENGIE group. The wind power data consists of 1,057,968 records from 1 January 2013 to 12 January 2018, obtained by sampling every 10 minutes from a wind farm containing 4 wind turbines. Each record contains 34 attributes.

Implementation Details: In our experiment, we utilize the historical records of 144 time slots to forecast the wind power generations in the next 144 time slots. For SDWPF, we sequentially split the dataset into 155 days, 30 days and 60 days for training, validation, and testing, respectively. For Engie, the dataset is split into 1296 days, 180 days and 360 days for training, validation and testing, respectively. Finally, the whole dataset is normalized with Z-score Normalization and inverted to the original scale when performing evaluation. Our proposed model is implemented with the PyTorch framework. The feature dimension dmodel for multi-head attention is set to 16 and the number of head is set to 2 for both datasets. The number of convolution kernels in CFB is also 16. For this 144 time slot prediction task, we downsampling the original scale with 3 and 2 times successively in encoder, while upsampling it to the original scale in decoder. The initial learning rate is 1e-4 and 1e-3 and decreases gradually.

1 https://opendata-renewables.engie.com

Adam optimizer is adopted to minimize the MSE loss until the training process is ended by early stopping strategy. We choose the best model on validation set as the final model and evaluate its performance on testing set for fair comparisons. Evaluation Metrics: We use two widely used metrics including mean average error (MAE) and root mean square error (RMSE) to evaluate all methods' performance on the whole wind farm. They are defined as:

where N is the number of turbines and m is the number of samples of each turbine. Besides, SDWPF introduced several invalid conditions of the records caused by its recording system. These invalid values will not be used in evaluation.

5.2 Baselines

In this paper, we compare the proposed HSTTN against seven deep learning methods, including two RNN-based models LSTM [Hochreiter and Schmidhuber, 1997], GRU [Cho et al., 2014], two Transformer-based models: Informer [Zhou et al., 2021], TSAT [Ng et al., 2022], and three spatial-temporal forecasting models: Baidu* 2 , DCRNN [Li et al., 2018], BiSTAT [Chen et al., 2022]. All methods are implemented on both preprocessed datasets under the same experimental setup for a fair comparison.

Table 1 summarizes the performance of all comparison methods on SDWPF and Engie respectively, and our proposed HSTTN performs the best on both datasets. LSTM, GRU, Informer and TSAT are time series forecasting models which lack spatial feature modeling. So we decouple the spatiotemporal input data to multiple temporal sequences in the manner mentioned in Section 4.1 and feed them into these models. As a result, our HSTTN achieves the lowest MAE and RMSE on both dataset. We can observe that when handling WPF as multivariate time series forecasting, Transformer-based models (Informer, TSAT, Baidu*, Bi-STAT and HSTTN) outperforms those RNN-based models LSTM, GRU and which imply the effectiveness of selfattention for capturing long-range temporal dependencies.

2 https://github.com/PaddlePaddle/PGL/tree/main/examples/ kddcup2022/wpf baseline

Informer and TSAT also utilize self-attention to learn temporal representations but lack spatial information modeling, which limits their performance. Compared with simple RNN, DCRNN improves the performance to some degree with the help of learning spatial representations explicitly, which demonstrates the importance of spatial features for multiturbine wind power forecasting. By modeling both spatialtemporal context, DCRNN and Bi-STAT outperforms common recurrent neural networks (LSTM, GRU) and temporal transformer models (Informer, TSAT) and reaches comparable results with our HSTTN on Engie dataset, but the performance is not satisfactory on SDWPF dataset. The reason is that DCRNN and Bi-STAT capture spatial context based on Euclidean connectivity and distance, while the turbine distributions on SDWPF are much more complex than that of Engie and the spatial context of wind power is also related to meteorological conditions and turbine status. Compared to the above methods, we not only capture the global and comprehensive spatial-temporal correlations by self-attention mechanism but also carefully designed the hourglass-shaped network architecture for long-term prediction, thereby achieving the state-of-the-art performance for both dataset.

5.3 Ablation Studies

In this subsection, we perform extensive analyses to verify the effectiveness of each component of the proposed HSTTN.

Effectiveness of Hierarchical Temporal Learning

• Spatial-Temporal Transformer Network (STTN): In this variant, we remove the downsampling, upsampling and skip-connection operations to explore the performance of the original temporal scale only.
• Two scale Spatial-Temporal Transformer Network (2STTN): This variant implement the downsampling and upsampling once each with the factor of 3, which leading to 2 temporal scale learning, to explore the effectiveness of coarse-grained semantic representations.
• Four scale Spatial-Temporal Transformer Network (4STTN): Similarly, we implement the downsampling the upsampling 3 times with the factors of 3, 2 and 2.
• HSTTNwithout skip-connections (NoSkip): To demonstrate the effectiveness of the skip-connections between encoder and decoder, we remove them in this variant.

In Table 2, we can observe that 2-STTN and HSTTN both perform better than STTN, which demonstrates the effectiveness of both the coarse-grained temporal dependencies and our hierarchical architecture. But the 4-STTN variant can't make further improvement, which may suffer from overfitting problems as the network goes deeper. The HSTNN outperforms NoSkip variant, which proves that aggregating information from different scales enables the model to make more precise predictions.

Importance of Capturing Global Information

• Temporal 1-D CNN Only (T-CNN): To verify the importance of global temporal information, we replace the transformer in the above T-Only with stacked 1-D convolutional neural networks to capture local temporal dependencies then make predictions.
• Spatial 1-D CNN Only (S-CNN): We replace the transformer in the above S-Only with stacked 1-D CNN to verify the significance of global spatial correlations.
• Temporal Transformer Only (T-Only): This variant only includes temporal Transformer layers to verify the effectiveness of our global temporal feature modeling and hierarchical temporal learning.
• Spatial Transformer Only (S-Only): We implement this variant including only spatial transformer layers to explore the effectiveness of global spatial information.

The performance of different variants is shown in Table 3. We can observe that T-Only can achieve a comparable result that contribute most to our HSTTN, which demonstrates the effectiveness of our hourglass-shaped hierarchical framework for capturing long-range temporal dependencies. We explore to extract spatial temporal features by CNN structure, which is widely used for local feature extraction. The performance of T-CNN and S-CNN can not match with T-Only ,S-Only and not to mention HSTTN on both datasets, which proves the importance of global information for wind power forecasting. Spatial features is only supplemental to WPF, so S-Only and S-CNN performs poorly.

Effectiveness of Spatial-Temporal Contextual Fusion

• Spatial-Temporal Transformer Only (ST-Only): To verify the effectiveness of our CFB, we implement a simple fusion variant without the contextual fusion block integrated in between. Then the outputs of the last RSTDL are simply concatenated to make predictions.

The experiments results are illustrated in Table 3. ST-Only slightly improves T-Only demonstrates the effectiveness of the global spatial information. Our HSTTN outperforms both T-Only and ST-Only reveals that the temporal and spatial features can not be casually combined and our CFBs that properly fuses spatiotemporal context features is effective.

Analysis of Different Hyperparameters

We conduct extensive experiments on three important hyperparameters in HSTTN to find the best settings.

• Kernel Size: the kernel size in contextual fusion block.
• Layer Num: the number of repeated residual spatiotemporal encoder/decoder layers.
• Dimensions: the number of embedded feature dimensions after the 1×1 convolution.

Table 4 records the hyperparameters settings and results. According to the results, we decide the kernel size, encoder layers, decoders layers and embed feature dimension as 1, 2, 1 and 16 for SDWPF and as 1, 2, 2, 16 for Engie.

6 Conclusion

In this work, we propose a hourglass-shaped encoder-decoder model termed Hierarchical Spatial-Temporal Transformer Network to deal with the challenging long-term wind power forecasting problem. The model design is motivated by two main limitations of existing works. First, most of these works are designed for short-term while lack of effective long-term prediction solutions. Second, existing wind power forecasting works lack properly designed module for spatial-temporal context feature mining. Thus, we adopt Transformer mechanism to capture both long-range temporal dependencies and global spatial correlations and carefully design a hierarchical temporal learning structure to facilitate long-term forecasting with complementary coarse-grained semantics. Moreover, we design a Contextual Fusion Block to further enhance the learned features and improve the performance.

Acknowledgments

This work was supported in part by the Guangdong Basic and Applied Basic Research Foundation (NO. 2020B1515020048), in part by the National Natural Science Foundation of China (NO. 61976250), in part by the Shenzhen Science and Technology Program (NO. JCYJ20220530141211024) and in part by the Fundamental Research Funds for the Central Universities under Grant 22lgqb25.