SCAT: A Time Series Forecasting with Spectral Central Alternating Transformers

1 Introduction

Time-series forecasting[Box et al., 2015] is one of the most important renewable energy power-generation technologies. Moreover, carbon neutrality[Khalifa et al., 2022] is a global goal for realizing the sustainable development of the world's resources. However, wind[Moradzadeh et al., 2021] and solar power[Doubleday et al., 2020] stations need more accurate power prediction algorithms[Wang et al., 2011; Elsaraiti and Merabet, 2022; Voyant et al., 2017] to improve energy utilization, which can support the rapid growth of power-generation capacity. As depicted in Figure 1, wind and solar power generation can be forecasted using input features such as weather conditions.

∗Corresponding author.

With the rapid proliferation of artificial intelligence, research focused on time-series forecasting[Le Guen and Thome, 2020] utilizing deep learning models has surged. In this array of models, the transformer architecture, initially devised for natural language processing (NLP), has emerged as a versatile tool with applications extending to diverse domains, including audio processing[Chan et al., 2015], computer vision (CV)[Dosovitskiy et al., 2020], and time-series analysis[Lim and Zohren, 2021]. Transformers have exhibited remarkable capabilities in capturing dependencies among elements and unveiling latent relationships among tokens within sequences. More importantly, various transformerbased variants, such as Autoformer, Reformer, Pyraformer, and ETSformer, have proved their mettle in time-series forecasting. These models have pushed the boundaries of prediction accuracy by leveraging innovative techniques like series decomposition.

This paper introduces a novel concept involving utilizing spectral clustering centers as pivotal components in attention mechanisms for time-series data. The proposed spectral central alternating transformer (SCAT) encompasses two main innovative designs:

• Series Clustering Center. Time-series data often comprise diverse components, with examples like wind and solar generation data encompassing weather-related information and generation capacity metrics. However, weather-related data often exhibit inaccuracies stemming from limitations in data acquisition equipment or challenges in advanced weather prediction techniques.

Notably, previous research has paid limited attention to the adverse impact of noise present in these sequences, particularly at the local level, which can significantly affect prediction accuracy. In contrast, SCAT addresses this issue by employing a preprocessing step that involves clustering and classifying the meteorological time-series data. This process identifies and separates central points within each sub-sequence, each representing distinct weather patterns. This crucial preprocessing step occurs before the primary model architecture is computed. Subsequently, we utilize these spectral clustering centers as computational cores to eliminate the redundancy of traditional attention computing structures and effectively capture global semantic information, enhancing the overall predictive performance of our model.

• Alternating Attention. We employ multivariate signals to predict single-channel power series in power timeseries forecasting[Das et al., 2023]. Unlike the prevalent transformer-based models that perform feature extraction based on query-key connections among tokens, alternating attention utilizes clustering centers as attention computation cores. This unique approach considers both the Euclidean distance and feature distance between tokens. After nonlinear propagation through activation functions, we implement sequence-level attention filtering, instead of token-level attention, thereby preserving the global characteristics of tokens within each channel to the maximum extent.

SCAT attained strong performance in time-series prediction through these two innovative designs. Existing time-series forecasting methods often adopt sequence decomposition[Xiao et al., 2023] or short-sequence selflearning modeling[Yao et al., 2023] to identify correlations among sub-sequences. However, when applied to power time-series forecasting, numerous indistinguishable anomalies in the sequences lead to a decline in the prediction accuracy[Bonifati et al., 2022]. We propose the SCAT, which incorporates spectral clustering centers to ensure that the model preserves global feature integrity during training. Moreover, we utilize alternating attention as the backbone for feature extraction on series tokens, considering both the Euclidean distance and vector distance perspectives. The implementation of sequence-level attention scores effectively mitigated token oversampling within each channel. Our proposed SCAT model achieved good results through extensive evaluation of multiple prediction datasets.

2 RelatedWork

This paper's research scope encompasses three primary facets: (i) the investigation of time series clustering; (ii) the development of time series prediction methodologies founded on transformer-based approaches; (iii) the exploration of the utility of alternative architectural models in the domain of power time series prediction.

2.1 Clustering in Time Series Forecasting

As an unsupervised learning algorithm, clustering achieves classification by computing feature distances among entities. K- means [MacQueen, 1967] initiates the process by randomly initializing cluster centers and then partitions samples by optimizing the mean square error within each class. Clustering algorithms that rely on Kullback-Leibler (KL) [Wang and Deng, 2018] divergence excel in spatially partitioning features by minimizing KL loss and bringing samples closer to the cluster center points. In contrast, subspace clustering (SC)[Zhang et al., 2018; Fan et al., 2021] linearly characterizes all samples by partitioning the subspace and optimizing feature representations across different samples. The Gaussian mixture model (GMM) [Peel and MacLahlan, 2000]clustering method assumes that each sample follows an independent Gaussian distribution and addresses the issue of an imbalanced sample distribution by optimizing the likelihood function[Zhang et al., 2021]. In contrast, the clustering method based on mutual information dispenses with any assumptions about the sample distributions. It achieves sample classification by maximizing the mutual information between the input variable, X, and the output variable, Y , effectively mitigating regression problems. The characteristic distribution of power time-series depends on the output curve of power-generation equipment. The clustering based on spectral clustering[Chen et al., 2017; Bianchi et al., 2020] constructs a similarity matrix through sample spacing and uses the topological structure of weather characteristics to classify samples accurately.

2.2 Transformer-Based Time Series Forecasting

The transformer[Vaswani et al., 2017] model offers a more versatile global feature extraction approach compared with the convolutional neural network, and its parallel computing architecture within the attention mechanism has been proved to be faster than the one-way computing method employed by the recurrent neural network. Many research efforts in time-series forecasting have also utilized the transformer architecture. Informer[Zhou et al., 2021] streamlines the calculation time complexity to O(nlogn) through ProbSpare Self-Attention and mitigates error propagation by employing one-step prediction during the forecasting stage. Autoformer further enhances the self-attention mechanism, utilizing Auto-Correlation for subsequence correlation and Series Decomp for time-series decomposition to analyze seasonal information. ETSformer [Woo et al., 2022] adopts an alternative approach, employing multilayered stacked feature extraction layers to unearth the remaining seasonal information embedded in the data's intermediate layers. For longer time-series prediction tasks, FEDformer [Zhou et al., 2022] transforms time-domain information into frequencydomain data and leverages low-rank approximation to deliver advanced performance while significantly reducing computational complexity. Pyraformer [Liu et al., 2022] implements stacked pyramid-shaped attention mechanisms for feature extraction to balance computational efficiency and spatial complexity. Reformer[Kitaev et al., 2020] enhances the computational architecture of attention by removing the query matrix from the traditional attention mechanism and replacing attention weights with functions of keys, thus significantly expanding the sequence length it can accommodate. Patching mechanism-based design, as seen in PatchTST[Nie et al., 2023], has achieved good results in time-series prediction tasks. This mechanism uses subsequences as tokens for feature extraction calculations, ensuring sequence locality. However, in power time-series tasks, the pronounced sequence locality often drifts the entire prediction owing to noise. To address this concern, we introduce the clustering center as the attention computing component and employ a novel token feature-extraction mechanism. This approach effectively balances local developmental trends with global features and optimizes prediction outcomes.

2.3 Other Power Time-Series ForecastingModels

ARIMA[Ariyo et al., 2014], a conventional regression variable algorithm, enjoys widespread usage in wind and solar power prediction. Simultaneously, the evolution of deep learning, particularly long short-term memory (LSTM)[Graves and Graves, 2012; Salinas et al., 2020; Shih et al., 2019], has made significant strides. LSTM's sequential computing mechanism aligns seamlessly with the structural characteristics of time-series flow data, yielding noteworthy research outcomes. Furthermore, equal-recurrent neural networks have garnered considerable attention and research results owing to their suitability for time-series analysis. In specific time intervals, hybrid models like ARMIANN[Panigrahi et al., 2018], which fuse ARIMA and artificial neural networks, have demonstrated enhanced predictive capabilities. Additionally, temporal convolutional network (TCN)[Bai et al., 2018] has reduced errors in wind power prediction through its innovative cavity convolution design. Moreover, the combination of generative adversarial networks[Yoon et al., 2019; Luo et al., 2018; Shen et al., 2018] and convolutional networks[Rushe and Mac Namee, 2019] has proved to be effective in improving hourly photovoltaic power predictions. Moreover, some researchers have explored the efficacy of simple linear layer models for time-series prediction, achieving impressive outcomes. DLinear[Zeng et al., 2023], for instance, deconstructs the original data into moving average classification and seasonal trend components. Subsequently, it routes these data through an independent channel of a linear layer to execute series predictions. NLinear, in contrast, enhances prediction results on unevenly distributed datasets through moving averages. SCAT, our time-series prediction algorithm based on the Transformer architecture, leverages alternating attention and centers around the cluster center. This unique approach endows SCAT with a potent combination of deep learning network generalization capability and a linear layer network's intuitive feature extraction prowess. Consequently, it performs better when predicting power-related time-series data.

3 Our Method

This section provides a comprehensive exposition of SCAT, including its operational principles and architectural design. In Section 3.1, we outline the formulation for power timeseries prediction. Section 3.2 delves into the intricacies of data preprocessing, while Section 3.3 offers an in-depth exploration of the SCAT architecture. Furthermore, Sections 3.4-3.6 sequentially detail the underlying design principles and calculation formulas for each constituent element within the SCAT framework. The detailed symbol introduction is shown in Table 1.

3.1 ProblemFormulation

The power time series comprises two components: the meteorological feature and the power series. These two elements exhibit a one-to-one correspondence in time. We aimed to predict the output power Y = {yl+1, yl+2, ..., yn}, denoted as yt at time t, based on an input vector x consisting of n-dimensional meteorological features of length L is X = {{x11 , x12 , ..., x1l }, {x21 , x22 , ..., x2l }, ..., {xn1 , xn2 , ..., xnl }}. We can abstract this problem as a univariate variable forecasting task for a multivariable time series. Before commencing spectral clustering, meticulous analysis and refinement of the dataset are essential.

3.2 Model Architecture

As shown in Figure 2, the computational process of the SCAT involves two key phases: cluster center identification and power prediction. Initially, the model determines the dataset's optimal cluster number and center using the spectral clustering algorithm applied to the input multivariable sequence. The model embeds this cluster center through the cluster center embedding into the subsequent alternating transformer computation architecture as a prompting feature.

In the query projection encoder layer of the alternating transformer, SCAT identifies the nearest cluster center to the current token using a matrix calculation derived from both the cluster center and token features. This utilization of the clustering center enhances the model's feature perception across the entire sequence space. Simultaneously, this approach prevents the attention calculation mechanism from being trapped in local optimal solutions while expanding the model's global perspective. Finally, the final predicted power is output through the linear decoding layer.

3.3 Spectral Clustering

We utilize a Gaussian kernel function in the spectral clustering of SCAT on power time series, as illustrated in the left section of Figure 2. The process begins with calculating the similarity matrix S for the input samples. Subsequently, the adjacency matrix W and the degree matrix D are constructed from S.

Once the clustering dimension k has been determined, the Laplacian matrix L is computed and normalized.

Using the normalized Laplacian matrix, we calculate the eigenvector f corresponding to the smallest k eigenvalues. f is then further normalized to create the [n ∗ k]-dimensional eigenmatrix F .

Each row in F is treated as a k-dimensional sample, resulting in n samples. We apply the selected clustering method to these samples, resulting in a clustering dimension of k. Finally, this process yields cluster partition divisions denoted as C = (c1, c2, ..., ck).

To attain optimal spectral clustering outcomes across datasets with varied distributions, we compute the intercluster distance to the intra-cluster distance ratio, manipulating multiple gamma values and category numbers, k. The matrix Bk represents the covariance between classes, while Wk represents the covariance matrix within the class. The specific formula is presented below:

The variable cq denotes the central point of class q, while ce represents the central point of the dataset. Additionally, nq represents the number of data points in class q, and Cq denotes the dataset corresponding to class q.

The Calinski-Harabasz (CH) metric guided our evaluation, where a higher CH score signifies reduced intracategory covariance, increased inter-category covariance, and enhanced overall classification. We automatically chose the parameters that yielded the most favorable outcomes to obtain the clustering results.

3.4 Alternating Transformers

The spectral clustering algorithm divides tokens into k dimensions based on time points, resulting in the clustering core [k ∗ k] and transforming [k ∗ k] to [k ∗ e] by onedimensional convolution in the cluster center embedding. Upon acquiring the cluster center, SCAT computes the input sequence matrix to derive the corresponding Q, ˆQ, and K. These components are then fed into the alternating transformer.

Q undergoes reduction via two convolutional layers to yield an intermediate feature in the computation with ˆQ. In this case, the feature dimensions of ˆQ and C are consistent.

Figure 3 shows the computational process of the alternating transformer, where Q = {q1, q2, ..., qL}, ˆQ = {ˆ q1, ˆ q2, ..., ˆ qL}, C = {c1, c2, ..., cL}. To capture spatial and semantic hierarchical features, we compute the attention values between cluster centers (C) and Q, along with the Euclidean distance between C and ˆQ. We combine these values and apply the sigmoid activation function for nonlinear transformation. This process yields an eigenmatrix representing tokens' mutual and ownership relationships with cluster centers.

At this stage, we consider the affiliation relationship between tokens and cluster centers as determined.

Consequently, we select the I-dimensional feature of each token belonging to the i - th cluster center instead of merging tokens with all cluster centers for vector representation. The Alternating attention matrix is alternately reconstructed using sequence-level features by indexing. The output prediction result is characterized using the softmax function and scaled accordingly. X represents the calculated matrix [aik] obtained in Figure 3:

This attention computation method transforms token-level features into sequence-level features, preserving the positional characteristics of tokens in the global context and effectively mitigating data anomalies resulting from local fluctuations.

3.5 Loss Function

We employ trade-loss as our loss function, which enhances the loss signal at abrupt points in the time series. ˆ y = ˆ yi - ˆ yi-1 and y = yi - yi-1, Trade loss achieves the maximum extreme value at Condition = (ˆ yi - ˆ yi-1) ∗ (yi -yi-1) < 0.

Trade loss as the loss function enables the model to assess the series' trend development state more effectively. In Equation 10, yi denotes the true value, and ˆ yi represents the predicted value.

4 Experiments

4.1 Datasets

We extensively examined 10 real-world datasets in our experiments. (1) ETT encompasses power transformer data with seven variables over 2 years, from July 2016 to July 2018, collected at 15-minute intervals. (2) Weather includes weather data spanning 21 variables in 2020, recorded at 10minute intervals. (3) Traffic consisted of traffic data across 862 variables gathered between January 2015 and December 2016. (4) Electricity includes 321 variables of electricity data gathered between July 2016 and July 2019, recorded at hourly intervals. (5) Windm comprises five wind power-generation datasets from various regions, incorporating 14 pertinent features, collected at 15-minute intervals. The details of datasets are presented in Table 2.

4.2 Forward process

Before conducting spectral clustering, we had to thoroughly analyze and refine the dataset. This refinement process encompassed two main aspects: handling missing points and addressing outliers. We used an interpolation method employing the average of neighboring critical values to address discrete missing data points. Conversely, we considered large sections with substantial minus points. Notably, values exceeding these limits were adjusted to correspond with the accepted maximum wind speed to account for differing operational parameters regarding the maximum wind speed across stations. Additionally, data normalization was conducted, particularly for attributes with disparate dimensions, such as wind direction and wind speed. Considering the directional nature of wind (ranging from 0°to 360°, we achieved normalization using the transformation formula (x - 0.00001) / 360.

4.3 Experimental Settings

The experimental settings are presented in baselines, evaluation criteria, and detailed experimental parameter settings.

Baselines:Our model selection process focused on contemporary models that have demonstrated good performance in recent time series prediction tasks. The 11 baseline models encompassed Autoformer[Wu et al., 2021], PatchTST [Nie et al., 2023], DLinear[Zeng et al., 2023], ETSformer[Woo et al., 2022], FEDformer[Zhou et al., 2022], Pyraformer [Liu et al., 2022], LightTS[Campos et al., 2023], MICN[Wang et al., 2023]and TimesNet[Wu et al., 2023]. Additionally, we included two models, HI[Cui et al., 2021] and STID[Shao et al., 2022], tailored explicitly for power time series prediction.

Evaluation Metrics: We employed the mean square error (MSE) and mean absolute error (MAE) as fundamental metrics for comparison.

Detailed Settings: The SCAT architecture includes two encoder layers and one decoder layer. We determined the optimal classification number for diverse datasets by combining gamma and k parameters through spectral clustering. We set the cluster center dimension to k, fixed the model dimension at 512, and used eight attention components. Additionally, we applied a dropout rate of 0.1. To mitigate randomness, we conducted three repeated experiments for each experimental group, determining the model's final performance based on the best and average results. The training batch size was 32. We utilized Adam with an initial learning rate in {10-3 , 5×10-4 , 10-4 , 5×10-5 }. The experimental framework used Pytorch and the algorithm ran on RTX 3080 GPU.

4.4 Results and Analysis

Table 3 presents the combined performance of the multivariate forecast univariate tasks of SCAT and the baseline models across 10 datasets. encompassing five wind power datasets. SCAT achieved good results on the Wind and weather datasets, yielding an average comprehensive prediction improvement of 10.2% and 49.1% respectively compared with DLinear. Notably, on the Wind dataset, SCAT's performance notably surpassed that of specifically designed models like HI and STID for power time-series, showcasing an overall performance enhancement of 40%. The correlation between wind power and meteorological factors contributed to SCAT's performance increase in the weather dataset compared with PatchTST's suboptimal performance. However, SCAT's sensitivity to highly fluctuating data distributions led to numerous ineffective feature calculations within its alternating transformer mechanism for ETT and electricity datasets, resulting in degraded prediction performance due to their smoother data distributions.

In long-term power-series prediction, the model utilizes future meteorological features obtained at present, leading to continuous errors accumulating during the series decomposition and inference processes. These errors escalated with longer forecast windows, causing rapid increases in forecast losses. SCAT excelled over long durations owing to the inherent spatial information stored in the cluster core, providing accurate spatial features consistently. Among the baselines, PatchTST performed most similarly to SCAT. This proximity in performance can be attributed to these datasets' shared concept of sequence-level learning, allowing for more comprehensive acquisition of semantic information and closer prediction of actual power values.

DLinear, employing a superficial linear layer for feature extraction, abandons complex methods, achieving enhanced prediction performance. However, in the visual prediction results, wind power remained zero for approximately 6 hours daily owing to insufficient wind speed, leading to continuous zeros in the power sequence. These actual but continuous zero values cannot be considered outliers. Nevertheless, in DLinear, these zeros significantly impacted the fully connected layer's calculations, causing substantial deviations in the prediction results.

Training Efficiency

We assessed the training efficiency of all models under identical conditions, and the results are presented in Figure 4. Notably, the linear predictive models DLinear and LightTS exhibited the shortest training times for a single epoch. In transformer-based models, ETSformer demonstrated the highest efficiency, followed by SCAT, which outperformed Pyraformer, Autoformer, and FEDformer in training efficiency. Conversely, TimesNet, relying on graph information mining, exhibited the slowest training efficiency among all models. It required almost four times the computational time of SCAT. In summary, SCAT showcased slightly superior computational efficiency compared with other transformerbased models.

5 Ablation Study

We comprehensively assessed each component's role in SCAT through ablation experiments, as shown in Fig. 5. SCAT's primary objective is to utilize cluster centers to influence global features in sequences of varying lengths. To evaluate this, we achieved it by randomly initializing cluster centers or setting them as zero vectors to eliminate global features' impact on SCAT deliberately. However, the random initialization strategy had a more significant negative impact. Simultaneously, we assessed the impact of the loss function on SCAT's predictive performance. The MSE indicator employing trade-loss exhibited a decrease across all datasets. Because trade-loss bolstered the sequence difference loss, it led to an attenuation in optimizing MAE. However, regarding comprehensive performance, SCAT using trade-loss showed lower predictive losses.

6 Conclusion and FutureWork

This paper introduced the SCAT model for time-series prediction, incorporating two significant innovations. First, it incorporates clustering centers as global covariables to adjust the model's prediction preferences in long-distance series predictions. Addressing the challenge of falling into local optimal solutions significantly enhances prediction accuracy during the prediction process. Second, the token-level feature selection is elevated to the sentence level by replacing the original full attention with an alternating attention mechanism. Additionally, two characteristic calculation methods, Euclidean distance and token similarity in numerical space, mitigate unreasonable calculation results in time-series prediction. However, during spectral clustering, the clustering logic of the algorithm is not aligned with the actual power numerical distribution, leading to clustering centers being biased toward theoretical centers rather than actual centers. In future research, we aim to explore the practical application of clustering algorithms in power time-series. We will focus on developing a clustering algorithm closely aligned with actual power distribution to enhance SCAT's prediction performance further.

Acknowledgements

This work is supported by the National Key Research and Development Program of China (No.2021ZD0112400), the 111 Project(No.D23006), the National Natural Science Foundation of China (Nos.62076045, 62202080), the Basic Project for Universities from the Educational Department of Liaoning Province(No.LJKFZ20220290), the Science and Technology Project of Liaoning Province (No.2023JH1/10400083), Dalian Major Projects of Basic Research (No.2023JJ11CG002).