Transformers in Time Series: A Survey

1 Introduction

The innovation of Transformer in deep learning [Vaswani et al., 2017] has brought great interests recently due to its excellent performances in various domains [Kenton and others, 2019; Dosovitskiy et al., 2021]. Over the past few years, numerous Transformers have been proposed to significantly advance the state-of-the-art performances of various tasks. There are quite a few literature reviews from different aspects, such as in NLP [Han et al., 2021], CV [Han et al., 2022], and efficient Transformers [Tay et al., 2022].

Transformers have shown great modeling ability for longrange dependencies and interactions in sequential data and thus are appealing to time series modeling. Many variants of Transformer have been proposed to address special challenges in time series modeling and have been successfully applied to various time series tasks, such as forecasting [Li et al., 2019; Zhou et al., 2022], anomaly detection [Xu et al., 2022; Tuli et al., 2022], and classification [Zerveas et al., 2021; Yang et al., 2021]. Specifically, seasonality or periodicity is an important feature of time series [Wen et al., 2021a]. How to effectively model long-range and short-range temporal dependency and capture seasonality simultaneously remains a challenge [Wu et al., 2021; Wen et al., 2022]. We note that there exist several surveys related to deep learning for time series, including forecasting [Lim and Zohren, 2021; Benidis et al. et al., 2021], classification [Ismail Fawaz et al., 2019], anomaly detection [Choi et al., 2021; Bl´azquez-Garc´ıa et al., 2021], and data augmentation [Wen et al., 2021b], but there is no comprehensive survey for Transformers in time series. As Transformer for time series is an emerging subject in deep learning, a systematic and comprehensive survey on time series Transformers would greatly benefit the time series community.

In this paper, we aim to fill the gap by summarizing the main developments of time series Transformers. We first give a brief introduction about vanilla Transformer, and then propose a new taxonomy from perspectives of both network modifications and application domains for time series Transformers. For network modifications, we discuss the improvements made on both low-level (i.e., module) and high-level (i.e., architecture) of Transformers, to optimize the performance of time series modeling. For applications, we analyze and summarize Transformers for popular time series tasks, including forecasting, anomaly detection, and classification. For each time series Transformer, we analyze its insights, strengths, and limitations. To provide practical guidelines on how to effectively use Transformers for time series modeling, we conduct extensive empirical studies that examine multiple aspects of time series modeling, including robustness analysis, model size analysis, and seasonal-trend decomposition analysis. We conclude this work by discussing possible future directions for time series Transformers, including inductive biases for time series Transformers, Transformers and GNN for time series, pre-trained Transformers for time series, Transformers with architecture level variants, and Transformers with NAS for time series. To the best of our knowledge, this is the first work to comprehensively and systematically review the key developments of Transformers for modeling time series data. We hope this survey will ignite further research interests in time series Transformers. A corresponding resource that has been continuously updated can be found in the GitHub repository 1 .

2 Preliminaries of the Transformer

2.1 Vanilla Transformer

The vanilla Transformer [Vaswani et al., 2017] follows most competitive neural sequence models with an encoder-decoder structure. Both encoder and decoder are composed of multiple identical blocks. Each encoder block consists of a multihead self-attention module and a position-wise feed-forward network while each decoder block inserts cross-attention models between the multi-head self-attention module and the position-wise feed-forward network.

2.2 Input Encoding and Positional Encoding

Unlike LSTM or RNN, the vanilla Transformer has no recurrence. Instead, it utilizes the positional encoding added in the input embeddings, to model the sequence information. We summarize some positional encodings below.

Absolute Positional Encoding

In vanilla Transformer, for each position index t, encoding vector is given by

where ωi is the hand-crafted frequency for each dimension. Another way is to learn a set of positional embeddings for each position which is more flexible [Kenton and others, 2019; Gehring et al., 2017].

Relative Positional Encoding

Following the intuition that pairwise positional relationships between input elements is more beneficial than positions of elements, relative positional encoding methods have been proposed. For example, one of such methods is to add a learnable relative positional embedding to keys of attention mechanism [Shaw et al., 2018].

Besides the absolute and relative positional encodings, there are methods using hybrid positional encodings that combine them together [Ke et al., 2021]. Generally, the positional encoding is added to the token embedding and fed to Transformer.

2.3 Multi-head Attention

With Query-Key-Value (QKV) model, the scaled dot-product attention used by Transformer is given by

where queries Q ∈ RN×Dk , keys K ∈ RM×Dk , values V ∈ RM×Dv , N,M denote the lengths of queries and keys (or values), and Dk,Dv denote the dimensions of keys (or queries) and values. Transformer uses multi-head attention with H different sets of learned projections instead of a single attention function as

1 https://github.com/qingsongedu/time-series-transformers-review

2.4 Feed-forward and Residual Network

The feed-forward network is a fully connected module as

where H′ is outputs of previous layer, W1 ∈ RDm×Df , W2 ∈ RDf×Dm , b1 ∈ RDf , b2 ∈ RDm are trainable parameters. In a deeper module, a residual connection module followed by a layer normalization module is inserted around each module. That is,

where SelfAttn(.) denotes self-attention module and LayerNorm(.) denotes the layer normalization operation.

3 Taxonomy of Transformers in Time Series

To summarize the existing time series Transformers, we propose a taxonomy from perspectives of network modifications and application domains as illustrated in Fig. 1. Based on the taxonomy, we review the existing time series Transformers systematically. From the perspective of network modifications, we summarize the changes made on both module level and architecture level of Transformer in order to accommodate special challenges in time series modeling. From the perspective of applications, we classify time series Transformers based on their application tasks, including forecasting, anomaly detection, and classification. In the following two sections, we would delve into the existing time series Transformers from these two perspectives.

4 NetworkModifications for Time Series

4.1 Positional Encoding

As the ordering of time series matters, it is of great importance to encode the positions of input time series into Transformers. A common design is to first encode positional information as vectors and then inject them into the model as an additional input together with the input time series. How to obtain these vectors when modeling time series with Transformers can be divided into three main categories.

Vanilla Positional Encoding. A fewworks [Li et al., 2019] simply introduce vanilla positional encoding (Section 2.2) used in [Vaswani et al., 2017], which is then added to the input time series embeddings and fed to Transformer. Although this approach can extract some positional information from time series, they were unable to fully exploit the important features of time series data.

Learnable Positional Encoding. As the vanilla positional encoding is hand-crafted and less expressive and adaptive, several studies found that learning appropriate positional embeddings from time series data can be much more effective. Compared to fixed vanilla positional encoding, learned embeddings are more flexible and can adapt to specific tasks. [Zerveas et al., 2021] introduces an embedding layer in Transformer that learns embedding vectors for each position index jointly with other model parameters. [Lim et al., 2021] uses an LSTM network to encode positional embeddings, which can better exploit sequential ordering information in time series.

Timestamp Encoding. When modeling time series in realworld scenarios, the timestamp information is commonly accessible, including calendar timestamps (e.g., second, minute, hour, week, month, and year) and special timestamps (e.g., holidays and events). These timestamps are quite informative in real applications but hardly leveraged in vanilla Transformers. To mitigate the issue, Informer [Zhou et al., 2021] proposed to encode timestamps as additional positional encoding by using learnable embedding layers. A similar timestamp encoding scheme was used in Autoformer [Wu et al., 2021] and FEDformer [Zhou et al., 2022].

4.2 AttentionModule

Central to Transformer is the self-attention module. It can be viewed as a fully connected layer with weights that are dynamically generated based on the pairwise similarity of input patterns. As a result, it shares the same maximum path length as fully connected layers, but with a much less number of parameters, making it suitable for modeling long-term dependencies.

As we show in the previous section the self-attention module in the vanilla Transformer has a time and memory complexity of O(N 2 ) (N is the input time series length), which becomes the computational bottleneck when dealing with long sequences. Many efficient Transformers were proposed to reduce the quadratic complexity that can be classified into two main categories: (1) explicitly introducing a sparsity bias into the attention mechanism like LogTrans [Li et al., 2019] and Pyraformer [Liu et al., 2022a]; (2) exploring the low-rank property of the self-attention matrix to speed up the computation, e.g. Informer [Zhou et al., 2021] and FEDformer [Zhou et al., 2022]. Table ?? shows both the time and memory complexity of popular Transformers applied to time series modeling, and more details about these models will be discussed in Section 5.

4.3 Architecture-based Attention Innovation

To accommodate individual modules in Transformers for modeling time series, a number of works [Zhou et al., 2021;

Liu et al., 2022a] seek to renovate Transformers on the architecture level. Recent works introduce hierarchical architecture into Transformer to take into account the multi-resolution aspect of time series. Informer [Zhou et al., 2021] inserts max-pooling layers with stride 2 between attention blocks, which down-sample series into its half slice. Pyraformer [Liu et al., 2022a] designs a C-ary tree-based attention mechanism, in which nodes at the finest scale correspond to the original time series, while nodes in the coarser scales represent series at lower resolutions. Pyraformer developed both intrascale and inter-scale attentions in order to better capture temporal dependencies across different resolutions. Besides the ability to integrate information at different multi-resolutions, a hierarchical architecture also enjoys the benefits of efficient computation, particularly for long-time series.

5 Applications of Time Series Transformers

In this section, we review the applications of Transformer to important time series tasks, including forecasting, anomaly detection, and classification.

5.1 Transformers in Forecasting

Here we examine three common types of forecasting tasks here, i.e. time series forecasting, spatial-temporal forecasting, and event forecasting.

Time Series Forecasting

A lot of work has been done to design new Transformer variants for forecasting tasks in the latest years. Module-level and architecture-level variants are two large categories and the former consists of the majority of the up-to-date works.

Module-level variants In the module-level variants for time series forecasting, their main architectures are similar to the vanilla Transformer with minor changes. Researchers introduce various time series inductive biases to design new modules. The following summarized work consists of three different types: designing new attention modules, exploring the innovative way to normalize time series data, and utilizing the bias for token inputs, as shown in Figure 2.

The first type of variant for module-level Transformers is to design new attention modules, which is the category with the largest proportion. Here we first describe six typical works: LogTrans [Li et al., 2019], Informer [Zhou et al., 2021], AST [Wu et al., 2020a], Pyraformer [Liu et al., 2022a], Quatformer [Chen et al., 2022], and FEDformer [Zhou et al., 2022], all of which exploit sparsity inductive bias or low-rank approximation to remove noise and achieve a low-order calculation complexity. LogTrans [Li et al., 2019] proposes convolutional self-attention by employing causal convolutions to generate queries and keys in the self-attention layer. It introduces sparse bias, a Logsparse mask, in self-attention model that reduces computational complexity from O(N 2 ) to O(N logN). Instead of using explicit sparse bias, Informer [Zhou et al., 2021] selects dominant queries based on queries and key similarities, thus achieving similar improvements as LogTrans in computational complexity. It also designs a generative style decoder to produce long-term forecasting directly and thus avoids accumulative error in using one forward-step prediction for long-term forecasting. AST [Wu et al., 2020a] uses a generative adversarial encoderdecoder framework to train a sparse Transformer model for time series forecasting. It shows that adversarial training can improve time series forecasting by directly shaping the output distribution of the network to avoid error accumulation through one-step ahead inference. Pyraformer [Liu et al., 2022a] designs a hierarchical pyramidal attention module with a binary tree following the path, to capture temporal dependencies of different ranges with linear time and memory complexity. FEDformer [Zhou et al., 2022] applies attention operation in the frequency domain with Fourier transform and wavelet transform. It achieves a linear complexity by randomly selecting a fixed-size subset of frequency. Note that due to the success of Autoformer and FEDformer, it has attracted more attention in the community to explore self-attention mechanisms in the frequency domain for time series modeling. Quatformer [Chen et al., 2022] proposes learning-to-rotate attention (LRA) based on quaternions that introduce learnable period and phase information to depict intricate periodical patterns. Moreover, it decouples LRA using a global memory to achieve linear complexity.

The following three works focus on building an explicit interpretation ability of models, which follows the trend of Explainable Artificial Intelligence (XAI). TFT [Lim et al., 2021] designs a multi-horizon forecasting model with static covariate encoders, gating feature selection, and temporal self-attention decoder. It encodes and selects useful information from various covariates to perform forecasting. It also preserves interpretability by incorporating global, temporal dependency, and events. ProTran [Tang and Matteson, 2021] and SSDNet [Lin et al., 2021] combine Transformer with state space models to provide probabilistic forecasts. ProTran designs a generative modeling and inference procedure based on variational inference. SSDNet first uses Transformer to learn the temporal pattern and estimate the parameters of SSM, and then applies SSM to perform the seasonaltrend decomposition and maintain the interpretable ability.

The second type of variant for module-level Transformers is how to normalize time series data. For example, Nonstationary Transformer [Liu et al., 2022b] mainly focuses on modifying the normalization mechanism as shown in Figure 2. It explores the over-stationarization problem in time series forecasting tasks with a relatively simple plugin series stationary and De-stationary module to modify and boost the performance of various attention blocks.

The third type of variant for module-level Transformer is utilizing the bias for token input. Autoformer [Wu et al., 2021] adopts a segmentation-based representation mechanism. It devises a simple seasonal-trend decomposition architecture with an auto-correlation mechanism working as an attention module. The auto-correlation block measures the time-delay similarity between inputs signal and aggregates the top-k similar sub-series to produce the output with reduced complexity. PatchTST [Nie et al., 2023] utilizes channel-independent where each channel contains a single univariate time series that shares the same embedding within all the series, and subseries-level patch design which segmentation of time series into subseries-level patches that are served as input tokens to Transformer. Such ViT [Dosovitskiy et al., 2021] alike design improves its numerical performance in long-time time-series forecasting tasks a lot. Crossformer [Zhang and Yan, 2023] proposes a Transformer-based model utilizing cross-dimension dependency for multivariate time series forecasting. The input is embedded into a 2D vector array through the novel dimension-segment-wise embedding to preserve time and dimension information. Then, a two-stage attention layer is used to efficiently capture the cross-time and cross-dimension dependency.

Architecture-level variants Some works start to design a new transformer architecture beyond the scope of the vanilla transformer. Triformer [Cirstea et al., 2022] design a triangular,variable-specific patch attention. It has a triangular tree-type structure as the later input size shrinks exponentially and a set of variable-specific parameters making a multilayer Triformer maintain a lightweight and linear complexity. Scaleformer [Shabani et al., 2023] proposes a multi-scale framework that can be applied to the baseline transformerbased time series forecasting models (FEDformer[Zhou et al., 2022], Autoformer[Wu et al., 2021], etc.). It can improve the baseline model's performance by iteratively refining the forecasted time series at multiple scales with shared weights.

Remarks Note that DLinear [Zeng et al., 2023] questions the necessity of using Transformers for long-term time series forecasting, and shows that a simpler MLP-based model can achieve better results compared to some Transformer baselines through empirical studies. However, we notice that a recent Transformer model PatchTST [Nie et al., 2023] achieves a better numerical result compared to DLinear for long-term time series forecasting. Moreover, there is a thorough theoretical study [Yun et al., 2020] showing that the Transformer models are universal approximators of sequence-tosequence functions. It is a overclaim to question the potential of any type of method for time series forecasting based solely on experimental results from some variant instantiations of such method, especially for Transformer models which already demonstrate the performances in most machine learning-based tasks. Therefore, we conclude that summarizing the recent Transformer-based models for time series forecasting is necessary and would benefit the whole community.

Spatio-Temporal Forecasting

In spatio-temporal forecasting, both temporal and spatiotemporal dependencies are taken into account in time series Transformers for accurate forecasting.

Traffic Transformer [Cai et al., 2020] designs an encoderdecoder structure using a self-attention module to capture temporal-temporal dependencies and a graph neural network module to capture spatial dependencies. Spatial-temporal Transformer [Xu et al., 2020] for traffic flow forecasting takes a step further. Besides introducing a temporal Transformer block to capture temporal dependencies, it also designs a spatial Transformer block, together with a graph convolution network, to better capture spatial-spatial dependencies. Spatio-temporal graph Transformer [Yu et al., 2020] designs an attention-based graph convolution mechanism that is able to learn a complicated temporal-spatial attention pattern to improve pedestrian trajectory prediction. Earthformer [Gao et al., 2022] proposes a cuboid attention for efficient spacetime modeling, which decomposes the data into cuboids and applies cuboid-level self-attention in parallel. It shows that Earthformer achieves superior performance in weather and climate forecasting. Recently, AirFormer [Liang et al., 2023] devises a dartboard spatial self-attention module and a causal temporal self-attention module to efficiently capture spatial correlations and temporal dependencies, respectively. Furthermore, it enhances Transformers with latent variables to capture data uncertainty and improve air quality forecasting.

Event Forecasting

Event sequence data with irregular and asynchronous timestamps are naturally observed in many real-life applications, which is in contrast to regular time series data with equal sampling intervals. Event forecasting or prediction aims to predict the times and marks of future events given the history of past events, and it is often modeled by temporal point processes (TPP) [Yan et al., 2019; Shchur et al., 2021].

Recently, several neural TPP models incorporate Transformers in order to improve the performance of event prediction. Self-attentive Hawkes process (SAHP) [Zhang et al., 2020] and Transformer Hawkes process (THP) [Zuo et al., 2020] adopt Transformer encoder architecture to summarize the influence of historical events and compute the intensity function for event prediction. They modify the positional encoding by translating time intervals into sinusoidal functions such that the intervals between events can be utilized. Later, a more flexible named attentive neural datalog through time (ANDTT) [Mei et al., 2022] is proposed to extend SAHP/THP schemes by embedding all possible events and times with attention as well. Experiments show that it can better capture sophisticated event dependencies than existing methods.

5.2 Transformers in Anomaly Detection

Transformer based architecture also benefits the time series anomaly detection task with the ability to model temporal dependency, which brings high detection quality [Xu et al., 2022]. Besides, in multiple studies, including TranAD [Tuli et al., 2022], MT-RVAE [Wang et al., 2022], and TransAnomaly [Zhang et al., 2021], researchers proposed to combine Transformer with neural generative models, such as VAEs [Kingma and Welling, 2014] and GANs [Goodfellow et al., 2014], for better performance in anomaly detection. We will elaborate on these models in the following part.

TranAD [Tuli et al., 2022] proposes an adversarial training procedure to amplify reconstruction errors as a simple Transformer-based network tends to miss small deviation of anomaly. GAN style adversarial training procedure is designed by two Transformer encoders and two Transformer decoders to gain stability. Ablation study shows that, if Transformer-based encoder-decoder is replaced, F1 score drops nearly 11%, indicating the effect of Transformer architecture on time series anomaly detection.

MT-RVAE [Wang et al., 2022] and TransAnomaly [Zhang et al., 2021] combine VAE with Transformer, but they share different purposes. TransAnomaly combines VAE with Transformer to allow more parallelization and reduce training costs by nearly 80%. In MT-RVAE, a multiscale Transformer is designed to extract and integrate time-series information at different scales. It overcomes the shortcomings of traditional Transformers where only local information is extracted for sequential analysis.

GTA [Chen et al., 2021c] combines Transformer with graph-based learning architecture for multivariate time series anomaly detection. Note that, MT-RVAE is also for multivariate time series but with few dimensions or insufficient close relationships among sequences where the graph neural network model does not work well. To deal with such challenge, MT-RVAE modifies the positional encoding module and introduces feature-learning module. Instead, GTA contains a graph convolution structure to model the influence propagation process. Similar to MT-RVAE, GTA also considers "global" information, yet by replacing vanilla multi-head attention with a multi-branch attention mechanism, that is, a combination of global-learned attention, vanilla multi-head attention, and neighborhood convolution.

AnomalyTrans [Xu et al., 2022] combines Transformer and Gaussian prior-Association to make anomalies more distinguishable. Sharing similar motivation as TranAD, AnomalyTrans achieves the goal in a different way. The insight is that it is harder for anomalies to build strong associations with the whole series while easier with adjacent time points compared with normality. In AnomalyTrans, prior-association and series-association are modeled simultaneously. Besides reconstruction loss, the anomaly model is optimized by the minimax strategy to constrain the prior- and series- associations for more distinguishable association discrepancy.

5.3 Transformers in Classification

Transformer is proved to be effective in various time series classification tasks due to its prominent capability in capturing long-term dependency. GTN [Liu et al., 2021] uses a two-tower Transformer with each tower respectively working on time-step-wise attention and channel-wise attention. To merge the feature of the two towers, a learnable weighted concatenation (also known as 'gating') is used. The proposed extension of Transformer achieves state-of-the-art results on 13 multivariate time series classifications. [Rußwurm and K¨orner, 2020] studied the self-attention based Transformer for raw optical satellite time series classification and obtained the best results compared with recurrent and convolutional neural networks. Recently, TARNet [Chowdhury et al., 2022] designs Transformers to learn task-aware data reconstruction that augments classification performance, which utilizes attention score for important timestamps masking and reconstruction and brings superior performance.

Pre-trained Transformers are also investigated in classification tasks. [Yuan and Lin, 2020] studies the Transformer for raw optical satellite image time series classification. The authors use self-supervised pre-trained schema because of limited labeled data. [Zerveas et al., 2021] introduced an unsupervised pre-trained framework and the model is pre-trained with proportionally masked data. The pre-trained models are then fine-tuned in downstream tasks such as classification. [Yang et al., 2021] proposes to use large-scale pre-trained speech processing model for downstream time series classification problems and generates 19 competitive results on 30 popular time series classification datasets.

6 Experimental Evaluation and Discussion

We conduct preliminary empirical studies on a typical challenging benchmark dataset ETTm2 [Zhou et al., 2021] to analyze how Transformers work on time series data. Since classic statistical ARIMA/ETS [Hyndman and Khandakar, 2008] models and basic RNN/CNN models perform inferior to Transformers in this dataset as shown in [Zhou et al., 2021; Wu et al., 2021], we focus on popular time series Transformers with different configurations in the experiments.

Robustness Analysis

A lot of works we describe above carefully design attention modules to lower the quadratic calculation and memory complexity, though they practically use a short fixed-size input to achieve the best result in their reported experiments. It makes us question the actual usage of such an efficient design. We perform a robust experiment with prolonging input sequence length to verify their prediction power and robustness when dealing with long-term input sequences in Table ??.

As in Table ??, when we compare the prediction results with prolonging input length, various Transformer-based model deteriorates quickly. This phenomenon makes a lot of carefully designed Transformers impractical in long-term forecasting tasks since they cannot effectively utilize long input information. More works and designs need to be investigated to fully utilize long sequence input for better performance.

Model Size Analysis

Before being introduced into the field of time series prediction, Transformer has shown dominant performance in NLP and CV communities [Vaswani et al., 2017; Kenton and others, 2019; Han et al., 2021; Han et al., 2022]. One of the key advantages Transformer holds in these fields is being able to increase prediction power through increasing model size. Usually, the model capacity is controlled by Transformer's layer number, which is commonly set between 12 to 128. Yet as shown in the experiments of Table ??, when we compare the prediction result with different Transformer models with various numbers of layers, the Transformer with 3 to 6 layers often achieves better results. It raises a question about how to design a proper Transformer architecture with deeper layers to increase the model's capacity and achieve better forecasting performance.

Seasonal-Trend Decomposition Analysis

In recent studies, researchers [Wu et al., 2021; Zhou et al., 2022; Lin et al., 2021; Liu et al., 2022a] begin to realize that the seasonal-trend decomposition [Cleveland et al., 1990; Wen et al., 2020] is a crucial part of Transformer's performance in time series forecasting. As an experiment shown in Table ??, we adopt a simple moving average seasonal-trend decomposition architecture proposed in [Wu et al., 2021] to test various attention modules. It can be seen that the simple seasonal-trend decomposition model can significantly boost model's performance by 50 % to 80%. It is a unique block and such performance boosting through decomposition seems a consistent phenomenon in time series forecasting for Transformer's application, which is worth further investigating for more advanced and carefully designed time series decomposition schemes.

7 Future Research Opportunities

7.1 Inductive Biases for Time Series Transformers

Vanilla Transformer does not make any assumptions about data patterns and characteristics. Although it is a general and universal network for modeling long-range dependencies, it also comes with a price, i.e., lots of data are needed to train Transformer to improve the generalization and avoid data overfitting. One of the key features of time series data is its seasonal/periodic and trend patterns [Wen et al., 2019; Cleveland et al., 1990]. Some recent studies have shown that incorporating series periodicity [Wu et al., 2021] or frequency processing [Zhou et al., 2022] into time series Transformer can enhance performance significantly. Moreover, it is interesting that some studies adopt a seemly opposite inductive bias, but both achieve good numerical improvement: [Nie et al., 2023] removes the cross-channel dependency by utilizing a channel-independent attention module, while an interesting work [Zhang and Yan, 2023] improves its experimental performance by utilizing cross-dimension dependency with a two-stage attention mechanism. Clearly, we have noise and signals in such a cross-channel learning paradigm, but a clever way to utilize such inductive bias to suppress the noise and extract the signal is still desired. Thus, one future direction is to consider more effective ways to induce inductive biases into Transformers based on the understanding of time series data and characteristics of specific tasks.

7.2 Transformers and GNN for Time Series

Multivariate and spatio-temporal time series are becoming increasingly common in applications, calling for additional techniques to handle high dimensionality, especially the ability to capture the underlying relationships among dimensions. Introducing graph neural networks (GNNs) is a natural way to model spatial dependency or relationships among dimensions. Recently, several studies have demonstrated that the combination of GNN and Transformers/attentions could bring not only significant performance improvements like in traffic forecasting [Cai et al., 2020; Xu et al., 2020] and multimodal forecasting [Li et al., 2021], but also better understanding of the spatio-temporal dynamics and latent causality. It is an important future direction to combine Transformers and GNNs for effectively spatial-temporal modeling.

7.3 Pre-trained Transformers for Time Series

Large-scale pre-trained Transformer models have significantly boosted the performance for various tasks in NLP [Kenton and others, 2019; Brown et al., 2020] and CV [Chen et al., 2021a]. However, there are limited works on pre-trained Transformers for time series, and existing studies mainly focus on time series classification [Zerveas et al., 2021; Yang et al., 2021]. Therefore, how to develop appropriate pre-trained Transformer models for different tasks in time series remains to be examined in the future.

7.4 Transformers with Architecture Level Variants

Most developed Transformer models for time series maintain the vanilla Transformer's architecture with modifications mainly in the attention module. We might borrow the idea from Transformer variants in NLP and CV which also have architecture-level model designs to fit different purposes, such as lightweight [Wu et al., 2020b; Mehta et al., 2021], cross-block connectivity [Bapna et al., 2018], adaptive computation time [Dehghani et al., 2019; Xin et al., 2020], and recurrence [Dai et al., 2019]. Therefore, one future direction is to consider more architecture-level designs for Transformers specifically optimized for time series data and tasks.

7.5 Transformers with NAS for Time Series

Hyper-parameters, such as embedding dimension and the number of heads/layers, can largely affect the performance of Transformers. Manual configuring these hyper-parameters is time-consuming and often results in suboptimal performance. AutoML technique like Neural architecture search (NAS) [Elsken et al., 2019; Wang et al., 2020] has been a popular technique for discovering effective deep neural architectures, and automating Transformer design using NAS in NLP and CV can be found in recent studies [So et al., 2019; Chen et al., 2021b]. For industry-scale time series data which can be of both high dimension and long length, automatically discovering both memory- and computational-efficient Transformer architectures is of practical importance, making it an important future direction for time series Transformers.

8 Conclusion

In this paper, we provide a survey on time series Transformers. We organize the reviewed methods in a new taxonomy, summarize representative methods in each category, discuss their strengths and limitations by experimental evaluation, and highlight future research directions.