SpecAR-Net: Spectrogram Analysis and Representation Network for Time Series

1 Introduction

With the advent of the era of "Internet of Things" and "Comprehensive Perception", various sensors have been extensively deployed and utilized, leading to an explosive growth in the scale of time series [Cook et al., 2020]. Extracting valuable information from massive time series has become increasingly crucial. As a result, the time series analysis has attracted a growing number of researchers. Currently, time series analysis has been widely applied in numerous fields, e.g., finance [Livieris et al., 2020], electricity [Cai et al., 2020], transportation [Gasparin et al., 2021], and the healthcare sector [Stoean et al., 2020], etc.

Recently, deep learning is playing a crucial role in time series analysis. With the powerful feature representation capability, many deep time series learning methods have been proposed and achieved great success in classification, anomaly detection, short/long-term forecasting, etc. One typical category of these methods is based on recurrent neural networks (RNNs) [Wang et al., 2022; Yu et al., 2021], where the sequence modeling is completed by recursively encoding the first-order dependency between the preceding and subsequent elements. However, when modeling long-term sequences, it is easy to encounter gradient vanishing and explosion problems, and it is also difficult to enjoy the advantages of parallel processing [Hochreiter, 1998]. Another typical category is the convolution-based methods [Aksan and Hilliges, 2019; Thill et al., 2021], which can easily process sequential data in parallel. However, limited by the computation mechanism of shared convolutions in the local receptive fields, convolution models are often insufficient to characterize the longterm relationships. To overcome those shortcomings of recurrent and convolutional networks, the Self-Attention (SA) based Transformer [Liu et al., 2022a; Liu et al., 2022b] has been proposed. Transformer balances the long-term dependency encoding capability and the benefits of parallel computing, resulting in widely used backbone in various sequence modeling tasks. However, time series is coupled with multiple patterns, and the temporal dependencies captured by point-by-point representation and aggregation are often submerged [Wu et al., 2021].

As a highly abstract information body, time series couple multiple components such as trend (overall envelope), periodicity (multiple frequency components), mutagenicity (high frequency components), etc. Considering such highlycoupled property, it is almost impossible for pure-timedomain learning models to achieve complete semantic representation from time series. To overcome such limitation, introducing frequency domain analysis into modern deep backbones is proven to be an effective technical approach, and has become a current research trend. There are two notable works among these efforts, FEDformer [Zhou et al., 2022] and TimesNet [Wu et al., 2023]. Motivated by the fact that time series tend to have sparsity in frequency domain, FEDformer uses a small mount of randomly selected frequency components to reduce the complexity of time-domain SA from O(N 2 ) to O(N), resulting in a more agile model than Informer (O(log(N)N)). In the meanwhile, FEDformer keeps both low and high frequency components. By exploiting the posterior of Fourier transform, TimesNet constructs a 2D period-time space by folding the time series according to the dominant periods and conducts multi-scale 2D convolution to capture the intra-period- and inter-period-dependency.

In summary, both methods are eventually trying to achieve better sequence representation by using a selected frequency components, whether in a prior way [Zhou et al., 2022] or a posterior (data-driven) way [Wu et al., 2023]. And from the experimental results, the data-driven selection mechanism can achieve better results. While none of these approaches discuss whether the patterns of chosen frequency components over time, i.e., the joint characteristics of timefrequency domain, are helpful to the downstream tasks. In particular, TimesNet solely use the frequencies with high response values via Fourier transform to select dominant periodic components from time series. The subsequent modeling process is still carried out in the time domain, utilizing the data repeatedly guided by significant periods. There is no explicit learning mechanism to further capture/utilize the timevarying properties of the frequency domain components with strong responses. Motivated by this, this paper attempts to investigate the effectiveness of time-frequency joint learning mechanism for time series analysis tasks on the basis of significant frequency domain component filtering. Then, a unified time-frequency spectrogram analysis and representation network (SpecAR-Net) is presented.

Overall, this approach has the following benefits: firstly, it overcomes the bottleneck of one-dimensional data representation by decoupling the multiple components of time series in a higher-dimensional data space. Secondly, a concise unified framework for learning cross-domain representation is constructed, which enables the joint analysis of time and frequency domain features in time series. Technically, to facilitate the universality of the proposed method, a plugand-play SpecAR-Block is designed, which is compatible for most deep sequential models. The experimental results demonstrate that SpecAR-Net achieves good performance in five mainstream tasks, including classification, anomaly detection, long-term forecasting, short-term forecasting and imputation. Our contributions are summarized in three folds:

1. A unified time-frequency joint representation framework is proposed. The framework decouples the features into three levels: global variation features (trend), local variation features (periodicity), and transient change features (mutagenicity), enabling more efficient deep semantic feature extraction for time series.
2. A plug-and-play time-series representation module, SpecAR-Block is proposed, which is compatible with various deep sequence modeling frameworks, e.g., RNNs, CNNs and Tranformers. By utilizing time frequency transformation and 2D multi-scale parallel complex convolutions, it can generate comprehensive semantic representation for input sequence.
3. A powerful deep sequential model with strong generalization ability is designed. SpecAR-Net has exhibited strong performance across a range of widely-used time series analysis tasks, e.g., anomaly detection, classification, long/short-term forecasting and imputation.

2 RelatedWork

Pure time-domain modeling. In essence, SpecAR-Net is a deep sequence modeling or encoding method. Initially, most of these methods were based on multi-layer perceptrons (MLP). For example, an extended MLP for predicting exchange rate trends using interval time series is presented in [Mat´e and Jime´nez, 2021]. LightTS [Zhang et al., 2022] introduced a fine-grained down-sampling strategy into an MLP and achieved excellent performance in long-term forecasting tasks. DLinear [Zeng, 2023] decomposed time series into trend and residual sequences and utilized two MLPs to model these sequences for forecasting tasks.

Then, as a method tailored for time series modeling, RNN was widely investigated. It utilizes a chain-like structure to simulate the dynamic behavior of time series, which helps extract temporal characteristics. Such as the long short-term memory (LSTM) model used in [Hochreiter and Schmidhuber, 1997]. And LSTNet proposed in [Lai et al., 2018b], which utilizes both CNNs and RNNs to extract short-term local dependencies between variables and explore long-term patterns in time series trends, respectively. More recently, LSSL [Albert Gu and Re., 2022] achieved effective modeling of long time series by parameterizing the continuous-time, recurrent, and convolutional views of the state space model.

Admittedly, RNNs are naturally suited for dealing with time series. However the risk of gradient vanishing/explosion and limitation of serial computing have obstacles for RNNs. In this context, CNNs are also favored. For example, dilated convolutions were utilized as an encoder to accept variablelength inputs for time series modeling [Bai et al., 2018]. TCN [Franceschi et al., 2019] employs multiple 1D convolutions to extract temporal information across different scales of feature maps, demonstrating certain advantages in extracting deep semantic features from time series. There's also research that indicates that CNNs exhibit superior performance to RNNs in time series modeling [Chen and Shi, 2021].

In recent years, Transformers have shown remarkable performance in the field of time series modeling [Nikita Kitaev and Levskaya, 2020]. By utilizing SA mechanisms, these methods possess inherent network architecture advantages in capturing temporal dependencies in time series. As a result, they have become popular approaches in the field of time series analysis. For instance, Informer [Zhou et al., 2021a] design ProbSparse SA mechanism and distillation operations to reduce the compuation complexity and memory consumption of the vanilla version. Inspired by the principle of exponential smoothing, ETSFormer [Woo et al., 2022] has been devised to improve the accuracy of time series prediction by using novel Exponential Smoothing Attention (ESA) and Frequency Attention (FA) mechanisms.

Frequency-guided modeling methods. These above methods provide many valuable ideas and practical tools for time series analysis. However, the modeling mechanisms are purely time-domain, which are difficult to describe and encode the highly coupled contents of the sequences, comprehensively. Considering such limitation, frequency information were incorporated into the deep models, which have achieved promising results, e.g., FEDformer [Zhou et al.,

2022] and TimesNet [Wu et al., 2023].

In order to use frequency information efficiently, FEDformer proposes to randomly select the frequency components, so that the following SA can automatically capture the important components from a compact frequency domain subspace, which maintains both high and low frequency components. Then the inverse Fourier transform is utilized to continue the time-domain modeling. FEDformer demonstrates the effectiveness of frequency component pre-filtering in modern deep backbones. Along this trail, TimesNet uses Fourier transform to locate the salient frequency components in the input series. According to these salient components, it reshapes the input as the time-period 2D representation, which helps the convolutions obtain more effective representation. Following this frequency component filtering mechanism, this work attempts to further tap the potential of the frequency-guided modeling in time series analysis. We propose to further enhance the expressive power of the backbone models by exploiting the time-varying patterns of the selected frequency components, which is ignored by existing methods.

3 Methodology

To establish a comprehensive unified representation for time series, this paper proposes SpecAR-Net from the perspective of joint time-frequency analysis. Firstly, Short Time Fourier Transform (STFT) is used for the mapping from time domain to time-frequency domain, resulting in a transform of data structure from 1D to 2D data space. Then, a group of multiscale parallel complex convolution blocks, which efficiently extracts and fuses time-frequency characteristics of the time series. Through this process, we achieve a unified representation of the time series in both time and frequency domains.

3.1 SpecAR-Block

As shown in Fig. 1, the backbone of SpecAR-Net is composed of several stacking SpecAR-Blocks. Concretely, given one time series sample, X ∈ RT×N , where T and N is time length and data dimension, respectively. A high-dimensional mapping of X is performed at the very beginning as

where X0 ∈ RT×M is the encoded features generated by the embedding layer Embed(·) : RN → RM , which consists of three components: position embedding, global time stamp embedding and scalar projection.

Then for the SpecAR-Net with L blocks, the l-th (l = 1, . . . , L) layer can be formalized as

where SpecAR(·) : RT×M → RT×M denotes the SpecAR time-frequency encoding process, the output Xl is calculated by SpecAR along with a short-cut connection of l-1-th layer.

As can be seen from the detailed part of SpecAR-Block on the below of Fig. 1, each block consists of three core modules: time-frequency transformation, multi-scale complex convolutions and feature aggregation. In specific, the time-frequency transformation (TFT) is performed to convert the temporal input features Xl-1 into time-frequency structured (i.e., spectrogram) complex tensor, Sl-1 ∈ CM×T×F , where F denotes the number of frequency bins. Then a group of parallel multi-scale 2D complex convolutions is used to encode the complex tensor. This process can be formalized as follows:

Where TFT(·) : RT →RT×F denotes a dimension/channelparallel time-frequency operator, which can be fulfilled by STFT (by default in this paper) or Wavelet Transform (WT). And MS-Conv† (·) denotes the parallel-computed complex convolutions with different dilation rates (sampling rates) in time-frequency receptive field of Sl-1 . Assume we have K different convolution blocks, then the output tensor will be in the form of S(l-1)† ∈ CM×T×F×K . More details of TFT(·) and MS-Conv† (·) are in Sec. 3.2 and 3.3.

Finally, for the feature aggregation stage, a block-wise average pooling is first conducted to compress the stacked feature tensor, S(l-1)† obtained fromMS-Conv† (·). Then a linear projection is used to transform the complex compressed features as real ones. This stage can be formalized as

Where, Avg Blk (·) : CK → C denotes the block-wise average pooling, Re/Im[·] is the element-wise real/complex part extractor, and Linear(·) denotes the complex-to-real linear projection. To further utilize the advantage of the skip connection, the fused time-frequency features, Xl† ∈ RM×T×F will be average-pooled along the frequency domain and transposed to get the shape-compatible output tensor in RT×M .

3.2 Time-Frequency Transformation

In order to decouple and analyze the periodic characteristics of time series while maintaining its temporal structure, we incorporate TFT in our SpecAR-Block. TFT can be fulfilled by the classic STFT or WT, which facilitates more efficient joint extraction of time-frequency features using 2D convolutions in subsequent learning stages. Following simplicity design principle, we use STFT by default. Fig. 2 illustrates the process of TFT for a given time series sample in an intuitive way.

Following the symbol definition above, for each channel of the given input sequence Xl ∈ RT×M for l + 1-th SpecARBlock, the TFT calculation is formalized as

where Slm ∈ CT×F is the discrete STFT results, i.e., spectrogram of m-th channel input sequence Xlm ∈ RT×1 . Both window size and number of sample points for FFT are 2n+1, the hop length is set as 1, so that the input and output can have the same time length. The commonly used Hamming window function h(t) = 0.54 - 0.46 cos ( πt n ) is employed to modulate the input sequence. Moreover, it is also used Hanning window, Rectangular window and Blackman window, etc.

After applying the TFT to the time series, the highly coupled pure time domain data is expanded into a time-frequency representation. This transformation will enable the subsequent feature learning part to more intuitively analyze the periodic components of the input sequence and their respective evolution trends on the time-frequency distribution. However, it should be noted that the spectrogram actually carries not only the amplitude but also the phase information in each time-frequency unit, these are retained in the complex numbers. Hence, to fully utilize those contents, special treatment should be taken into consideration in the subsequent learning stage, which is discussed in the following section. Beyond that, as a crucial role in accurately representing frequencydomain features, window length determines the frequency resolution of the resulting spectrogram. Therefore, the effect of window length is also investigated in the experiments.

3.3 Multi-Scale Complex Convolutions

To make full use of the information in the spectrogram representation, i.e., the phase and the amplitude, the complex convolutions are utilized in SpecAR-Net. In addition, multiscale kernels for parallel convolutions are introduced to alleviate the contradiction of time-frequency resolution of TFT.

In order to avoid introducing more learning parameters, we use different dilation rates to achieve multi-scale feature extraction. Then the small network is designed to be constructed with K complex convolution blocks with different dilation rates but the same kernel size, 3 × 3. Given the input timefrequency tensor of l-th SpecAR-Block, Sl ∈ CM×T×F , The forward computation process can be roughly expressed as

Where Conv†k (·; d[k]) : CM → CM denotes the k-th convolution block with dilation rate of d[k] = 2k + 1. The specific calculation process for each Conv†k is as follows: assuming a complex convolution kernel w = (a + j ⊙ b) , and a complex input tensor h = (c + j ⊙ d), the complex convolution process, denoted as follows:

Finally, the ouput tensors of all the complex convolution blocks will be stacked together in block-wise to form the multi-scale feature tensor Sl† ∈ CM×T×F×K .

3.4 Temporal Order Preserving

To capture the global trend patterns of the input time series, the temporal order preserving (TOP) constraint is incorporated into our SpecAR-Net. This constraint is achieved by adding an order regression loss term on the basis of the original prediction loss. In practice, we use a temporal-shared learning function to construct such TOP loss term.

Given the final embeddings of SpecAR-Net, X∗ ∈ RT×M for an input sequence. The learning function Φ(·; u) : RM → R will encode each X∗ [t] as follows:

Where u is the temporal-shared learning parameters and t = 1, . . . , T is the time index of X∗ [t]. Based on this order regression mechanism, the TOP loss term for the current input sequence can be formalized as

Where LTOP denotes the TOP loss term, λ is the order regression penalty factor. And R(u) is the regularization term for loss LTOP, which can be written as the L1 or L2 norm of u, and we choose L2 in practice, i.e., R(u) = 12 ∥u∥22 . The loss can be expressed as the weighted summation of LTOP and the original loss for the current learning task, e.g., mean square error for forecasting or cross-entropy for classification.

4 Experiments

To verify the effectiveness and superiority of SpecAR-Net, a comprehensive set of experiments is conducted over 5 mainstream tasks, i.e., classification, anomaly detection, long-term forecasting, short-term forecasting and imputation. The benchmark datasets and corresponding experimental configurations are shown in Tab. 1. The backbones of the compared state-of-the-art (SoTA) models including RNNs (LSTM, LSTNet, LSSL), CNNs (TCN, TimesNet), MLPs (LightTS, DLinear) and Transformers (Autoformer [Wu et al., 2021], FEDformer, Informer, Reformer [Nikita Kitaev and Levskaya, 2020], Pyraformer [Liu et al., 2022a], ETSformer, Non-stationary Transformer [Liu et al., 2022b]). Furthermore, for specific tasks, cutting-edge models are also mentioned in the SoTA comparison this experiment. Specifically, N-HiTS [Challu et al., 2023] and N-BEATS [Oreshkin et al., 2019] are compared in short-term forecasting. Transformer [Xu et al., 2022] is selected for comparison in anomaly detection. For classification, Rocket [Dempster et al., 2020] and Flowformer [Wu et al., 2022] are compared.

4.1 Results onMainstreamTasks

Compared to other baseline methods, SpecAR-Net has achieved the best performance across all 5 tasks, as shown in Tab. 2 (where red and blue font denotes the best and secondbest results, respectively. ∗ in the Transformers indicates the name of ∗former.). Additionally, the results further validate the good generalization ability of SpecAR-Net, which can be regarded as a unified framework in time series analysis.

Classification (CLA) task can intuitively show the performance of our method in terms of high-level semantic representation of time series. The data used in this experiment is sourced from the UAE dataset (10 subsets) [Bagnall et al., 2018], comprising ten sub-datasets that encompass practical tasks such as gesture recognition, action recognition, audio recognition, and medical diagnosis. As shown in Fig. 3, SpecAR-Net has achieved remarkable results in classification task, with an average classification accuracy of 74.7%, surpassing other SoTA methods such as TimesNet (73.6%) and Flowformer (73%). It is worth noting that, compared to SpecAR-Net, TimesNet exhibits lower classification accuracy on most datasets, with an average accuracy reduction of 1.1%. Feature extraction can be conducted simultaneously in both time- and frequency-domains by SpecAR-Net, facilitating the capture of higher-level semantic representations.

Anomaly Detection (AD) plays a vital role in ensuring the orderly and secure operation of industrial production. However, anomaly detection often requires capturing exceptional signal within big data, which can easily get overwhelmed, making the detection task highly challenging. To fully validate the performance is such task, 5 widely-used datasets are employed, i.e., SMD [Su et al., 2019], MSL and SMAP [Hundman et al., 2018], SWaT [Mathur and Tippenhauer,

2016], and PSM [Abdulaal et al., 2021]. These datasets cover various real-world industrial applications, including service monitoring, spatial and earth sensing, and water treatment. The results are presented in Tab. 3. It is evident that our method achieved the optimal performance in the anomaly detection task, outperforming other comparative methods. The advanced Transformer-based approaches, FEDformer and Autoformer, have also both achieved good performance (84.97% and 84.26%). Frequency-domain information is introduced into the attention mechanism of both models, further highlighting the effectiveness of such information in time series representation. Comparatively, our method enables the joint extraction of deep-level time-frequency features from both the time- and frequency-domains, thereby facilitating the capture of abnormal patterns existing in time series.

Forecasting. In the long-term forecasting (LF) task, a set of benchmark datasets were utilized, including ETT [Zhou et al., 2021b], Electricity, Traffic, Weather, ExchangeRate [Lai et al., 2018a] and ILI (see download links in Appendix), which cover the application demands of 5 major real-world scenarios. Each dataset contains a segment of continuous time series, and sample data are obtained from these datasets using a sliding window approach. In the experiments, the input past length was set to 96, with ILL for 36. The prediction lengths is [96, 192, 336, 720], with ILI for [24, 36, 48, 60]. In the short-term forecasting (SF) task, we utilized the M4 dataset [Makridakis et al., 2018], which comprises 100,000 time series . These data were collected at different sampling rates, including yearly, quarterly, monthly, weekly, daily, and hourly intervals, covering a wide range of domains such as finance, industry, and demographics. For our experiments, the prediction sequence lengths is [6, 8, 13, 16, 24, 48]. Especially, all the results are averaged from four different prediction lengths for long-term forecasting, the results of shortterm forecasting tasks are calculated as weighted averages from multiple datasets with varying sample intervals. The experiments are conducted in two rounds in total. In the first round, MSE is used as loss function, and it achieved good results in both short-term and long-term forecasting tasks, although it did not reach the optimal level. See Tab.7 and 9 in Appendix for more details. In the second round, SpecAR-Net was conducted by introducing a order-preserving into the loss function. As shown in Tab. 4 and 5, our method achieves the best performance in both long- and short-term forecasting, indicating a positive role of the "order" information in time series forecasting. The order-preserving is equivalent to using the "order" information as prior knowledge to constrain the learning process of the model and compensate for the lost "order" information during feature extraction, ensuring that the model output possess a certain degree of sequentiality. Meanwhile, it also shows that our method is highly scalable.

Imputation (IMP) task relies on historical data to recover the missing data. This technique serves as the foundation of big data analytics, ensuring the temporal and spatial integrity of time series, thus supporting various subsequent tasks such as forecasting, classification, and anomaly detection. This experiment was conducted on 6 benchmark datasets, including ETT (4 subsets), Electricity and Weather. Random masking with masking rates of [12.5%, 25%, 37.5%, 50%] was used to simulate missing values. The experiment was conducted in two rounds. The first round of the experiment was conducted without order-preserving. And SpecAR-Net exhibits consistent performance with TimesNet, which is the best-performing method among the comparison methods. See Tab.11 in Appendix for more details. Tab. 6 presents the experimental results (average from 4 different mask ratios) after incorporating order-preserving, where SpecARNet achieves the best performance. This indicates that the monotonicity constraint is beneficial for capturing the global trend patterns in time series. Moreover, it also suggests that SpecAR-Net possesses strong capabilities in extracting timeand frequency-varying patterns.

4.2 Detailed Analysis

Model Complexity & Performance. To further analyze the performance of SpecAR-Net in the representation of time series, we selected comparable models with better performance in classification and forecasting tasks for model complexity analysis. Results in Fig. 4.2 show that better performance can be obtained by our method in the condition of less learnable parameters. This further illustrates the superiority of the proposed time-frequency joint learning mechanism.

Effects of TF Resolution. The window length directly affects the time- and frequency-resolution of STFT, which reflect the richness of information in the time and frequency domains, which has a significant impact on extracting timefrequency variation. Therefore, in this experiment, different window lengths of [4, 8, 16, 24, 48, 96, 192, 336] were selected to investigate their effects on the model performance. Fig. 5 demonstrates that SpecAR-Net achieves optimal performance when the prediction lengths are [96, 192, 336, 720], corresponding to window lengths with [4, 24, 192, 192]. This finding indicates that the requirements for time-frequency resolution vary across different temporal analysis tasks, suggesting a varying dependency on both time- and frequencyfeatures. According to the Heisenberg uncertainty principle [Mallet and others, 1999], it is impossible for the timeand frequency-resolution to simultaneously reach their optimal values. In order to ensure that our model has good time series representation capability while maintaining a suitable computational complexity, window lengths are set as [8, 16, 24] in this paper.

5 Conclusions

SpecAR-Net can be used as a universal foundational model for time-frequency representation and analysis of time series. Through the time-frequency transformation, SpecARNet overcomes the limitations of semantic representation in 1D time series caused by the coupling of multiple components such as trending, periodicity, and abruptness. This facilitates the simultaneous extraction and fusion of timefrequency variation patterns from a 2D space. Experimental results demonstrate that SpecAR-Net achieves optimal performance in 5 tasks, including classification, anomaly detection, imputation, long- and short-term forecasting.

Acknowledgements

This work is supported by National Natural Science Foundation of China (No. 62206258).

Contribution Statement

Yi Dong∗ , Liwen Zhang∗ and Youcheng Zhang∗ are authors of equal contributions: Yi contributed code implementation, experimental operation and the draft of the paper; Liwen contributed idea, algorithm design, and writing of introduction, method and rebuttal; Youcheng contributed experiment design, paper structure and paper polishing. Zhe Ma† is the corresponding author.