Learning to Extrapolate and Adjust: Two-Stage Meta-Learning for Concept Drift in Online Time Series Forecasting

1 Introduction

Accurate time series forecasting [Hyndman and Athanasopoulos2018, Benidis et al.2022] is of great importance in many domains, such as stock market [Cavalcante and Oliveira2014, Shahi et al.2020], weather [Bi et al.2023], energy [Yang et al.2023, Hong et al.2020], etc. The nonstationary nature of the times series data in many scenarios makes the model trained on the historical data outdated and leads to unsatisfying forecasting on the new data [Liu et al.2023b,Pham et al.2023]. Thus, there is a growing interest in online time series forecasting, where the deployed model can swiftly adapt to non-stationary environments over time.

In this paper, we address the challenge of concept drift in online time series forecasting, a critical issue that arises when the underlying data distribution or environment evolves continuously over time. The problem of concept drift has drawn considerable attention in the literature, as reviewed in [Lu et al.2019]. Traditionally, researchers have classified these drifts based on observed data patterns. For example, [Liu et al.2023b] categorizes concept drift into real and virtual drifts, while other frameworks classify drifts as sudden, incremental, gradual, or recurring [Lu et al.2019]. These categorizations have historically been useful in distinguishing and understanding different drift patterns in classification problems [Lu et al.2019]. However, in time series forecasting, concept drifts often overlap and evolve within and across periods. For instance, electric consumption may slowly increase year over year due to gradual population growth and economic factors, while abrupt changes might occur due to factors such as changes in electricity pricing and holidays within the same annual cycle. These complex drifts are typically interwoven, making it challenging to classify them into existing categories.

Motivated by these observations, we propose a novel perspective on mitigating concept drift by distinguishing between intra-period and inter-period drifts, which we term macrodrift and micro-drift, respectively. As depicted in Figure 1, macro-drift represents stable drifts that endure over relatively long durations and across periods, where evolving trends may be predictable-for instance, the demographic and economic factors contributing to the monthly increase in traffic volume at a crossroad. Conversely, micro-drift involves transient, sample-level changes occurring within the broader context of macro-drift, characterized by erratic and unpredictable variations from sample to sample. Instances of this include sudden traffic surges during rainy days or holidays that fluctuate daily within the same month.

The different characteristics of the drifts calls for different learning strategies. Specifically, to address a stable, predictable, long-termmacro-drift, we cannot simply apply the current optimal model to future data, as the drift will carry on, altering future data patterns. Therefore, we must investigate the evolution of models over time and endeavor to predict the model to align with subsequent data. Most existing works design different strategies, e.g., historical information retrieval [Arik et al.2022] and fast learning [Pham et al.2023], to capture concept drifts within recent observed data failing to adaptively update the model based on the influence range of drifts and ignoring to adapt the model to the future data which limits the generalization ability. On the other hand, addressing micro-drifts, characterized by erratic and transient samplelevel changes within the same period based on macro-drift, requires a more tailored approach. Close monitoring of data at the sample level is crucial, necessitating real-time adjustments to the model for each sample based on its divergence from the current context. This meticulous monitoring allows for the effective capture of sample-specific changes, ensuring that the model remains responsive to rapid drifts within the period. In this context, while numerous forecasting algorithms have been proposed [Benidis et al.2022], there remains a pressing need for a general and versatile model-agnostic approach capable of effectively managing both macro-drifts and micro-drifts across diverse prediction models.

In this paper, we tackle the concept drift problem in online time series forecasting by addressing both macro-drifts and micro-drifts through a unified meta-learning framework named LEAF (Learning to Extrapolate and Adjust for Forecasting). Designed to be model-agnostic, LEAF is compatible with a variety of deep prediction models including MLPs, RNNs, CNNs, and Transformer-based models. By learning a lowdimensional latent embedding of the model parameters, LEAF adeptly navigates the model dynamics in response to concept drift, optimizing in a reduced space to both capture drift characteristics and mitigate over-fitting. The framework is articulated in two stages: an extrapolation stage to address stable, long-termmacro-drifts by learning and predicting their evolving patterns, and an adjustment stage that responds to unpredictable, short-term micro-drifts by adjusting model parameters with a sample-specific surrogate loss. Note that macro-drift typically corresponds to the long-term dynamics of time series, e.g., the dynamics of the trend or the seasonal component. Unlike the naive seasonal-trend decomposition which models these components directly, in the extrapolation stage we model it by learning a more powerful and adaptive meta-learned extrapolation network. This approach not only helps to capture evolving trends and dynamics but also enables real-time adjustment to abrupt changes, offering a comprehensive solution to handling concept drift in online forecasting scenarios. Our theoretical analysis, supported by empirical evidence from real-world applications, underscores the effectiveness of LEAF in improving forecasting accuracy in the presence of concept drift. We test the LEAF framework on various benchmark datasets extensively. It is incorporated into different deep learning algorithms including MLP, RNN, CNN and Transfornmer-based models. The experimental results verify its universal effectiveness and robustness across various scenarios and different forecasting algorithms 1 .

2 RelatedWork

Following the early success of LSTMmethods [Sutskever et al.2014, Salinas et al.2020] and CNN-based methods (e.g., TCN [Bai et al.2018]), Transformer-based models have surged to the forefront [Wen et al.2023] in forecasting, including LogTrans [Li et al.2019], Reformer [Kitaev et al.2020], Informer [Zhou et al.2021], Autoformer [Wu et al.2021], FEDformer [Zhou et al.2022], Quatformer [Chen et al.2022], etc. Some recent developments with SOTA performance include DLinear [Zeng et al.2023], PatchTST [Nie et al.2023], and TimesNet [Wu et al.2022]. Unfortunately, all these deep learning algorithms cannot handle concept drift quite well.

A common approach to address concept drift involves optimization-based meta-learning or model-based metalearning [Huisman et al.2021]. For example, DeepTime [Woo et al.2022] treats different lookback windows and forecast horizons as tasks, learning mappings from time indices to values that generalize effectively. [You et al.2021] treats historical and future data as tasks and utilizes Model-Agnostic MetaLearning (MAML) [Finn et al.2017] to adapt the model's parameters. Techniques initially developed for sequential domain adaptation and generalization, such as DDG-DA [Li et al.2022] and TDG [Liu et al.2023a], are also applicable to online time series forecasting, where they adapt the model to changing data distributions in various ways. In addition to meta-learning approaches, self-adaptive methods [Arik et al.2022] employ backcasting self-supervised objectives to enhance a model's adaptability to future distributions during prediction. FSNet [Pham et al.2023] introduces per-layer adapters and associative memory to dynamically adjust layer outputs and facilitate the learning of evolving patterns. SOLID [Chen et al.2024] proposes a detection and adaptation framework, which first quantifies model's susceptibility to the distribution shift, and then adapts the model with historical data sharing similar contexts on the sample-level. RevIN [Nie et al.2023] is a simple yet effective model-agnostic approach, which normalizes input samples before modeling and then reverses the process after making predictions. While these techniques collectively offer rich strategies to mitigate the challenges posed by concept drift, they fall short in adequately considering the specialties of concept drift in time series data.

1 Code, data and supplementary of the paper are available at https: //github.com/vc12301/LEAF

3 Methodologies

3.1 ProblemDefinition: Online Time Series Forecasting via Meta-Learning

Time series forecasting. A time series is an ordered sequence of observations denoted as Z = (z1, z2, · · · , zT ) ∈ RT×c , where T is the total number of time steps, zi ∈ Rc , and c is dimensionality. Time series forecasting aims at learning a prediction model fθ to predict the next O-steps at time t given a look-back window of length I as zt-I+1:t f (·;θ) ---→ zt+1:t+O, where θ is the model parameter. For simplicity, in the remainder of the paper, we denote the input zt-I+1:t and the output zt+1:t+O as x and y and a set of input and output as X and Y, respectively.

Online time series forecasting. Time series data often arrives in a streaming fashion with frequent concept drift. An appropriate approach is to continuously update the model over a sequence of periods. At each online forecasting period p, the model accesses and learns from the most recent data available and makes predictions for the future. Subsequently, the ground truth is revealed, allowing for updating the model once again. This cyclic process ensures that the model remains up-to-date and adapts to the changing nature of the time series.

Meta-learning. Note that simply fitting a prediction model to the most recent observed data is insufficient due to concept drift, and the model needs to effectively generalize to future data. Meta-learning, also known as learning to learn, is a machine learning approach where a model learns how to learn from experience or previous tasks, enabling it to quickly adapt and generalize to new tasks. In the context of online time series forecasting, our objective is to learn how to adapt to future data. Our meta-learning based LEAF algorithm is trained over a sequence of online forecasting periods, referred as the meta-training phase, and evaluated over another sequence of periods called the meta-test phase. The procedure is depicted in Figure 2.

3.2 Learning to Extrapolate and Adjust

Model Overview. Formally, at each online forecasting period p, we have a training set Dp train = {Xp,Yp} and a test set Dp test = { ˜Xp, ˜Yp}. Without loss of generality, we assume that both the training set and the test set contain B samples. Specifically, Xp = {x(i) p }Bi=1 and ˜Xp = {˜x(i) p }Bi=1 . The main objective of LEAF is to leverage knowledge gained from historical periods and generate model parameters to make accurate forecasts on Dp test in the presence of concept drift. To achieve this goal, LEAF learns two functions: extrapolation E(·; ϕe) and adjustment A(·; ϕa), addressing macro- and micro-drift, respectively, to generate model parameters θp at period p. The objective can be formulated as:

where the extrapolation function E(·;ϕe) is used to anticipate the macro-drift which takes as the optimal parameters θ∗p-k:p-1 from the previous k periods as input and infer parameters θp for period p, the adjustment function performs sample-specific parameter adjustment considering micro-drift within each sample, and L(·) is the prediction loss. Figure 3(b) illustrates the framework of LEAF.

Learning to Extrapolate in Latent Space

During the extrapolation stage, the meta-model LEAF aims at inferring a prediction model based on previously optimal models. To handle the possible concept drift especially macrodrift, we need to update the model effectively. In practice, we observe that often only a subset of patterns change in the concept while the majority remain invariant. In other words, we do not need to update all model parameters. To elegantly model it, we introduce a latent embedding H, which can be decoded to the model parameter θ, and the extrapolation is performed on this latent space.

In period p, we have Dp train = {Xp,Yp} and Dp test = { ˜Xp, ˜Yp}. In particular, Dp train is identical to the test data Dp-1 test , as shown in Figure 2. In other words, if we optimize the prediction model on Dp train , we can obtain the optimal model at period p - 1 in the latent space as follows:

where SGD represents gradient descent on Dp train with respect to prediction loss. This optimization process leads to the optimal latent embeddingH∗p-1 . Subsequently, we introduce an extrapolation network NN(·) that infers Hp based on the previous k optimal latent embeddings H∗p-k:p-1 and a decoder to generate model parameters, which can be formulated as:

Moving forward period p + 1, Dp test (or Dp+1 train ) is revealed. The above steps can be repeated to obtain H∗p , Hp+1 and θp+1. NN(·) can be any sequential model, e.g., LSTM and Transformer. We utilize a simple LSTM here as default to verify the capability of the framework itself. Besides, we investigate the network selection of the extrapolation network (see Supplementary A.6).

Learning to Adjust via Surrogate Loss

As the historical and forecast horizon of time series tend to coexist within an identical domain, it is quite beneficial to performmodel adaptation to capture the micro-drift at inference (test) time according to the input ˜X. To this end, we introduce a meta-learnable surrogate loss that adjusts or modulates the latent embedding Hp for each sample {˜x(i) p }Bi=1 in Dp test to obtain sample-specific embeddings {H(i) p }Bi=1 . This surrogate loss is implemented using neural networks and applied during the test phase, enabling the adjustment of the latent embedding based on sample-specific characteristics.

The rationale behind this approach is that micro-drift is often caused by external events, such as weather conditions or holidays. Although data affected by micro-drift may exhibit different patterns, they tend to deviate from their context in a similar manner. For example, the electricity consumption pattern during the summer typically differs from that in the winter, while during holidays it tends to decrease regardless of the season. By incorporating a meta-learnable surrogate loss, we aim to capture and adjust for these sample-specific deviations caused by micro-drift. Specifically, during the meta-training phase, the surrogate loss is learned to capture and compensate for different types of deviations. During the meta-testing phase, the model utilizes the learned surrogate loss to adjust the latent embedding based on the specific characteristics of each individual sample. To achieve this, the loss network takes into account the following three factors to capture samplespecific micro-drift: 1) Sample and base prediction.The sample itself ˜x(i) p and the base prediction with inferred model parameters in extrapolation stage f(˜x(i) p ; θp) are introduced as basic characteristics; 2) Base latent embedding. As we optimize the model in a latent space, the inferred latent embedding in the extrapolation stage Hp is passed to the loss network, providing the contextual information and underlying pattern; 3) Sample's relationship to training data. Furthermore, we introduce a relation network to account for sample's relationship to its context. The relation network R(·) receives embeddings of the sample ˜x(i) p and training set Dp train and returns a vector representing their relationship:

where g(·) is an embedding function, and the training data embedding is computed as the mean pooling of embeddings of all samples in the training set. The relation network captures the similarities or dissimilarities in their patterns, which is valuable in capturing the micro-drift. To sum up, the final loss network s(·) are defined as

The surrogate loss guides the adjustment of the latent embedding for each sample using gradient descent, which can be further decoded to obtain sample-specific model parameters:

where α is the learning rate, and H(i) p and θ(i) p stand for sample-specific latent embedding and model parameters, respectively. Finally, the parameters are loaded to make forecasts with f(˜x(i) p ); θ(i) p ).

Model Learning

In this subsection, we outline the training procedure of LEAF, as depicted in Algorithm 1. We denote the parameters of extrapolation module and adjustment module as ϕe and ϕa, respectively. Additionally, ω represents the parameter of the decoder.

At each period p, as shown in Figure 3(b), after extrapolation and adjustment stages (the "inner loop"), we make predictions on Dp test. The test set is then utilized to update the parameters of meta-learners ϕa, ϕe, ω (the "outer loop"). More precisely, the optimization process of "outer loop" is performed by minimizing the following objective:

where the first term is the prediction loss on the test set, the second term regularizes the extrapolation in Eq. (3) to output

Algorithm 1: Training Procedure of LEAF

Input: Number of meta-training periods P , data {Dp train , Dp test }Pp=1 , learning rates η, α, µ, number of historical periods in extrapolation stage k, prediction model f(·) 1 Randomly initialize the parameters ϕe , ϕa , and ω of LEAF ; 2 Let ψ = {ϕe ,ϕa ,ω} ; 3 Randomly initialize the latent embeddingH0 ; 4 Create a queue Q of size k storing the recent k optimal latent embeddings; 5 Initialize Q as Q = [H0, · · · ,H0] ; 6 for p = 1, 2, · · · , P do 7 Evaluate inner-loop prediction loss Ltrain p on training dataset Dp train ; 8 Perform gradient descent H∗p-1 = Hp-1 - η∇Hp-1 Ltrain p ; 9 Q.Deque().Enque(H∗p-1 .detach()) 10 ComputeHp and θp using Eq. (3); /* Extrapolation stage */ 11 for ˜x(i) p in ˜Xp do 12 /* Adjustment stage: traverse all test inputs, and conduct sample-specific adjustment */ 13 Compute Lsurrogate (˜x(i) p ) using Eq. (5); 14 H(i) p =Hp -α∇Hp Lsurrogate (˜x(i) p ); 15 θ(i) p = Decoder(H(i) p ), and load θ(i) p into f(·); 16 Evaluate prediction loss Ltest p,i w.r.t. ˜x(i) p ; 17 end 18 Compute LLEAF using Eq. (7); 19 Perform gradient descent ψ = ψ - µ∇ψLLEAF ; 20 end

latent embedding that is close to the optimal latent embedding, and γ is the coefficient of the regularization.

Recall that online forecasting often involves a warm-up phase. Since we optimize the parameters of the prediction model in the latent space, the traditional training strategy is not directly applicable. To address this issue, we randomly generate a latent embedding Hinit, which is then decoded into parameters of prediction model using the decoder. During the training procedure of the warm-up phase, Hinit remains fixed and the decoder is trained using the prediction loss on warmup dataset. Subsequently, at the onset of the meta-learning, we initialize H0 with Hinit and ω with the learned decoder parameter from warm-up phase. Figure 3(a) illustrates the warm-up strategy.

Remarks. LEAF generates the parameters for the last few layers of the target prediction model. It is important to note that LEAF is a model-agnostic framework, allowing different types of layers. For instance, when using DLinear as the target prediction model, we could generate the parameters for the linear layer. In the case of PatchTST, we could generate the parameters for the last transformer block, which includes the Query/Key/Value projection networks and a feedforward network. To apply LEAF to different types of layers, we require the parameter shapes of the network layers. By flattening the parameters to be generated, we can determine the width of the decoder in LEAF. The generated parameters are then appropriately reshaped and loaded into the respective layers, allowing for flexibility and compatibility across different network architectures.

3.3 Theoretical Analysis

In this section, we provide a theoretical analysis for the simplified scenarios to justify our LEAF framework, where we assume a linear model ˆy =Wx for the forecasting task. During each online forecasting period p, the dataset {Xp,Yp}

is drawn from Pp, and we define that feature-target pair z = [x; y]. Assuming that we need to forecast at the current period P + 1, with optimal parameters obtained from the preceding periods, and our objective is to learn weights WP+1 that generalizes well at period P +1. In the simplified analysis, we introduce two mild conditions to characterize maco-drift and micro-drift.

Assumption 1 (Macro-drift). Pp+1, the data distribution for period P + 1, is a composite of Pp and an unknown distribution P∆ that accounts for macro-drift and is independent of Pp.

Assumption 2 (Micro-drift). A transformed sample z′ = f (z, P)+ϵ embodies micro-drift, and f (z, P) = z+αΣ- 12 (z-µ) is a deterministic function with unknown scalar α responsible for the incremental change within each sample considering its distribution context; µ and Σ stand for mean and covariance of P; ϵ follows a normal distribution N(0, diag(σ2 )) denoting the stochastic noise. We denote this data distribution as P ′ .

We summarize the theoretical analysis results by the following two propositions, with proofs provided in the Supplementary A.1.

Proposition 1. Given the complete datasets from preceding periods {Xp,Yp}Pp=1 , the associated optimal weights {W∗ p }Pp=1 and the input data {XP+1} for period P + 1. We approximate weights for period P + 1 as

This estimation exhibits much more generalization than the estimation with the last period:

Proposition 2. Considering the micro-drift, We possess the dataset {X′p,Y′p}Pp=1 , the associated optimal weights {W∗ p }Pp=1 and the input data {X′P+1} for period P + 1. A sample-level weight adjustment achieves less prediction error than weight estimation in Proposition 1:

Proposition 2 states that introducing a sample-dependent weight modification can alleviate the micro-drift. It implies that performing the sample-specific weight adjustment can reduce the prediction error in the context of sample-level microdrift.

Although our theoretical framework focuses on simplified linear models, the insights are highly relevant to real-world applications. This relevance is underscored by the fact that in numerous practical scenarios, only the parameters in the final linear layer of deep neural networks are updated, aligning with our theoretical approach. Moreover, even for nonlinear layers, practical evidence supports the effectiveness of our extrapolation and adjustment principles, which is supported in Table 3 in Supplementary where LEAF is applied to the last linear layer of a complex deep model.

4 Experiments

In this section, we conduct experiments to answer two questions: (RQ1) Can LEAF outperform SOTA model-agnostic concept drift adaptation methods in online time series forecasting scenarios? (RQ2) How do different components of LEAF contribute to resolving concept drift problems?

4.1 Experimental Settings

Datasets. We evaluate our method on six time series forecasting datasets. (1) ETT-small 2 [Zhou et al.2021] dataset contains observations of oil temperature along with six power load features over two years. For ETTh2, the model update interval (number of timesteps in each Dtrain) is 288, the lookback window is 96 and the forecast horizon is 24. For ETTm1, the model update interval is 672, the look-back window is 288 and the forecast horizon is 24. (2) ECL 3 dataset records hourly electricity consumption of 321 users over three years. We randomly sample 12 users. The model update interval is 288, the look-back window is 96 and the forecast horizon is 24. (3) Load dataset contains three real-world univariate electric load benchmarks in different types of areas at 15-minute intervals from 2020 to 2022. Figure 4 and Figure 5 ( in Supplementary A.2) visually demonstrate the presence of concept drift within each benchmark, including various types of macro and micro drifts. More details of the dataset are included in Supplementary A.2. Results with different forecast horizons

2 https://github.com/zhouhaoyi/ETDataset

3 https://archive.ics.uci.edu/dataset/321/ electricityloaddiagrams20112014

(ranging from 48 to 336) are shown in Table 5. Results on more datasets (NASDAQ-100 and WTH from economic and meteorological domains, respectively) are included in Supplementary A.3 and case studies on the forecast abilies of LEAF under concept drift are included in Supplementary A.8.

Baselines. We compare our method with multiple modelagnostic baselines, including: (1) Na¨ıve: it only warms-up the model and then freezes henceforth; (2) Na¨ıve†: it trains on warm-up and meta-train dataset and then freezes at meta-test; (3) Retrain: it updates the last few layers of backbone model at each period by gradient descent using all available data; (4) Fine-tune: it updates the last few layers of backbone model at each period by gradient descent using only the data in the period; (5) ER [Chaudhry et al.2019]: it employs a memory bank of most recent samples; (6) DER++ [Buzzega et al.2020]: a variate of ER with knowledge distillation; and (7) SOLID [Chen et al.2024]: a detection and adaptation framework that first quantifies model's susceptibility to distribution shift and then adapts the model with historical data sharing similar contexts on the sample-level. Since the original work employs an offline approach, we implement a variant SOLID* that continually fine-tunes the model parameters for fair comparison. We also compare our method with FSNet [Pham et al.2023], a recent SOTA for concept drift in online time series forecasting that uses TCN [Bai et al.2018] as the backbone and incorporates an experience replay. More details of baselines are included in Supplementary A.5.

Implementation Details. For all benchmarks, we split the data into warm-up/meta-train/meta-test by the ratio of 0.1/0.6/0.3. In warm-up phase, we use Adam [Kingma and Ba2014] with fixed learning rate of 0.001 to optimize the prediction model wrt the mean squared loss. The warm-up epoch is 10 and warm-up batch size is 32. In meta-training and meta-testing phases, at each period p, we use Adam with learning rate of 0.001 to obtain the optimal latent embedding and update parameters of meta-learners, the update epoch is 50 and 1, respectively. We implement Decoder, relation network R(·), and loss network s(·) with MLPs. We perform crossvalidation on meta-train to select appropriate hyper-parameters for all methods. We notice that LEAF is able to achieve competitive results with minor hyperparameter tuning. We report the hyperparameter sensitivity in Supplementary A.6. Besides, all baselines are well-tuned using Bayesian optimization from Neural Network Intelligence toolkit 4 . All experimental results are the average of the five independent trials with different random seeds.

4.2 Performance Comparison (RQ1)

The results of comparison performance are shown in Table 1 and the standard deviations are shown in Table 11. We apply model-agnostic methods to five types of prediction models, including LSTM [Hochreiter and Schmidhuber1997], DLinear [Zeng et al.2023], PatchTST [Nie et al.2023], TimesNet [Wu et al.2022] and TCN [Bai et al.2018]. We place FSNet [Pham et al.2023] in TCN category since it employs a TCN-based backbone. More details of backbone prediction models are included in Supplementary A.5. We use mean squared error (MSE) and mean absolute error (MAE) as evaluation metrics and they are averaged over meta-test periods. From Table 1, we can observe that: (1) our proposed LEAF significantly improves forecast performance in almost all cases, especially in Load benchmarks, for different types of prediction models. Overall, LEAF has a 14.4% reduction in error compared with second-best method. Especially, LEAF has an average decrease of 35.4% in MSE when using LSTM as prediction model, 29.5% when using TimesNet; (2) LEAF performs surprisingly well on Load-1, Load-2, and ETTm1 datasets, while the advantage is not so obvious on ECL dataset. This is because time series from Load-1, Load-2, and ETTm1 datasets exhibit much stronger and more complex concept drift; (3) Retrain, ER and DER++ show competitive performance in ECL dataset, as time series in this dataset holds strong seasonality and recurring patterns. These methods incorporate a memory of samples which helps alleviate catastrophic forgetting and remembering recurring patterns in history. We note here ER and its variant are orthogonal to our work which can be embedded easily into LEAF. Moreover, we report the running time complexity of different fine-tune methods in Tables 7, 8 and 9, wherein LEAF is not significantly more computationally expensive than Fine-tune, despite its superior performance compared to Fine-tune. Besides, sample-specific adjustment does not significantly contribute to the computational burden, as the adjustment is performed on latent space for one iteration. The complete running time comparison experiments are in Supplementary A.4.

4 https://nni.readthedocs.io/en/stable

4.3 Ablation Studies (RQ2)

We conduct ablation studies to validate the effectiveness of extrapolation and adjustment modules. We evaluate the performance of five variants of LEAF upon DLinear and PatchTST. Considering that LEAF can be treated as an advanced method of fine-tuning, we begin with the most basic fine-tuning method and gradually incorporate designed modules in LEAF to construct thevariants. The variants include: (1) Fine-tune that is foundation of LEAF and is performed in the parameter space; (2) Latent fine-tune performs fine-tuning in the latent space that uses an average of last five H∗ instead of the extrapolated latent embedding H; (3) EN that introduces the extrapolation module; (4) Latent fine-tune + A that incorporates the adjustment stage; and (5) EN + A that incorporates the adjustment stage on extrapolation module and is identical to standard LEAF.

The results are shown in Table 2. We observe first that fine-tuning model in the latent space (Latent fine-tune) can significantly improve the forecast performance in almost all benchmarks. This outcome verifies that optimizing model in a low-dimensional latent space is rational. Furthermore, EN introducing our proposed extrapolation module surpasses the performance of Latent fine-tune, thereby confirming its effectiveness in extrapolating the macro-drift. Lastly, the inclusion of the sample-specific adjustment yields a further enhancement in predictive performance, demonstrating the effectiveness of this stage in alleviating micro-drift. Moreover, we plot prediction results of different variants in Figure 7 (see Supplementary A.7).

5 Conclusion and FutureWork

In conclusion, our proposed LEAF framework offers a promising solution to the concept drift problem in online time series forecasting. By integrating meta-learning, LEAF enhances the capabilities of deep prediction models by acquiring the ability to extrapolate and adapt to macro- and micro-drift, respectively. This model-agnostic framework can be applied to various deep prediction models, making it versatile and applicable in different domains. In the future, we plan to extend our framework to handle more intricate concept drift scenarios, and one potential direction is the combination of LEAF and continual learning methods.