Non-collective Calibrating Strategy for Time Series Forecasting

1 Introduction

Time series forecasting remains challenging due to the inherent diversity and complexity of the data, which are influenced by factors such as seasonal fluctuations, trends, and domainspecific patterns [Deng et al., 2024a]. To investigate the most effective deep learning architecture, researchers have endeavored from convolutional neural networks (CNNs) [Borovykh et al., 2017] and recurrent neural networks (RNNs) [Lai et al., 2018] to graph neural networks (GNNs) [Wen et al., 2023] and Transformers [Zhang et al., 2024]. However, empirical debates persist regarding the performance of these models in time series forecasting. For instance, some studies have revisited the argument that a properly designed simple multi-layer perceptron (MLP) can outperform advanced Transformer-based models [Zeng et al., 2023; Lu et al., 2018]. With SOTA models emerging almost daily, disagreements over performance can incur significant inefficiencies for decision-makers during model selection. Consider a scenario where a trained model is already performing effectively in a production environment, yet numerous novel SOTA models continue to emerge. Determining whether any of these newmodels has the potential to surpass the deployed model typically requires extensive trial-and-error experimentation. This raises a critical question: rather than exhaustively testing each new SOTA model, could the existing trained model be enhanced at minimal cost?

† Feng Hong and Yanwei Yu are corresponding authors.

In this study, we instead draw attention to harness trained time series forecasting methods rather than designing a new model from scratch. An intuitive solution involves calibrating time series predictions using prior knowledge, such as rulebased methods that apply exponential smoothing and filtering to reduce noise and constraint bounds. Despite their simplicity, such approaches demand rigorous domain knowledge, as improper handling can degrade performance. Motivated by the effectiveness of learning-based classification calibration [Platt, 2000], we would explore its potential in time series forecasting models.

Classical calibrating typically involves keeping the fully trained model frozen while post-processing only the outputs of the forecasting models [Kuleshov et al., 2018]. Generally, this process calibrates the predictive bias across all target variables and prediction horizons simultaneously, a strategy we term as collective calibrating in this study. In contrast, when the calibrating process is applied separately to grouped or individual variables or prediction horizons, we introduce the term non-collective calibrating. To investigate the effect of these two strategies, we started by conducting a preliminary experiment, with the key discovery illustrated in Figure 2 and described as follows. Employing a classical collective calibrating strategy - where an additional MLP to be finetuned (illustrated in the bottom part of Figure 2) is attached to the existing trained inference model (represent by the blue rectangle)- we observed premature triggering (e.g., around 10 epochs) of the early-stopping monitor. Conversely, when the single MLP was split equally by the number of neurons into five smaller, independent MLPs denoted as MLP1 to MLP5, each dedicated to be finetuned for a single target variable and equipped with its own optimizer and early-stopping monitor (illustrated in the upper part of Figure 2), we observed significant variation in the early-stopping epochs across monitors. While some stopped in fewer than 10 epochs, others required substantially more epochs to achieve optimal performance on validation data.

Our heuristic hypothesis is that, due to varying levels of learnability across different targets, optimizing all variables simultaneously under the same objective loss function may fail to fully capture the unique distribution of each target variable - a phenomenon we refer to as multi-target learning conflict (MTLC).

To test the above hypothesis, we have conducted rigorous ablation study to compare the forecasting performance of the two strategies. Our findings support that the non-collective calibrating strategy outperforms its collective counterpart regarding generalization performance (referred to Table 2). We metaphorically term this approach the Socket+Plug strategy (SoP), where the well-trained inference model serves as a Socket, providing the fundamental shared predictive ability, while each additional module (e.g., MLP) as a Plug, tailored to its specific calibrating target. Extensive experiments further demonstrate that SoP can generally enhance the performance when existing deep forecasting models are taken as the Socket, as highlighted in Figure 1. We further investigated two specific forms of SoP: variable-wise SoP and step-wise SoP. Experimental results indicate that, under the same model architectures, both step-wise and variable-wise SoP exhibit similarly strong performance. It is worth noting that SoP leverages off-the-shelf Socket outputs without requiring any modifications to the Socket, making it easily combined with existing SOTA deep forecasting methods. Moreover, we applied SoP to the spatio-temporal meteorological forecasting task, utilizing Unet as the Socket. The experimental results validate its effectiveness, highlighting its potential for integration with advanced weather forecasting systems, such as PanGu [Bi et al., 2023] and FengWu [Chen et al., 2023a], thereby underscoring its applicability to real-world weather forecasting tasks. In summary, the key contributions of this study are summarized as follows:

• To our best knowledge, we are the first to identify and emphasize the detrimental impact of the MTLC phe-

nomenon in deep forecasting, which limits the learning potential of time series models. Specifically, we demonstrate that the traditional early-stopping mechanism in model training could fail to fully exploit the capabilities of these models.

• We propose SoP, a universal and model-agnostic calibrating strategy, developed to calibrate the performance of fully trained deep forecasting models with minimal design costs. By employing a dedicated optimizer and early-stopping monitor for each predicted target, SoP effectively mitigates the MTLC issue.
• We derive two specific forms of SoP, referred to as target-wise SoP, which include variable-wise SoP and step-wise SoP. Both demonstrate significantly better performance than classical collective calibrating and can be used as quick-start versions of SoP, eliminating the need to determine the Counts hyperparameter for the Plug.
• We conduct extensive experiments on seven time series benchmark datasets and the spatio-temporal ERA5 meteorological dataset, whose results demonstrate the effectiveness of SoP, achieving up to a 22% improvement in forecasting performance. This showcases its potential for advancing time series and spatio-temporal forecasting tasks.

2 Preliminaries

2.1 Definitions

Definition 1 (Time Series Forecasting). Given a historical observation series X = {x1, x2, . . . , xT } ∈ RN×T , which represents past T time steps and N variables, let X:,t denote the values of all variables at time step t, and Xn,: denote the values across all past time steps for variable n. Time series forecasting aims to learn a model f(·) that predicts the future values ¯ Y = f(X) = {xT+1, xT+2, . . . , xT+S} ∈ RN×S over the future S time steps, a.k.a., S prediction horizons.

Definition 2 (Collective & Non-collective calibrating). Traditional calibrating typically involves keeping the inference model frozen while re-training part of model layers collectively for all target variables across all prediction horizons. We define this type of calibrating as collective calibrating. In contrast, when the calibrating process is applied separately to grouped or even individual variables or prediction horizons, we denote this as non-collective calibrating. With these definitions, our proposed SoP strategy falls under non-collective calibrating.

Definition 3 (Optimized Plug & Plug Counts). When implementing SoP, one can decide how many variables or prediction horizons are optimized together as a group, each with its own optimizer and early-stopping monitor. We refer to each such group of variables or horizons as an Optimized Plug and the number of total optimized plugs is denoted as the Plug Counts. Specifically, for a prediction target Y ∈ RN×S : If n variables along the N-dimension form an optimized plug to predict Yplug ∈ Rn×S , SoP creates the plug counts as M = Nn . If s horizons along the S-dimension form an optimized plug to predict Yplug ∈ RN×s , SoP creates the plug counts as M = Ss . Two special remarks are derived as:

Remark 1. There are two types of target-wise SoP: when n = 1, M = N, it is referred to as the variable-wise SoP; and when s = 1, M = S, it is referred to as the step-wise SoP. Remark 2. When n = N and s = S, M = 1, in which case

SoP reduces to the classical collective calibrating strategy.

3 Methodology

To clarify the data flow in the model operation based on SoP, this section first introduces the model inference process given the input, followed by an explanation of the calibrating process of SoP. Without loss of generality and for clarity, we use a variable-wise SoP (where M = N, as described in Remark 1.) to illustrate the process.

3.1 Model Inference

As shown in Figure 3, the proposed SoP strategy operates on the basis of two key components: the Socket and the Plug. The Socket is adopted to capture complex correlations among variables and deliver the high-quality foundational forecasts. To this end, it is recommended to harness a trained SOTA forecasting model, such as a Transformer-based variant, as the Socket. Given an input sample X ∈ RN×T , the Socket can preliminarily infer the prediction of the future sequence ˆ Y as follows:

The obtained ˆ Y is then divided into M groups along the dimension of variables or horizons. Specifically, in the case of variable-wise SoP, ˆ Y is divided into N groups based on each variable index i, forming ˆ Yi,: ∈ RS . Each ˆ Yi,: undergoes layer normalization to produce the normalized values Vi,:, a technique demonstrated to improve the stability of the training convergence [Ba et al., 2016], and then is processed by a Plugi for calibrating.

where MLPi: RS→Rd→RS is a projection that projects the sequence representation from S to the hidden dimension d and finally returns to the S dimension, which consists of two d-dimension hidden layers and a GELU activation [Hendrycks and Gimpel, 2016]. The symbol ⊙ represents the element-wise multiplication [Han et al., 2024]. Finally, the predicted outputs of the N Plugs are concatenated to obtain the final forecasting results ¯ Y as follows:

3.2 Non-collective Model Calibrating

The calibrating process through variable-wise SoP is presented in Algorithm 1. Let a batch of data X ∈ RB×N×T and the corresponding ground truth Y ∈ RB×N×S , where B represents the batch size, be provided. The Mean Squared Error (MSE) serves as the loss function L. The role of each Plugi is to calibrate the preliminary forecasting results ˆ Yi,: for each target variable i. Specifically, each Plugi is equipped with a dedicated optimizer opti for minimizing its training loss Ltrain i and a monitor mnti for early-stopping associated with the validation loss Lval i , enabling exclusive calibrating.

Algorithm 1 Training algorithm illustrated through variablewise SoP

Input: Trained Socket; training dataset Dtrain; validation dataset Dval.

• 1: for i = 1 to N do 2: initialize the parameters of Plugi ← θi 3: repeat 4: fetch a batch of data (X, Y) from Dtrain 5: ˆY = Socket(X) 6: ¯Y = Plugi ( ˆY) 7: Ltrain i ←MSE(¯Y,Y) 8: update θi using optimizer opti by minimizing Ltrain i 9: if up to the end of an epoch then 10: snapshot Plugi if Lval i on Dval is minimal 11: end if 12: until early-stopping of mnti is triggered 13: end for
• 14: Output: N snapshots of trained Plugs

4 Experiments

In this section, we present the experimental setup and results of SoP, focusing on the following research questions:

• RQ1: How effective is SoP in enhancing the performance of existing time series forecasting models such as DLinear, TimesNet, and others?
• RQ2: How effective is the target-wise SoP, specifically in terms of variable-wise and step-wise calibration?
• RQ3: How does the performance differ between noncollective and collective optimizer deployment?
• RQ4: How effectively can SoP extend its functionality to spatio-temporal forecasting?

4.1 Experimental Setup

Datasets. We initially conducted extensive experiments on benchmark time series datasets, including ETTh1, ETTh2, ECL, Exchange, Weather, and Solar-Energy.

Inference models. SoP is a model-agnostic method that can be broadly applied to any trained deep neural networks. We assess the effectiveness of SoP by applying it to seven SOTA forecasting methods as the inference models, also referred to as Sockets, which are categorized as follows: (1) Transformer-based methods including FEDformer [Zhou et al., 2022], PatchTST [Nie et al., 2023], and iTransformer [Liu et al., 2024b]; (2) MLP-based methods including DLinear [Zeng et al., 2023], TSMixer [Chen et al., 2023b], and SOFTS [Han et al., 2024]; and (3) a TCN-based method: TimesNet [Wu et al., 2023].

Experimental settings. We utilized the open library TSLib [Wang et al., 2024], which enabled the easy reproduction of aforementioned inference models. For SOFTS, which is not included in TSLib, we reproduced it using the hyperparameter settings provided in the original paper [Han et al., 2024].

4.2 Effectiveness of SoP to Enhance Inference Models (address RQ1)

Table 1 reports the averaged performance across all horizons, comparing the inference models and their enhancement using variable-wise SoP across seven benchmark datasets. The results demonstrate that SoP significantly enhances forecasting accuracy across various baselines, whether MLP- or Transformer-based methods. Notably, all Model+Plug methods achieve an improvement rate exceeding 10% in MSE on the Exchange dataset. Based on prior findings [Qiu et al., 2024], which identified datasets like Exchange as having lower inter-variable correlations and datasets like Weather as having higher inter-variable correlations, we conclude that variable-wise SoP performs better with datasets characterized by lower inter-variable correlations than with those exhibiting higher inter-variable correlations. This also implies the importance of understanding dataset characteristics to optimize the potential of SoP further. Notably, while DLinear underperforms compared to TimeNet, the integration of SoP enables DLinear+Plug surpasses TimesNet+Plug on most datasets.

Figure 4 illustrates the performance of applying SoP to each of the three models with prediction horizons S = {96, 192, 336, 720} on the ECL and Exchange datasets. The experimental results demonstrate that the SoP strategy consistently decreases the MSE at diverse prediction horizons. As the prediction horizon expands, the performance improvement brought by SoP becomes increasingly evident, manifested by the growing difference between the solid and dashed lines of the same color.

4.3 Effectiveness of Target-wise SoP (address RQ2)

To investigate the effectiveness of target-wise SoP, we examine the performance SoP with different plug counts on the Weather dataset, which contains a total of 21 target variables.

Effect of SoP from Variable View. Given a total of 21 target variables, we specifically setup the plug counts along the Ndimension to form {21, 7, 3, 1} plugs in turn. The experimental results, shown in Figure 5, reveal several key observations. First, most non-collective SoP clearly enhances the performance of inference models, with SoP using plug count of 7 achieving the best performance, followed closely by variablewise SoP. The variable-wise SoP, by eliminating the need to fine-tune the plug count, is thus suggested as a quick-start version of SoP. Second, plug count of 1 (i.e., collective calibrating) generally results in the worst performance, even performing worse than the inference model without calibrating. This suggests the MTLC indeed degrades the performance of most models. Lastly, models with simpler architectures, such as DLinear, benefit more from incorporating SoP.

Effect of SoP from Step View. Given that S ∈ {96, 192, 336, 720}, we set up the plug counts along the Sdimension as follows: for S = 96, the plug counts form {96, 32, 16, 2, 1}; for S = 192 , the plug counts form {192, 64, 32, 4, 1}; for S = 336, the plug counts form {336, 128, 64, 8, 1}; and for S = 720, the plug counts form {720, 240, 120, 15, 1}. As shown in Figure 6, the experimental results reveal that MTLC degrades the performance of most models, except for TSMixer, which performs best with collective calibration when S = 96. In contrast, noncollective SoP significantly enhances model performance, with PatchTST, implementing step-wise SoP, achieving the best results across most prediction horizons. Overall, the step-wise SoP, by eliminating the need to experiment with specific plug counts, performs considerably well and can be chosen as a quick-start version of SoP from the step view.

4.4 Ablation Study of Non-collective and Collective Optimizer Deployment (address RQ3)

As highlighted by the MTLC phenomenon in Figure 2, we conducted a more rigorous ablation study to compare the effects of non-collective and collective optimizer deployment.

Table 2 summarizes the performance achieved by deploying two different calibrating strategies. The improvements are calculated relative to the performance of the Socket without calibrating, with the superior improvement in each comparison highlighted in bold. Promotion 1 is achieved through variable-wise SoP, where each plug is optimized independently by a specific optimizer. Promotion 2 is achieved via collective calibrating, where all plugs share a unified optimizer. The results show that while both calibrating approaches yield performance improvements, SoP leads to more substantial gains. The average improvement achieved by Promotion 1 is approximately twice that of Promotion 2. A striking example of this difference can be observed with the TSMixer model on the Exchange dataset, where collective calibrating results in a degradation in model performance, while the corresponding SoP strategy leads to significant improvements.

4.5 Experiments on Spatio-temporal Forecasting (address RQ4)

Both time series forecasting and spatio-temporal forecasting aim to predict future values based on historical data, but spatio-temporal forecasting is more complex in terms of data scale and model complexity. This section demonstrates the effectiveness of SoP in spatio-temporal forecasting.

Dataset. The experiments were conducted on the ERA5 dataset [Alibaba Group, 2023], which includes five meteorological variables: T2M, U10, V10, MSL and TP.

Inference model. The inference model selected as the Socket is Unet [Ronneberger et al., 2015]. A direct prediction strategy is employed, which generates predictions for the next 20 prediction horizons without iterative steps.

Experimental settings. For spatio-temporal prediction, a data sample is represented as X ∈ RN×H×W×T , with Y ∈ RN×H×W×S as the corresponding output. In this forecasting task, observations from the past two time steps (T = 2) involving five variables (N = 5) are used to predict the same five variables over the next 20 prediction horizons (S = 20). H ×W denotes the spatial range, with W = H = 160. Experimental results. Next, we provide the analysis of the experimental results in detail.

The Effect of SoP on Spatio-temporal Variables. As shown in Figure 7 (a), the averaged test MSE of the five variables significantly decreased after applying SoP to Unet, particularly for the variables T2M, U10, and V10. This improvement can be attributed to the inherently stronger cyclicity of T2M, which enhances its predictability compared to TP and MSL, which aligns with findings in previous studies [Liu et al., 2021]. Figure 8 displays a test sample to visualize the benefits of applying SoP to the Unet model for spatio-temporal predictions of T2M and V10. It can be observed that predictions from Unet alone appear overly smoothed and obscurer at certain spatial locations (highlighted in the black box). In contrast, Unet+SoP produces more textured and sharper predictions that are closer to the ground truth, not only at the first prediction horizon but also at the 20th prediction horizon.

The Effect of SoP on Prediction Horizons. As present in Figure 7 (b), the incorporation of the SoP consistently reduces the test MSE across each of the 20 prediction horizons. Notably, the difference in MSE between Unet+Plug and Unet gradually decreases as the prediction horizon increase. This phenomenon can be attributed to the reduced predictability of spatio-temporal forecasts as the horizon progresses [Bi et al., 2023]. Overall, SoP demonstrates its effectiveness both in short- and long-term spatio-temporal forecasting.

（a）

（b）

Figure 7: (a) The averaged MSE for each of five variables. (b) The averaged MSE at each prediction horizon.

5 RelatedWorks

Forecasting Models. Deep learning has significantly advanced time series forecasting, resulting in the development of numerous deep forecasting models [Deng et al., 2024b]. The architecture of deep models has progressed from CNNsbased frameworks [Wu et al., 2023], MLPs [Zeng et al., 2023] to Transformer [Liu et al., 2024b], achieving considerable performance improvements across various benchmarks. Recently, the rise of LLMs has opened new possibilities for time series forecasting [Liu et al., 2024a; Zhong et al., 2025]. However, some studies have shown that these popular LLMs often perform similarly to, or even worse than, simpler methods, while also significantly increasing computational costs horizon.

horizon.

horizon.

horizon.

Figure 8: Visualization of a spatio-temporal forecasting case study for prediction horizons S=1 and S=20.

during both training and inference [Tan et al., 2024]. As a result, there is no universally superior model architecture in the research community of time series forecasting.

Forecasting Strategies. Recent studies have shown that well-designed forecasting strategies-such as multi-view learning [Deng et al., 2022; Lu et al., 2019] and model calibration-can significantly influence predictive performance. Calibration, in particular, plays a critical role in aligning model predictions with the ground truth [Zhang et al., 2023], or in providing more reliable confidence intervals [Sun et al., 2023]. Early research employed Platt Scaling to transform the original predictions into calibrated probabilities [Platt and others, 1999]. More recently, non-parametric isotonic regression was introduced by [Berta et al., 2024] to calibrate binary classifiers. [Huang et al., 2023] designed an abductive reasoning mechanism to minimize the discrepancy between the inferred and the true values to adjust the predictions. [Zhang et al., 2023] employed the envelope-based bounds modeling to correct the initial predictions. While many previous studies have applied learning-based methods to calibrate model outputs, most have focused on collective calibrating. None of these approaches have noticed the detriment of MTLC in deep forecasting and proposed a solution through noncollective calibrating. To this end, this study has developed an effective calibrating strategy from two views that enhances the performance of deep forecasting models.

6 Conclusions

This study identifies a detrimental phenomenon, termed MTLC, which hampers the learning capability during the training of deep forecasting models. To address this issue, we propose an effective SoP strategy that calibrates well-trained forecasting models, regardless of their specific architectures. Additionally, we derive two specific forms of target-wise SoP for quick-start implementation. Comprehensive experiments demonstrate the superiority of the proposed method, even without exhaustive hyperparameter searching. We hope this approach paves the way for further advancements in time series forecasting. Future research could focus on improving calibrating efficiency through parallel Plug training.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Nos. 62402463, 41976185, 62176243, 62176221 and 72242106) and Major Basic Research Project of Shandong Provincial Natural Science Foundation (ZR2024ZD03).

Contribution Statement

Bin Wang and Yongqi Han are co-first authors with equal contributions.