AdaMixT: Adaptive Weighted Mixture of Multi-Scale Expert Transformers for Time Series Forecasting

1 Introduction

Time series forecasting is essential for various fields, aiming to accurately present future values according to historical observations. The rapid advancement of deep learning has spurred significant research in this area, with applications in traffic prediction [Zhao et al., 2024; Jiang et al., 2024; Yuan and Li, 2021; Chen et al., 2024], recommender systems [Lin et al., 2024a; Lin et al., 2024b], and weather forecasting [Zhou et al., 2021; Liu et al., 2025; Miao et al., 2024b].

The recent success of attention mechanisms has prompted researchers to investigate the potential of Transformer-based models by redefining time series forecasting tasks as the future token prediction [He et al., 2024b; Chen et al., 2025; Yuan et al., 2024]. Existing research in this field can be categorized into two classes based on different tokenization methods, as illustrated in Figure 1. The first approach uses timestamps as tokens, exemplified by models such as Autoformer [Wu et al., 2021] and FEDformer [Zhou et al., 2022]. The second approach adopts patch-level tokenization, where a patch represents a window of timestamps, with PatchTST [Nie et al., 2022] being a notable example. Compared with timestamp-level tokenization, patch-level tokenization more effectively captures temporal patterns, thereby achieving superior forecasting performance. Despite their remarkable success, these methods still face challenges related to limited generalizability. Most existing research focuses primarily on single-scale temporal features, which lack a careful consideration of multi-scale characteristics in time series data. Furthermore, the effective fusion of multi-scale features has not been thoroughly explored, leading to limitations in forecasting under complex scenarios.

† Corresponding author.

To address the above challenges, we propose a novel time series forecasting architecture named Adaptive Weighted Mixture of Multi-Scale Expert Transformers (AdaMixT). By incorporating multi-scale temporal features, AdaMixT significantly enhances both predictive accuracy and generalizability, providing an innovative approach to multi-scale feature fusion. Specifically, AdaMixT leverages both General Pretrained Models (GPM) and Domain-specific Models (DSM) for time series forecasting, combining the extensive knowledge from GPM with the specialized feature extraction capabilities of DSM to better support downstream tasks. In this framework, input data is segmented into multi-scale patches; smaller patches capture high-frequency features for highresolution representations, while larger patches capture lowfrequency features for low-resolution representations. To effectively integrate multi-scale features, AdaMixT adopts a weighted fusion strategy to assign different weights for the output of each model, generating the final prediction. In summary, we make the main contributions as follows:

• We propose a novel multi-scale patch design that combines GPM with DSM, enabling the capture of both shortterm and long-term temporal patterns in time series. This integration leverages the open-world knowledge of pretrained models and the specialized feature extraction capabilities of domain-specific architectures, thereby improving forecasting accuracy and model generalizability.
• To the best of our knowledge, this work is the first to introduce an adaptive mechanism for multi-scale feature fusion in time series forecasting. By utilizing multiple models as experts and dynamically assigning weights based on the characteristics of time series data, our approach significantly enhances the model's adaptability to various tasks, ensuring robust scalability across different real-world application scenarios.
• We conduct extensive experiments across multiple time series forecasting benchmarks, demonstrating that AdaMixT outperforms existing forecasting approaches.

2 RelatedWork

2.1 Time Series Forecasting

In recent years, considerable research efforts have been directed toward leveraging Transformer-based models for time series forecasting. Early approaches, such as Informer [Zhou et al., 2021], adopt a sequence-to-sequence framework, treating each timestamp as an individual token. Similarly, Autoformer [Wu et al., 2021] incorporates classical analysis concepts such as decomposition and autocorrelation, while FEDformer [Zhou et al., 2022] utilizes a Fourier-enhanced structure to achieve linear computational complexity. However, [Zeng et al., 2023] highlights the limitations of treating each timestamp as a token, particularly in capturing intricate temporal patterns. To overcome this drawback, models such as Crossformer [Zhang and Yan, 2023] and PatchTST [Nie et al., 2022] are inspired by patch-based visual transformers [Dosovitskiy et al., 2020], representing windows of multiple timesteps as patches and using these as tokens for improved performance. Concurrently, the impressive capabilities of Large Language Models (LLMs) have led to their application in time series forecasting. Models like GPT4TS [Zhou et al., 2023] and TIME-LLM [Jin et al.,

2023] have shown promising results, further demonstrating the potential of LLMs in this field.

Despite these advancements, existing methods [Miao et al., 2024a; Yuan et al., 2021; Yuan et al., 2020] still face challenges in effectively capturing and fusing multi-scale features. To overcome these limitations, we propose AdaMixT, which enhances robust multi-scale feature extraction and includes an efficient fusion mechanism. By combining the strengths of GPM and DSM, AdaMixT offers a comprehensive and efficient solution for time series analysis.

2.2 Multi-scale Feature Learning

The analysis of multi-scale features plays a pivotal role in numerous fields. In computer vision, multi-scale features enable the extraction of information at varying spatial resolutions within an image, facilitating the analysis of both finegrained local details and overarching global structures [Das and Dutta, 2020; He et al., 2024a]. Meanwhile, multi-scale features have also been widely applied in domains such as knowledge graphs [Li et al., 2025; Li et al., 2022]. In recent years, multi-scale analysis has also been increasingly adopted in time series forecasting. For instance, TimesNet [Wu et al., 2022] transforms one-dimensional sequences into twodimensional tensors to capture diverse periodic patterns. In addition, MICN [Wang et al., 2023] utilizes convolutions with varying kernel sizes to learn features at multiple temporal scales. TimeMixer [Wang et al., 2024] enhances predictive performance by decomposing multiscale series and effectively blending their seasonal and trend components.

Unlike existing works [Miao et al., 2025; Yuan et al., 2023], AdaMixT learns multi-scale features during the training phase, enabling more efficient feature representation. Furthermore, instead of relying on straightforward feature fusion methods such as concatenation or addition, we design a gating network to score multi-scale temporal features and propose an innovative multi-scale feature fusion mechanism, which further enhances the performance and generalizability.

2.3 Mixture of Experts

Mixture of Experts (MoE) refers to a model consisting of different components, known as experts, each specialized in handling distinct tasks or specific aspects of data. Initially introduced in the literature [Jacobs et al., 1991], it has since been extensively studied and refined in subsequent works [Aljundi et al., 2017]. The advent of sparsegated MoE [Shazeer et al., 2017], particularly within large Transformer-based language models [Lepikhin et al., 2020], has revitalized this technique, expanding its applicability and effectiveness. Traditional MoEs typically rely on feedforward networks to select models based on specific scenarios. Unlike these designs, AdaMixT introduces multi-scale feature fusion into time series analysis for the first time. By adopting a self-learning mechanism, AdaMixT focuses on feature fusion rather than model selection, enabling the model to automatically identify critical multi-scale features and achieve efficient integration.

3 Method

In this section, we first provide a formal definition of the problem along with necessary notations and then explain the details of the overall model structure.

3.1 ProblemDefinition

Multivariate time series forecasting involves predicting future values based on historical observation. The objective is to forecast future values for the next K timestamps, given the observations from the previous L timestamps. A multivariate time series comprises multiple related time series, each representing a different variable. Let xt = [x(1) t , x(2) t , . . . , x(M) t ] ∈ RM×1 be a multivariate signal, where x(i) t denotes the i-th variate at time t, for 1 ≤ i ≤M.

In Equation (1), our goal is to develop a model that accurately predicts the values for the next K timestamps based on the recent history spanning L timestamps. Here, L and K are referred to as the look-back window and the prediction horizon, respectively. For a single variable, ˆx(i) L+1 , . . . , ˆx(i) L+K represents the sequence of future values in the next K time steps for the i th variable, which is the prediction target. In contrast, x(i) 1 , . . . , x(i) L denote the observed data for the i-th variable over the past L time steps. The function F, parameterized by Φ, leverages these observations to effectively forecast the future sequence values.

3.2 Model Structure

The architecture of AdaMixT is illustrated in Figure 2. The model leverages GPMs (e.g., GPT2 [Radford et al., 2019] and Llama [Touvron et al., 2023]) and DSMs (e.g., PatchTST [Nie et al., 2022]) as experts to efficiently extract multi-scale features from time series, thereby achieving accurate prediction objectives. The entire framework consists of three core modules: Multi-Scale Feature Extraction, Expert Pool, and Adaptive Weighted Gating Network (AWGN).

As detailed in Algorithm 1, AdaMixT adopts a channelindependent framework to process each variable. The process begins with instance normalization to standardize each variable. Following this, the time series is segmented into patches of varying lengths using a patch-based tokenization, enabling the capture of multi-scale temporal features spanning both short-term and long-term dependencies. These patches are subsequently processed by multiple experts, including GPMs for general feature representation and DSMs tailored for time-series forecasting. Each expert extracts features from input patches and integrates their outputs based on the scoring results of Adaptive Weighted Gating Network. The fused results are then passed through Linear Head to generate the final predictions. Furthermore, while AdaMixT inherently employs a channel-independent design, its architecture is highly flexible and can be seamlessly extended to other patch-based time series models, such as Crossformer [Zhang and Yan, 2023], for modeling inter-variable dependencies. This versatility establishes AdaMixT as an efficient

Algorithm 1 Forcasting Process of AdaMixT

Input: Multivariate time series X1:L with M variables, Look-back window L, Prediction horizon K Parameters: Learning rate η, Batch size B, Patch length P, Stride S, Scale factors {F1, F2, . . . , Fn} Output: Predictions ˆXL+1:L+K 1: for i ← 1 toM do 2: x norm(i) 1:L ← Instance Normalization(x(i) 1:L) 3: end for 4: for i ← 1 toM do 5: for j ← 1 to n do 6: p(i) j ←MF Extract(x norm(i) 1:L, P · Fj, S · Fj) 7: end for 8: end for 9: for i ← 1 toM do 10: for j ← 1 to n do 11: E(i) j ←Expert Modelj(p(i) j ) 12: end for 13: G(i) ←Gating Network(x norm(i) 1:L) 14: fused feature(i) ← ∑nj=1 G(i) j · E(i) j 15: end for 16: for i ← 1 toM do 17: ˆx(i) L+1:L+K ←Linear Head(fused feature(i) ) 18: end for 19: return ˆXL+1:L+K

and generalizable solution for time series forecasting tasks.

Forward Process. We denote the i-th univariate sequence of length L as x(i) 1:L = (x(i) 1 , . . . , x(i) L ), where i = 1, . . . ,M. The input sequence (x1, . . . , xL) is divided into M univariate sequences x(i) ∈ R1×L, each independently processed by a model consisting of multiple experts, with each sequence using a different patch length. Finally, different weights are assigned to each expert model, and multi-feature fusion is performed. After passing through the final Linear Head, the corresponding prediction results are returned as ˆx(i) = (ˆx(i) L+1 , . . . , ˆx(i) L+K ) ∈ R1×K .

Multi-scale Feature Extraction. Time segments composed of multiple consecutive timestamps are essential for learning effective predictive representations [Nie et al., 2022]. Building on this idea, we incorporate multi-scale features of time series into the modeling process by dividing each input univariate time series x(i) 1:L into patches of varying lengths, which may be overlapping or non-overlapping. Specifically, we define the base patch length P and stride S, and utilize Scale factors = {F1, F2, . . . , Fn} to adjust the scale sizes, where n denotes the total number of defined scales. Through this patching process, n patch sequences x(i)(j) 1:L ∈ RPj×Nj are generated, where Nj = ⌊ L-Pj Sj ⌋ + 2 represents the number of patches derived from the j-th patching operation. Here, Pj = P · Fj denotes the adjusted patch length, and Sj = S · Fj represents the adjusted stride at the j-th scale. To ensure alignment, the original sequence x(i) 1:L is padded at the end with Sj repeated values before performing the patching process.

By adopting patch lengths of varying scales, smaller Pj values enable the x(i)(j) 1:L branch to specifically focus on shortterm temporal features with finer granularity, thereby achieving high-resolution modeling. In contrast, larger Pj values are better suited for effectively capturing long-term seasonal variations and trend characteristics, thus achieving the goal of multi-scale feature extraction.

Expert Pool. We propose the concept of an Expert Pool, designed to integrate the strengths of different models to enhance the performance of multivariate time series forecasting. The Expert Pool consists of two core types of models: General Pre-trained Models and Domain-Specific Models. GPMs (e.g., GPT-2 [Radford et al., 2019] and Llama [Touvron et al., 2023]) demonstrate exceptional performance in understanding complex time series data due to their strong generalizability and rich feature representation learning. In contrast, DSMs (e.g., PatchTST [Nie et al., 2022]) focus on precise pattern extraction in time series data, capturing fine-grained features and effectively compensating for the limitations of GPMs in specific tasks.

Unlike existing methods [Jin et al., 2023; Wang et al., 2023] that typically utilize either GPM or DSM individually, our approach combines these two types of models, significantly improving prediction performance through the synergistic effects. Both types of models are based on the Transformer architecture, whose core mechanism is the Attention Mechanism. Specifically, patches are first projected into the Transformer latent space of dimension D using a trainable linear projection Wp ∈ RD×P , and a learnable positional encoding Wpos ∈ RD×N is added to preserve temporal order. The transformed patches are then processed by the multi-head attention module, where Query (Q), Key (K), and Value (V) matrices are computed as follows: Q(i) h = (x(i) d )T WQ h , K(i) h = (x(i) d )T WK h , V(i) h = (x(i) d )T WV h , where WQ h ,WK h ∈ RD×dk and WV h ∈ RD×D . The attention output O(i) h ∈ RD×N : is then computed using a scaled dot-product: (O(i) h )T = Softmax ( Q(i) h K(i)T h √dk ) V(i) h .

Adaptive Weighted Gating Network. In this study, we proposed AWGN, which differs from traditional simple feature fusion methods such as feature addition or concatenation. This innovative approach effectively considers the importance of different features, thereby enhancing the accuracy of time series forecasting. As shown in Figure 2(b), AWGN is a three-layer MLP network that dynamically assigns weights G(x(i) ) to the expert models based on the characteristics of each input sequence x(i). These weights are subsequently applied to the output features E(p(i) ) of the expert models, as described in Equation (2). Here, x(i) represents the time series features before patch partitioning, p(i) denotes the sequence after partitioning, and n indicates the total number of experts in the expert model pool.

Dataset # Features # Timesteps Frequency Domain

Finally, the features weighted by AWGN are aggregated and

ETTh2 Traffic Weather Figure 3: Average MSE and MAE comparison with different model variants on ETTh2, Traffic, and Weather datasets.

passed through Linear Head to generate the final prediction values. This scoring and weighted aggregation mechanism offers a more efficient approach to integrating multi-feature information for time series forecasting tasks.

Loss Function. We utilize the Mean Squared Error (MSE) loss to quantify the discrepancy between the predictions and the ground truth. The loss in each channel is gathered and averaged over M time series to get the overall objective loss.

Instance Normalization. This technique has been recently introduced to address the distribution shift between training and testing data [Ulyanov et al., 2016]. It works by normalizing each time series instance x(i) to have a zero mean and unit standard deviation. Specifically, each x(i) is normalized before patching, and the mean and standard deviation are added back to the output prediction afterward.

4 Experiments

4.1 Experimental Setup

Dataset. We conduct experiments on eight widely recognized benchmarks, namely Weather, Traffic, Electricity, ILI, and four ETT datasets (ETTh1, ETTh2, ETTm1, and ETTm2), which were originally introduced by [Wu et al., 2021]. The statistics are summarized in Table 1.

Baselines. We evaluate AdaMixT against ten SOTA baselines across four categories: (i) LLM-based models, including TIME-LLM [Jin et al., 2023] and GPT4TS [Zhou et al., 2023]; (ii) Multi-scale time series models, such as TimeMixer [Wang et al., 2024], MICN [Wang et al., 2023], and TimesNet [Wu et al., 2022]; (iii) Transformerbased models, including iTransformer [Liu et al., 2023], PatchTST [Nie et al., 2022], FEDformer [Zhou et al., 2022], and Autoformer [Wu et al., 2021]; and (iv) the MLP-based model, DLinear [Zeng et al., 2023].

Implementation Details. To ensure the fairness of experiments, all baseline models are configured according to the settings in their official open-source repositories and evaluated using a unified evaluation framework, TSLib 1 . All experiments are implemented based on PyTorch [Paszke, 2019]

1 https://github.com/thuml/Time-Series-Library

and executed on an NVIDIA A100-80G GPU. The default optimizer for the experiments is Adam [Kingma and Ba, 2014], and each experiment is repeated three times, with the average results reported.

4.2 Performance Comparison

Table 2 presents the experimental results for long-termmultivariate forecasting, highlighting the substantial advantages of our model. Specifically, compared to LLM-based methods, our model reduces MSE and MAE by 13.19% and 5.35%, respectively. When compared to Multi-scale models, the reductions are 24.99% and 15.14%, respectively. Relative to the DLinear model, MSE and MAE decrease by 22.73% and 13.35%. Furthermore, our model exhibits even more significant improvements over other transformer-based models. Notably, even when compared with SOTA models such as TIME-LLM [Jin et al., 2023] and PatchTST [Nie et al., 2022], our model consistently delivers superior performance. These results strongly demonstrate the robustness and superiority of our approach across various datasets.

4.3 Model Analysis

Ablation Study. To comprehensively evaluate the contributions of the key components-GPM, DSM, and AWGN-to the overall performance of the AdaMixT, we conducted a series of ablation experiments. These experiments systematically removed each component, allowing us to compare the performance of the resulting model variants and quantify the impact of each element.

As shown in Figure 3, the ablation study results demonstrate that the complete AdaMixT model achieves the best performance on the ETTh2, Weather, and Traffic datasets, while removing any single module leads to a decrease in prediction accuracy. For example, on the ETTh2 dataset, removing GPM and DSM results in performance degradation, highlighting the critical role of GPM in extracting general feature representations and the significant value of DSM in modeling temporal features. Notably, the most substantial performance drop is observed when AWGN is removed, with both MSE and MAE significantly higher than other variants. This finding underscores the pivotal role of AWGN in multi-scale feature fusion and its importance in improving prediction accuracy. Similar trends are observed on the Weather and Traffic datasets, further confirming the importance of each module in enhancing model performance.

Results with Different Models. In the main experiments, AdaMixT utilizes GPT-2 [Radford et al., 2019] as GPM. To further validate the effectiveness of the model, we also perform comparative experiments using BERT and LLaMA-7B. The experimental results in Figure 4 indicate that the performance differences among BERT, GPT-2, and LLaMA-7B are minimal on the ETTh1, Weather, and Traffic datasets. This finding demonstrates the high robustness of AdaMixT in selecting GPMs, as it consistently delivers stable performance across various pretrained models.

Hyperparameter Sensitivity. We conducte a sensitivity analysis on 3 key hyperparameters, including the number of layers in the backbone model, the look-back window, and the number of experts. Results are shown in Figure 5. Based on the results, we summarize the following conclusions:

• Backbone Layers: The number of layers in the backbone is positively correlated with the predictive performance. This indicates that, even after fusion, GPM retain favorable scaling laws with respect to layer depth, which positively influences model performance.
• Look-backWindow: The Look-backwindowdirectly affects prediction accuracy, especially at extended forecasting horizons. This observation aligns with the patterns

found in traditional models, demonstrating that longer historical inputs can effectively enhance performance.

• Number of Experts: The experimental results indicate that increasing the number of expert models can effectively capture features at different scales, thereby significantly improving the accuracy of time series forecasting. This result validates the effectiveness of the feature fusion mechanism in the AdaMixT. However, caution should be taken as the model may be prone to overfitting.

Scale Factors Study. As shown in Algorithm 1, the setting of scale factors determines the granularity of feature extraction. The selection of the appropriate parameters is critical to the accuracy of the prediction. To further investigate this, we conduct an analysis of the impact of different feature scale settings on prediction performance, using the ILI and ETTh1 datasets as examples.

Through a cyclic study of these two datasets, we find that the cyclicity of ILI is significantly longer than that of ETTh1. Based on this observation, we employ two expert models in the experiment, with the scale factor for GPM set to 1, and the scale factor range for the DSM set to {1/2, 1, 2}. The results, as shown in Table 3, indicate that for datasets with longer periodicities, using larger scale factors improves prediction performance (and vice versa). This finding suggests that when selecting scale factors, the intrinsic characteristics of the time series should be considered to achieve optimal prediction results.

InferenceTime Study. In order to evaluate the practical applicability, we compare the inference time of AdaMixT with current similar methods. As shown in Figure 6, AdaMixT

demonstrates superior inference time compared with other complex multi-scale feature fusion models and LLM-based models. This advantage is primarily attributed to the fact that AdaMixT does not require complex prompt generation, postprocessing, or frequency domain transformations.

5 Conclusion and FutureWorks

In this paper, we present AdaMixT, which is designed to address the limitations of existing methods in terms of generalizability and multi-scale feature fusion. AdaMixT incorporates three key innovations: Multi-scale Feature Extraction, Expert Pool, and Adaptive Weighted Gating Network, significantly enhancing the model's performance. To the best of our knowledge, this work is the first to combine the general feature representation capability of GPM with the fine-grained modeling ability of DSM to construct the expert pool. Additionally, we design AWGN that evaluates the weights of different models, enabling effective fusion of multi-scale features. Extensive experimental results on multiple benchmark datasets demonstrate that our approach outperforms existing forecasting approaches.

Our model offers new insights into multi-scale feature fusion in time series analysis. In the future, we aim to further explore multi-scale fusion methods and strengthen the complementary strengths of GPM and DSM, which will be a crucial research direction moving forward.

Acknowledgments

This work is supported by the National Key Laboratory under grant 241-HF-Z01-01. Guoliang Li is supported by the National Key R&D Program of China (2023YFB4503600), NSF of China (62232009), Zhongguancun Lab, Huawei, and Beijing National Research Center for Information Science and Technology (BNRist).