NEST: Tackling Dataset-Level Distribution Shifts via Regime-Oriented Mixture-of-Experts

1 Introduction

Time series forecasting serves as a cornerstone for understanding and managing complex systems, ranging from the global web infrastructure [Basu et al., 1996] and power grids [Boehme et al., 2007] to intricate physical phenomena like space weather dynamics [Pulkkinen, 2007]. In these domains, the ability to provide accurate long-term predictions is essential for proactive resource allocation, anomaly detection, and ensuring system stability. However, real-world data streams are challenging to model due to their high nonstationarity and significant distribution shift across datasets.

∗Corresponding Author

This distribution shift is not random; it is a manifestation of diverse underlying behavioral modes and evolving system states and can be recurrent during a long period. For example, as illustrated in Figure 1, this heterogeneity is driven by different underlying behavioral modes, each generating a traffic pattern with a distinct statistical signature. Icon (a) illustrates low-volume, baseline traffic might be generated by automated system check-ins or background processes; icon (b) illustrates standard daily peaks, which could correspond to individual users engaging in typical interactive sessions; icon (c) illustrates the highest traffic surges, often reflecting coordinated, large-scale activity, such as many users accessing a popular service simultaneously during a major event. A monolithic forecasting model, when trained on data containing this mixture of behaviors, is forced to learn a single, "averaged" representation that fails to accurately capture the specific dynamics of any individual mode, leading to suboptimal performance.

Despite its prevalence, few existing methods address this dataset-level shift. The predominant focus remains on mitigating the local temporal shift between the historical lookback window and the future prediction window. While valuable, this paradigm primarily corrects for evolving nonstationarity within a small temporal vicinity; it does not explicitly model the global, structural challenge where the dataset itself is a composite of distinct operational regimes.

To address this fundamental problem, we argue for a systematic approach rooted in the identification of distinct operational regimes. We hypothesize that the diverse dynamic behaviors within a dataset can be effectively differentiated by moment-entropy metrics. Specifically, we employ low-order statistical moments (mean and variance) to capture distribution intensity, alongside Singular Value Decomposition Entropy (SVDEn) to quantify structural complexity. By mapping data slices into this moment-entropy space, we apply unsupervised clustering to automatically partition the dataset into functionally distinct categories. This categorization ensures that specialized experts can be trained to master the specific patterns inherent to each regime, effectively capturing the dynamic reorganization of inter-variable dependencies that occurs as the system transitions between different types of behavior.

To this end, we propose NEST: Tackling Dataset-Level Distribution Shifts via Regime-Oriented Mixture-of-Experts. NEST is a two-phase MoE framework designed to address dataset-level distribution shifts (DDS) by explicitly modeling the transitions between underlying system regimes. The process begins with unsupervised regime discovery, where we identify distinct operational modes through clustering in the moment-entropy space. Based on these discovered regimes, we then execute a structured two-phase training protocol to cultivate a heterogeneous expert pool. In Phase 1, we facilitate structural specialization within a heterogeneous expert pool; here, individual experts learn to model regime-specific sequence dynamics by evolving unique variate-attention patterns that capture the coupling logic of each mode. In Phase 2, the router generates an initial weight based on temporal content, which is then refined through static topological modulation to anchor the decision via geometric proximity to regime centroids. By dynamically recomposing these specialized kernels, NEST effectively transforms the challenge of distribution shifts into a manageable process of regime identification and dependency reconfiguration.

The key contributions of our work are as follows:

• Structural Regime Modeling: We propose NEST, a two-phase MoE framework that models the evolution of complex system regimes by dynamically recomposing inter-variable dependencies. By decoupling evolving dynamics into specialized kernels, NEST effectively captures the structural transitions between operational modes that drive dataset-level distribution shifts.
• Principled Discovery and Routing: We introduce an unsupervised discovery module using moment-entropy metrics to provide a mathematical basis for regime partitioning. Complementarily, we design a regime-oriented router that achieves precise expert orchestration by fusing expert weights initialization with distance-aware geometric modulation.
• Empirical Validation and Interpretability: NEST achieves state-of-the-art (SOTA) performance across diverse benchmarks, including heterogeneous network traffic and multi-year physical phenomena. Extensive analysis of attention polymorphism and weight permutation further validates its unique capability in the struc-

tural recomposition of variable dependencies.

2 RelatedWork

2.1 Deep Learning for Time Series Forecasting

Early efforts leveraged RNNs like LSTM [Siami-Namini et al., 2018] and GRU [Weerakody et al., 2021] for temporal gating, though sequential constraints hindered long-horizon efficiency. The transition to Transformers [Wu et al., 2020] introduced global dependencies, yet quadratic complexity prompted sparse or hierarchical variants like Informer [Zhou et al., 2021] and Pyraformer [Liu et al., 2021]. Subsequent innovations focused on domain-specific kernels: Autoformer [Wu et al., 2021] utilized auto-correlation, while FEDformer [Zhou et al., 2022] applied frequency-domain attention via Fourier/wavelet transforms [Torrence and Compo, 1998; Cleveland et al., 1990]. Recent paradigms emphasize structural rethinking: PatchTST [Nie et al., 2022] processes sub-series patches for local semantics, while iTransformer [Liu et al., 2023a] inverts the architecture to model multivariate correlations by treating whole variates as tokens. Concurrently, DLinear [Zeng et al., 2023] demonstrated the efficacy of simple MLP-based decomposition. More recently, unified backbones like UniTS [Gao et al., 2024] have emerged as foundation models, achieving robust zero-shot generalization across diverse forecasting tasks.

2.2 Handling Distribution Shifts

Recent studies address non-stationarity in time series by adapting normalization or learning strategies at the instance or window level. Representative methods include instancewise [Ogasawara et al., 2010] learnable input normalization [Passalis et al., 2019], and reversible instance normalization (RevIN [Kim et al., 2021]), which removes nonstationary components during training and restores them during inference. Dish-TS[Fan et al., 2023] proposes a general neural paradigm that mitigates distribution shift by employing a dual framework to separately model the distributions of the input and output spaces. FAN[Ye et al., 2024] performs normalization by leveraging frequency-domain information. Subsequent work extends these ideas through slice-based normalization, such as SAN [Liu et al., 2023b], where each slice is defined within the look-back window and the forecast horizon. However, as these methods focus on modeling distribution shifts between the look-back and forecast windows, they exhibit strong adaptability to short-term fluctuations but limited generalization under prolonged or regime transitions within the dataset, as they overlook the discrepancies induced by variations across training instances.

3 Preliminaries

3.1 ProblemDefinition

The primary task addressed in this paper is long-term, multivariate time series forecasting. Given a historical multivariate time series, the goal is to learn a forecasting function f(·) that accurately predicts a sequence of future values based on a window of past observations.

Formally, let the input be a look-back window of length L, denoted as x = (x1, x2, . . . , xL), where each multivariate observation xt ∈ Rd is a d-dimensional vector at time step t. The model must learn a function f that maps the input window to the subsequent H time steps, where H is the prediction horizon:

Here, ˆy = (ˆ y1, ˆ y2, . . . , ˆ yH), where each ˆ yt ∈ Rd , is the predicted sequence. The objective is to minimize the discrepancy between the prediction ˆy and the ground truth future sequence

The fundamental challenge we address is dataset-level distribution shift (DDS), where a long-term time series is composed of multiple, heterogeneous operational regimes. Unlike local non-stationarity, DDS reflects the structural evolution of the system over extended horizons. For instance, in ionospheric total electronic content(TEC) data, the decadal solar cycles introduce profound shifts in signal characteristics that span years; similarly, in network traffic, the transition from background maintenance to coordinated user surges represents a fundamental change in system behavior. Mathematically, a monolithic function f struggles in this scenario because the mapping between history and future is not invariant. Instead, the underlying inter-variable dependencies are dynamically reorganized as the system transitions between regimes. When a single model is forced to minimize a global objective across this mixture of distinct behaviors, it inevitably learns an "averaged" representation. This averaging effect obscures the specific coupling logic of individual regimes, leading to a loss of fidelity in capturing the precise dynamics required for robust long-term forecasting. To resolve this, it is necessary to identify these regimes and master their unique patterns via specialized kernels.

4 Methodology

4.1 Overall Framework

To address the challenge of dataset-level distribution shift, we propose NEST: Tackling Dataset-Level Distribution Shifts via Regime-Oriented Mixture-of-Experts. The framework is built upon the principle of regime-specific structural modeling, where the heterogeneous operational modes within a long-term time series are first identified and then masterfully captured by a specialized expert pool. Instead of learning a single averaged representation, NEST anchors its predictive logic in the distinct dynamical patterns inherent to different system states.

The NEST architecture comprises three key components:

• Unsupervised Regime Discovery: A principled module that partitions the dataset into functionally distinct regimes. By employing moment-entropy metrics, it characterizes each data segment through its statistical distribution and structural complexity, ensuring a robust mathematical basis for regime identification.
• Specialized Expert Pool: A heterogeneous ensemble consisting of specialized "regime experts" and a "shared expert". Each regime expert is trained to achieve structural specialization, evolving unique variate-attention
• patterns to capture the specific inter-variable dependencies of its assigned regime, while the shared expert retains global, overarching knowledge.
• Regime Oriented Router: A dynamic gating network that orchestrates the expert pool through a functional division of labor. The router achieves precise expert selection by integrating content-based temporal regime alignment with geometric contextual modulation, enabling the model to accurately recompose specialized patterns in response to evolving system dynamics.

The NEST pipeline, illustrated in Figure 2, follows a twophase process. Phase 1 consists of an unsupervised regime discovery step followed by the training of a specialized expert pool on the discovered data partitions. In Phase 2, these experts are frozen, and the router is trained to adaptively aggregate their predictions for context-aware forecasting.

4.2 Unsupervised Regime Discovery via Data Clustering

The foundational step of our NEST framework is the automated identification of distinct operational regimes within the time series dataset. This is achieved through an unsupervised clustering process performed as a pre-processing step before the main model training.

First, we transform the training time series into a set of overlapping instances, or "slices". Each slice si, starting at time index i, is defined as a sequence of length SL = L+H, where L is the input window length and H is the prediction horizon:

Next, for each slice si, we extract a low-dimensional feature vector vi designed to capture its core dynamic properties. This vector includes: Moment Metric: We compute the mean (µ) and variance (σ2 ) of the slice, representing its central tendency and volatility. Entropy Metric: To capture intrinsic complexity, we compute the Singular Value Decomposition Entropy (SVDEn). This involves constructing a trajectory matrix X from the slice si with an embedding dimension d:

We then perform SVD on X to obtain its singular values {Xσj }kj=1 . These values are normalized to form a probability distribution pj = Xσj /∑kl=1 Xσl . The SVDEn is the Shannon entropy of this distribution:

The final feature vector for each slice is the concatenation of these metrics: vi = [µ(si), σ2 (si), SVDEn(si)]. With the training set represented as a collection of feature vectors {vi}, we apply K-Means clustering to partition the data into M clusters. The objective is to find the set of centroids C = {c1, . . . , cM} that minimizes the within-cluster sum of squares. Each centroid cm serves as a prototypical representation of its regime and is utilized by the router.

A crucial detail of our regime discovery process is the use of data slices spanning both the look-back window and the prediction horizon. This design choice ensures that the discovered regimes are defined by the complete dynamic behavior of a sequence, including its evolution into the future, thus creating more forecasting-oriented clusters. At inference time (i.e., during validation and testing), only the look-back window of length L is available. To assign an incoming sequence to a regime, we compute its feature vector (mean, variance, SVDEn) using solely this available L-length data. We then operate under the assumption that the statistical properties of the look-back window serve as a sufficient proxy to identify the most likely regime for the complete L+H dynamic. That is, from the historical look-back window to the forecasting window, the data distribution remains consistent, which can be ensured by the method discussed in Section 2.2. This procedure strictly avoids data leakage, as the ground truth future values are never used to compute features or determine regime assignment at inference time. The regime centroids are established exclusively from the training set, and the mapping during inference relies only on historical data.

4.3 The Multi-Expert Pool

The predictive core of the NEST framework is a heterogeneous pool of expert models, each designed to capture different facets of the time series dynamics. This pool contains two types of experts: specialized regime expert sets and a single, generalist shared expert.

Regime Experts: Corresponding to the M regimes discovered via moment-entropy metrics, the model contains M

specialized regime experts, denoted as {Em}Mm=1 . Each expert Em is responsible for learning the specific dynamic properties of its assigned regime m. To effectively capture the inherent pattern of the time series, each expert is implemented as a variate-attention module specifically optimized to capture regime-specific inter-variable dependencies. Given an input sequence x ∈ RL×C , where L is the look-back window and C is the number of variables, the expert focuses on the spatial correlations across the channel dimension:

This design ensures that each expert masters the dynamic reorganization of variables within its assigned regime. The resulting attention maps provide an explicit representation of how variable dependencies shift across different operational states.

Shared Expert: To capture global patterns common across all regimes, the pool includes a single shared expert, Es, which remains active across different data slices throughout Phase 1 training and employs variate attention to provide a stable, foundational forecast.

4.4 Regime-Oriented Router

The core of NEST's adaptability is the regime-oriented router, which dynamically orchestrates the expert pool through a two-step process to ensure stable and precise expert selection under complex distribution shifts.

Initial Weight Generation

The router first dynamically generates a preliminary weight distribution based on the input's temporal content. For the M regime experts, the input embedding xemb is processed by a lightweight Transformer encoder and a regime-specific linear head, yielding the preliminary weight distribution:

In parallel, the output of the same encoder is passed through a different linear head and a Sigmoid activation function to compute the weight of the shared expert, ws:

Using a Sigmoid function allows the model to treat the shared expert as an independent, additive component, without competing directly with the specialized regime experts for the same weight budget.

Geometric Contextual Modulation

To anchor the content-driven decisions within the global data manifold, we modulate winit using the input's proximity to regime centroids in the moment-entropy space. For an input slice si, we calculate the Euclidean distance dm between its moment-entropy metric vector vi and each regime centroid cm:

The distance is converted into a similarity weight ˜ωm via an inverse quadratic function. A softening factor α ∈ (0, 1] is applied to facilitate smooth collaboration during regime transitions:

The final adaptive weights w′m are produced by the elementwise modulation of temporal relevance and geometric context:

Final Prediction and Training.

The modulated scores are renormalized to produce the final regime expert weights wregime. A separate gate computes the weight ws for the shared expert. The final prediction ˆy is the weighted sum of all expert outputs:

The router training objective minimizes the Mean Squared Error (MSE) loss between the prediction ˆy and the ground truth y:

5 Experiments

5.1 Experimental Setup

Datasets. To evaluate NEST's ability to capture evolving variable dependencies across diverse domains, we conduct experiments on several large-scale benchmarks: CESNETTIMESERIES24 [Koumar et al., 2025]: A high-fidelity network traffic dataset containing 40 weeks of activity from over 275,000 IPs on an ISP network. Its high variability and heterogeneous behavioral modes present an authentic challenge for regime identification. We select five representative IP series for primary evaluation. Ionospheric TEC: A specialized physical dataset characterizing Earth's ionospheric dynamics which features decadal solar cycles and complex spatiotemporal relations. Public Benchmarks: To demonstrate generalizability, we include the Weather and ETT datasets(ETTh1, ETTh2), which are standard benchmarks for evaluating longterm forecasting robustness under varying non-stationarity.

Baselines. We compare our proposed NEST model against a comprehensive suite of SOTA and classical deep learning models for time series forecasting. The baselines include the unified foundation model UniTS[Gao et al., 2024]; Transformer-based models such as iTransformer[Liu et al., 2023a], PatchTST[Nie et al., 2022], FEDformer[Zhou et al., 2022], Autoformer[Wu et al., 2021], and Informer[Zhou et al., 2021]; and the MLP-based DLinear[Zeng et al., 2023]. This diverse set of models allows for a rigorous comparison against different architectural paradigms. We perform hyperparameter search for a fair comparison.

Implementation Details. To ensure a fair comparison, all models were trained and tested on a single NVIDIA V100 32GB GPU. We adopt the same strict chronological split for all benchmarking datasets: 70% for training, 10% for validation, and 20% for testing. We conduct our main experiments with a fixed lookback window of 512 time steps.

5.2 Main Results and Analysis

The main forecasting results are presented in Table ??, which provides a comprehensive comparison of NEST against all baseline models across nine datasets and four prediction horizons. The analysis of these results highlights the significant and quantifiable advantages of our proposed architecture in handling complex, non-stationary long-span time series datasets.

Superiority Across Diverse Benchmarks. NEST consistently achieves state-of-the-art (SOTA) performance, substantially outperforming all competitive baselines. As indicated in the "1st Count" row of Table ??, NEST secures the best MSE and MAE results in 32 out of 36 experimental settings across 9 distinct datasets. The performance gains are particularly pronounced on high-variability datasets; for instance, on the CESNET1 benchmark, NEST surpasses the strong baseline PatchTST by 5.49% and iTransformer by 14.82% in MSE on average across four different prediction horizons. This consistent dominance across multi-year physical cycles and heterogeneous network traffic validates NEST's capability to effectively track and model the dynamic reorganization of inter-variable dependencies, which is a structural challenge that monolithic models and local non-stationarity mitigators fail to address.

Robustness on Heterogeneous Network Traffic. NEST's efficacy is most prominent on the five large-scale CESNET benchmarks. Characterized by 40 weeks of real-world ISP traffic, these datasets exhibit extreme variability and regime transitions. NEST's superiority here stems from its systematic handling of non-stationarity: the unsupervised regime discovery module utilizes moment-entropy metrics to categorize data slices into functionally distinct modes, such as high-volatility surges versus stable, low-complexity periods.

5.3 Ablation Study on NEST Components

To quantify the individual contributions of the core components in NEST, we evaluate five variants: (1) the full NEST model; (2) w/o Router, which replaces the regime-oriented router with a simple parameter averaging of experts; (3) w/o Kmeans, which replaces the moment-entropy based clustering with naive sequential data partitioning; (4) w/o R&K, which removes both modules; and (5) Dist. Router, which utilizes only the distance-based modulation ˜ωm without the temporal initial weight score winit. The results in Table ?? demonstrate that the full NEST configuration achieves superior performance in nearly all metrics, validating the design of our regime-aware paradigm.

Efficacy of the Regime-Oriented Router: Removing the router(w/o Router) leads to a consistent performance drop, with MSE increasing markedly on datasets like ETTh1 (0.515 to 0.699). This underlines that simple expert averaging fails to capture the structural transitions between regimes. Furthermore, the performance gap between NEST and Dist. Router confirms the necessity of our routing strategy: while geometric distance provides a global anchor, the initial weight score winit is crucial for identifying the temporal state of the system.

Necessity of Moment-Entropy Based Discovery: The w/o Kmeans variant, which lacks principled regime partitioning, underperforms the full model across diverse benchmarks. For instance, on ETTh2, the MSE increases from 0.234 to 0.246. This justifies our use of moment-entropy metrics as a sufficient mathematical basis to discover functionally distinct operational modes, as naive partitioning fails to ensure the structural specialization of experts.

Synergistic Effect and Robustness: The w/o R&K variant generally yields the poorest results, highlighting the synergy between regime discovery and adaptive routing. The overwhelming trend across datasets confirms that the integration of principled regime discovery and regime-oriented routing is vital for robust forecasting under structural distribution shifts.

5.4 Hyperparameter Sensitivity Analysis

We evaluate NEST's robustness against key hyperparameters: model dimension (dmodel), attention heads (nheads), and regime count (M). As shown in Figure 3, NEST maintains remarkable stability across varying architectural configurations. Specifically, performance remains consistent for dmodel ∈ {64, 128} and nheads ∈ {4, 8, 16}, with slight MSE increases at dmodel = 256. This insensitivity suggests that NEST's efficacy is derived from its regime-aware logic rather than exhaustive parameter tuning. In contrast, the regime count M (Data Slices) exhibits a more data-dependent impact. Empirical results across multiple benchmarks suggest that M = 8 serves as a robust and versatile configuration, providing a sufficient hypothesis space for experts to capture the diverse behavioral modes within the momententropy space. At the same time, we find that a smaller M (e.g., M = 4) can occasionally yield competitive results on datasets with relatively low regime complexity. This indicates that while NEST benefits from a diverse expert pool to handle high dataset-level distribution shifts, it maintains a degree of parameter efficiency, allowing for a pruned architecture when the underlying system transitions are less intricate.

5.5 Study on Regime Experts

To investigate the internal mechanism of NEST, we conduct a multi-level interpretability analysis to verify the structural specialization of the expert pool and the efficacy of the regime-oriented router.

Visualizing Variate-Attention Map. We first visualize the Query-Key (QK) attention matrices of different regime experts to observe their focus on inter-variable dependencies. The attention patterns across experts exhibit heterogeneity, especially in the high attention score QK-pair vicinity. Each expert evolves a distinct variate-attention map that adapts to the input sample, representing a specific coupling logic of variables. This polymorphism confirms that experts have effectively specialized in different system dynamics rather than converging to a redundant averaged representation.

Quantifying Expert Dissimilarity via CKA. To further quantify the functional divergence between experts, we employ Centered Kernel Alignment (CKA) to measure the similarity of their attention matrices. As shown in Figure 4, the low pairwise CKA scores across the expert pool indicate significant representational dissimilarity. This high degree of heterogeneity proves that our moment-entropy based regime discovery successfully facilitates the emergence of non-redundant, specialized experts.

Expert Permutation and Router Sensitivity. To validate the necessity of precise expert orchestration, we conduct a weight permutation experiment. For each batch sample, experts with CKA similarity above a predefined threshold are eligible for pairing and routing score swapping, where pairs with higher similarity are prioritized for the swap operation. The results, summarized in Table ??, show a significant performance degradation across all test samples. Even when two experts appear relatively similar, the router's precise allocation remains critical. This "collapse" upon permutation demonstrates that the efficacy of NEST stems from the exact match between the detected regime and the expert's specialized variate-attention kernel, rather than mere parameter ensemble.

6 Conclusion

In this paper, we addressed the challenge of dataset-level distribution shifts by introducing NEST, a two-phase MoE framework that models the alternation of underlying regimes. By combining moment-entropy based regime discovery, structurally specialized experts, and a regime-oriented router, NEST decomposes heterogeneous temporal dynamics into reusable forecasting kernels and adaptively recomposes them for each input. Extensive evaluations demonstrate that NEST achieves SOTA performance across diverse benchmarks, particularly on highly non-stationary long-term datasets.

Acknowledgments

This work was supported by the grants from the Natural Science Foundation of China (U2541212, 62572035), and Beijing Natural Science Foundation (L248032). Thanks for the computing infrastructure provided by Beijing Advanced Innovation Center for Big Data and Brain Computing. Haoyi Zhou is the corresponding author.