FreSH: Frequency-Segmented Hierarchical Multi-Expert Framework for Multivariate Time Series Classification

1 Introduction

Multivariate time series classification has attracted significant attention due to its broad applications in healthcare [An et al., 2023], industrial equipment fault diagnosis [Farahani et al., 2023], and human action recognition [Li et al., 2023]. Accurate time series classification provides crucial support and insights for decision-makers. However, inherent properties of time series data, such as complex dynamics, noise, and class imbalance, make MTSC a particularly challenging task [Ismail Fawaz et al., 2019].

Traditional MTSC algorithms, such as DTW [Wang et al., 2017], primarily rely on feature statistics or signal processing techniques. As datasets become more complex, these methods struggle to scale to modern, high-dimensional time series and fail to generalize across diverse application scenarios [Ruiz et al., 2021]. Recently, deep learning has emerged as the dominant paradigm for MTSC. CNN-based methods, such as OS-CNN [Tang et al., 2020], excel in learning spatial hierarchical features through convolutional filters but are limited in comprehensively modeling global features, often requiring additional designs to compensate for this shortcoming. RNN-based [Karim et al., 2017] methods face challenges in capturing long-term dependencies due to vanishing gradients. Transformer-based models [Wen et al., 2022; Zuo et al., 2023] propose to handle long-range dependencies through self-attention mechanisms but fall short in extracting local pattern features at adjacent time points.

Despite the promising progress achieved by existing methods in time series classification, several key limitations hinder their performance and practicality. First, existing models often struggle to effectively capture the intricate, multiscale nature of time series data. They typically process either time-domain data directly [He et al., 2015] or a holistic frequency-domain [Yi et al., 2023], thereby failing to distinguish and analyze the distinct information carried by different frequency bands. The diversity of the MTSC dataset poses challenges for existing models in balancing differences between datasets and across categories. As shown in Figure 1, different models exhibit significant performance variations on different types of UEA datasets, and their accuracy is markedly affected by the number of samples in different categories, with performance improvements accompanied by considerable overhead. Furthermore, recent complex models, particularly those leveraging global self-attention mechanisms [Zhou et al., 2021], suffer from high computational costs and poor scalability, making them unsuitable for realtime applications or large-scale datasets. Simultaneously, these models lack the adaptability to specialize their processing based on the local characteristics of the data. Finally, conventional loss functions in MTSC, such as CrossEntropy [Wu et al., 2022a] and Focal Loss [Lin et al., 2017], present a dilemma: the former is often dominated by majority classes, while the latter can over-correct for difficult samples. This makes them suboptimal for handling the joint challenges of class imbalance and varying sample difficulty, which are common in real-world time series datasets.

Motivated by these challenges, we propose a FrequencySegmented Hierarchical Multi-Expert Framework for time series classification, i.e., FreSH. Our main goal is to go beyond a single-view approach by using the frequency domain and proposing a segmentation strategy that allows us to specifically analyze different frequency components. To fix the lack of flexibility and high costs of current models, we design an efficient and adaptive hierarchical multi-expert system. This architecture uses dedicated local experts for specific data segments while a global expert provides overall context, all within a lightweight framework that avoids complex and slow mechanisms like global attention. An adaptive gating mechanism fuses these outputs, dynamically weighting local frequency bands and global frequency spectrum. We also aim for a better optimization strategy by introducing a polynomial Loss as an alternative to standard loss functions, which we believe can create a better balance when optimizing for both easy and difficult samples, as well as majority and minority classes. Our major contributions can be summarized as follows:

• We propose a frequency-aware modeling paradigm that adaptively captures the multi-scale and multi-band characteristics of multivariate time series, offering a principled alternative to single-view time- or frequency-domain approaches for time series classification.
• We design FreSH, a lightweight frequency-segmented hierarchical multi-expert framework that enables adaptive specialization across frequency components by integrating local and global experts, achieving effective multi-scale representation with low computational overhead.
• Extensive experiments on 30 UEA benchmark datasets validate the effectiveness and generalization capability of our FreSH. Our method outperforms diverse state-of-the-art baselines while achieving advanced efficiency.

2 Methodology

2.1 ProblemFormulation

Let X = {Xi}n1 represent a multivariate time series dataset, where each sample Xi ∈ Rd×l represents the observations of d variables over l time steps, the goal of multivariate time series classification is to learn a classifier fθ to accurately predict the corresponding label, i.e., Xi ∈ Rd×l fθ -→ ˆ Yi ∈ Rc .

2.2 Framework Overview

We propose FreSH, a frequency-domain expert framework for multivariate time series classification, which integrates localized and global modeling within a mixture-of-experts network. The working pipeline is shown in Figure 2 (a). We conduct mixup for different samples to enrich the data diversity first, and then transform the data into the frequency domain and process it by a Hierarchical Frequency-Informed MoE (HiFiMoE) module, shown in Figure 2 (b). The output is fed to the prediction layer for classification.

2.3 Data Preprocessing

Given an input multivariate time series sample Xi ∈ Rd×l , where d is the number of variables and l is the sequence length, we first conduct mixup [Zhang et al., 2017] to enrich data diversity. We transform it into the frequency domain using the Fast Fourier Transform:

where s = ⌊ l2 ⌋ + 1. This frequency-domain representation allows the model to exploit periodic patterns and frequencyspecific information that are difficult to capture in the time domain. The resulting signal F is then zero-padded to a fixed length spadded to ensure divisibility and consistency across samples.

2.4 Frequency Segment Division

To capture the unevenly distributed information in the frequency domain, we divide the spectrum into multiple segments. This segmentation allows specialized experts to focus on specific bands, enabling more targeted feature learning and preparing for adaptive fusion later. Specifically, we divide the entire frequency spectrum into K equal-length segments:

where each segment F(k) ∈ Cd×lk corresponds to a specific frequency band, and lk×K = spadded. This segment division operation offers the opportunity for the specific utilization of individual frequency components that may carry unique patterns relevant for time series classification.

2.5 Hierarchical Frequency-InformedMoE

Multi-Segment Expert Module

For each frequency segment F(k) , we design a dedicated multi-segment expert module consisting of M parallel local experts {E(k) m (·)}Mm=1 , each implemented by a lightweight MLP. These experts are intended to capture diverse, potentially complementary representations of the intra-segment features. Specifically, the generated hidden representation of the m-th expert and k-th frequency segment H(k) m is:

When M = 1, the output of the single expert is directly used. For M > 1, the outputs of all experts are averaged to yield the representation of the k-th frequency segment H(k) :

H(k) processes a single frequency band F(k) and integrates the output results of M expert networks by averaging, thereby fully leveraging the collaborative advantages of multiple expert networks to significantly enhance overall performance. This design can balance expressiveness and computational efficiency, avoiding intra-segment gating while still allowing for ensemble effects among experts.

Global Expert Module

While segment-wise experts focus on local frequency bands, it is equally important to model global dependencies that span the entire frequency spectrum. To this end, we incorporate N global experts {Ui(·)}Ni=1 , each processing the complete frequency spectrum signal F ∈ Cd×spadded to extract holistic representations:

To adaptively weight the contributions of different global experts based on the input, we introduce a global gate

σglobal(·), which is implemented as a simple linear layer followed by a softmax:

where F is the input full-spectrum frequency representation, and λ provides the normalized weights for all N global experts. The final global representationHglobal is then computed as a weighted sum of all the experts:

where Hglobal ∈ Cd×spadded . This mechanism allows the model to dynamically adjust which global experts to emphasize, providing flexibility to adapt to varying signal characteristics.

Adaptive Gating Mechanism

Beyond global expert fusion, we also introduce a segmentlevel gating mechanism to adaptively combine segment-wise representations. The original full frequency domain information F is fed into a segment gate σsegment(·), which directly produces normalized weights β for each segment:

The local representation is then computed as a weighted combination of segment outputs:

where Hlocal ∈ Cd×spadded and βk provides the softmaxnormalized importance weights for segment k, enabling the model to dynamically balance contributions from different frequency bands.

We have achieved global frequency spectrum representation Hglobal and fused representation from individual frequency segments Hlocal. Then, we leverage a gate network σreweight for global representationHglobal to adaptivelymodulate the channel in the complete frequency spectrum. Finally, to yield a comprehensive representation, we combine the original frequency-domain signal F, the adaptively fused local representation Hlocal, and the adaptively fused global representation Hglobal :

Here, γ is a learnable scalar obtained through the local and global two-path features, and ˆα denotes the adaptive gating weight.

2.6 Prediction Layer

The final frequency-domain representationH∗ is transformed back into the time domain using the inverse FFT:

This reconstructed time-domain signal is passed through a fully connected classification layer and a softmax activation to produce class probabilities ˆ y:

where W and b represent the weight and bias of the linear layer. This end-to-end pipeline enables the model to predict the class label based on frequency-aware representations.

2.7 Optimization Objective Function

In MTSC tasks, data distributions are complex, class imbalance is common, and datasets differ significantly. Cross Entropy-Loss, which is widely used in prior work, struggles with class imbalance and fails to distinguish between easy and hard samples, leading to poor handling of minority classes and complex instances. To address this, we propose adopting Polynomial Loss [Leng et al., 2022] for time series classification.

Polynomial Loss was initially proposed to improve optimization and calibration performance in general classification tasks. However, the original Polynomial Loss introduces high-order terms and multiple hyperparameters, which may complicate the optimization process in MTSC scenarios. In this case, we simplify and customize it into an improved loss function called P-Loss.

We propose P-Loss to optimize the original formula by retaining only the core second-order adjustment term while preserving the stability and robustness advantages. The secondorder formulation is adopted because it provides nonlinear gradient corrections with negligible computational overhead, effectively optimizing the classification boundaries for multichannel data without introducing the noise associated with higher-order terms. Specifically, given N samples, where ˆ yi represents the predicted probability of the true class for the i-th sample, P-Loss is defined as:

The first term is the negative log-likelihood, maximizing the predicted probability ˆ yi, part of the standard cross-entropy loss. The second term is a second-order adjustment that penalizes deviations from the true label, with λP controlling its weight.

We also incorporate mixup with a simple adaptive adjustment. If the loss stagnates after several training rounds, the mixing ratio is dynamically reduced to bring the mixed samples closer to the original ones, improving generalization without harming baseline performance. Combined with P-Loss, this design simplifies optimization while preserving model stability and robustness, ultimately demonstrating stronger generalization on diverse MTSC benchmarks.

2.8 Computational Complexity Analysis

Most operations in FreSH, except for the frequency-domain transformation, are linear in the sequence length, which makes the framework computationally efficient. The overall computational complexity of a single forward pass through FreSH can be expressed as O(dn log n+mhn+ghn), where d is the number of channels (variables) in the multivariate time series, n is the frequency-domain vector length of the input, m is the number of experts per frequency segment, g is the number of global experts, and h is the hidden dimension of each expert. The O(dn logn) term comes from the FFT and inverse FFT transformations applied across d channels, which dominate for large n. The segment-wise expert networks contribute O(mhn), as each of the k segments independently applies m small MLPs on n/k elements. Similarly, the global experts contribute O(ghn) by processing the full n-dimensional vector through g MLPs. Since d, m, g, and h are small constants compared to n in practice, the computational complexity grows approximately as O(dn logn), making the model efficient and scalable for long multivariate time series.

3 Experiments

3.1 Experimental Setups

We conduct experimental comparison on the UEA MTSC benchmarks [Bagnall et al., 2018], which span applications such as human activity recognition, speech processing, medical EEG, and audio analysis. The datasets differ substantially in sequence length, dimensionality, and train/test sizes, enabling a robust assessment of generalization.

To systematically evaluate the effectiveness of our proposed FreSH, we conduct comprehensive comparisons against a wide range of state-of-the-art baselines.

On one hand, our main experiments focus on comparisons with dedicated multivariate time series classification methods. Specifically, we consider both traditional and recent MTSC approaches, including DTWD, ShapeNet [Li et al., 2021], TapNet [Zhang et al., 2020], TNC [Tonekaboni et al., 2021], TS2Vec [Yue et al., 2022], InfoMin [Tian et al., 2020], InfoTS [Luo et al., 2023], AutoTCL [Zheng et al., 2023], MPTSNet [Mu et al., 2025], and FreRA [Tian et al., 2025]. These methods are evaluated on 30 UEA multivariate time series datasets, providing a thorough and fair comparison with existing MTSC models.

On the other hand, to comprehensively position our proposed FreSH, we compare FreSH with representative time series representation models, including LSTNet [Lai et al., 2018], LSSL [Gu et al., 2021], TimesNet [Wu et al., 2022a], PatchTST [Nie et al., 2022], FlowFormer [Wu et al., 2022b], FEDformer [Zhou et al., 2022b], SCINet [Liu et al., 2022], DLinear [Zeng et al., 2023], Crossformer [Zhang and Yan, 2023], MICN [Wang et al., 2023], and ModernTCN [Luo and Wang, 2024]. Following the general setting [Mu et al., 2025], we conduct the comparison on 10 UEA datasets. All experiments are conducted on an NVIDIA A100. We report average accuracy (Avg. Acc.), average rank (Avg. Rank), and the number of best-accuracy datasets (Top-1 count) to evaluate MTSC performance and enable fair comparisons with baseline methods.

3.2 Experimental Results

Table 1 presents an extensive empirical evaluation of FreSH against 10 state-of-the-art Multivariate Time Series Classification (MTSC) baselines across a diverse collection of 30 UEA datasets. The quantitative results demonstrate the overwhelming superiority and robustness of our proposed method.

In terms of overall performance across all 30 datasets, FreSH establishes a new state-of-the-art benchmark. It achieves the highest average accuracy of 76.1% and the best (lowest) average rank of 3.4 among all compared methods. When compared to the second-best performer, FreRA, FreSH not only improves the average accuracy by 0.7% (76.1% vs.

75.4%) but also demonstrates greater consistency, surpassing FreRA by a margin of 0.6 in average ranking (3.2 vs. 3.8). This indicates that FreSH maintains high performance stability across varying data domains.

A deeper analysis of the specific 10 MTSC datasets (Table 2) further highlights the model's adaptability. In this challenging subset, the performance gap between FreSH and existing methods becomes even more pronounced. Compared to the strong baseline ModernTCN, which ranks second in both metrics, FreSH delivers a substantial improvement: it boosts the average accuracy by 1.3% and advances the average ranking by a full position (1.0).

Beyond the average metrics, the ranking distribution across datasets further confirms the core advantages of our method. In the evaluation of the complete UEA dataset, FreSH achieved the highest number of Top-1 results, significantly surpassing the next best model. Furthermore, in the evaluation setting across 10 datasets, FreSH achieved best accuracy on 5 datasets and second best accuracy on 4 datasets.

Overall, this outstanding performance across multiple metrics and settings demonstrates that FreSH is an excellent model capable of handling complex temporal patterns, significantly outperforming both traditional and recently proposed MTSC models, with remarkable adaptability and effectiveness.

3.3 Ablation Study

We conduct a comprehensive ablation study to investigate the contributions of the key components in our model.

• w/o-GlobalExperts: Remove the global experts and only rely on segment-wise experts for feature extraction.
• w/o-SegmentExperts: Remove the segment-wise experts, relying solely on global experts.
• w/o-P-Loss: Replace the P-loss with cross-entropy loss.
• w/o-Mixup: Disable the adaptive mixup data augmentation.
• w/o-β: Disable adaptive gating fusion for Segment experts.
• w/o-λ: Disable adaptive gating fusion for Global experts.

The ablation study confirms that all components of the FreSH framework are critical to its performance. The largest drop in accuracy occurs when removing P-Loss (76.1% to 72.9%), a strong indicator of its central role in achieving robust optimization for imbalanced and difficult samples. The hierarchical multi-expert framework is also essential, as removing either the global experts (76.1% to 73.3%) or the segment experts (76.1% to 73.1%) significantly degrades performance, validating the synergy between local feature analysis and global information integration. While the mixup augmentation also contributes, its removal leads to a comparatively smaller drop (76.1% to 74.6%), showing it is a valuable but supplementary component. Under the premise of retaining the linear expert, ablation experiments were conducted on the adaptive gated fusion of local (β) and global (λ) experts. Although the removal of this fusion module led to a decrease in average accuracy (2.7% and 1.8%), its performance still significantly outperformed the setting where all experts were removed simultaneously, with dual comparisons validating that the linear expert provides a stable baseline and the adaptive fusion further achieves effective collaboration among experts.

3.4 Efficiency Comparison

To assess the prediction precision and computational efficiency in realistic settings, we benchmark all models on a real-world vibration dataset. We report model parameter volume, average batch latency, total test time, and classification accuracy to characterize the trade-off between computational cost and predictive performance.

Table 4 shows large efficiency gaps. DLinear is extremely fast (0.66 ms/batch; 0.187 s total) but underfits vibration signals, yielding low accuracy (44.89%). In contrast, TimesNet, Crossformer, and other Transformer-based models achieve competitive accuracy at much higher cost (25.3-63.9 ms/batch). MPTSNet further demonstrates diminishing returns: despite strong capacity, its > 88M parameters lead to prohibitive latency (101.4 ms/batch; > 123 s total), limiting practicality. By contrast, our proposed FreSH achieves the best balance between performance and efficiency. With only 54,243 parameters, it remains fast (1.2 ms/batch; 0.344 s total) while achieving 94.37% accuracy.

These results indicate that FreSH effectively reconciles efficiency and accuracy, making it well-suited for real-world vibration analysis where both are critical.

3.5 Hyper-Parameter Analysis

We conduct quantitative experiments to evaluate the contributions of different components of FreSH. Figure 3 shows average accuracy on 30 UEA datasets under various hyperparameter settings. Figure 3 (a) tests the number of frequency segments. Figure 3 (b) examines experts per segment and analyzes global experts.

Experimental results shown in Figure 3 highlight the critical role of structural design in our framework. Specifically, segmenting the frequency spectrum into 3 bands strikes a balance between capturing fine-grained local patterns and preserving sufficient global context. Likewise, assigning 3 experts enables diverse feature extraction within each band without introducing unnecessary redundancy.

In contrast, overly fine segmentation fragments the spectrum and dilutes useful information, while too many experts per segment increases the risk of overfitting and leads to unstable representations.

These findings confirm that carefully calibrating the segmentation granularity and expert allocation is essential for effectively modeling both local and global dependencies in multivariate time series classification.

4 RelatedWork

4.1 Time-DomainMTSCMethods

A substantial portion of multivariate time series classification research has concentrated on modeling signals directly in the time domain. Notably, the MC-DCNN [Zheng et al., 2014] applies one-dimensional convolutions to capture intervariable relations, pairing them with fully connected layers for classification purposes. Building on this, the Multiscale Convolutional Neural Network (MSCNN) designs convolution kernels of multiple sizes to extract multiscale features [Cui et al., 2016]. Hybrid architectures have also been explored, such as LSTM-FCN [Karim et al., 2017], which leverages LSTM layers for short- and long-term dependency capture while CNN layers extract salient time series patterns. Its enhanced version MLSTM-FCN [Karim et al., 2019] further refines this combination for performance gains. More recent developments like TimesNet [Wu et al., 2022a] disentangle complex temporal variations into intraand inter-period components for improved local and global feature modeling, whereas MS-GNet [Cai et al., 2024] integrates graph convolution for inter-series correlation alongside multi-head attention for intra-series feature learning. Likewise, PatchTST [Nie et al., 2022] partitions sequences into local patches to capture hierarchical patterns, while Autoformer [Wu et al., 2021] proposes a novel auto-correlation mechanism that captures long-range dependencies by calculating the periodic similarity of time series.

Compared with this line of methods, our FreSH's hierarchical multi-expert architecture allows us to specifically extract the crucial frequency characteristics, achieving superior computational efficiency and adaptability.

4.2 Frequency-DomainMTSCMethods

In parallel, a growing body of work has sought to harness frequency-domain representations for MTSC, leveraging spectral analysis to uncover new optimization pathways. For instance, the Frequency-improved Legendre Memory Model [Zhou et al., 2022a] augments Legendre memory structures with spectral components, markedly improving long-term sequence classification. CrossFormer [Zhang and Yan, 2023] unifies frequency-domain decomposition with Transformer architectures to jointly refine local and global feature extraction, while MPTSNet [Mu et al., 2025] explicitly utilizes amplitude information to detect salient periodicities for rapid feature localization. Other methods bypass the time domain entirely: FreTS [Yi et al., 2023] demonstrates the compactness of spectral information and constructs a frequency-domain MLP to achieve state-of-the-art performance, and FreRA [Tian et al., 2025] proposes a parameterized augmentation strategy with time-frequency consistency constraints for robust training.

Existing frequency-domain methods typically perform a single, global analysis of the entire spectrum, which overlooks the unique information contained in different frequency bands. Our FreSH addresses this by segmenting the spectrum and using an adaptive expert system to process each segment individually, enabling a fine-grained and specialized analysis of each frequency band.

5 Conclusion

To advance multivariate time series classification, we propose FreSH, which combines frequency-domain analysis with an adaptive expert system.

FreSH transforms time series into the frequency domain and applies spectral segmentation: segment experts capture band-specific features, global experts model full-spectrum context, and adaptive gating fuses them into a more balanced representation.

FreSH further introduces P-Loss to address the limitations of cross-entropy and focal loss, and incorporates an adaptive mixup strategy to improve robustness and generalization.

Extensive experiments on a broad range of UEA benchmarks and a real-world vibration dataset demonstrate that FreSH consistently achieves superior or competitive accuracy compared to state-of-the-art methods, while maintaining high computational efficiency and a compact parameter footprint. The strong performance across diverse datasets underscores the framework's practical applicability and robustness, highlighting its potential for real-world deployment scenarios where both accuracy and efficiency are essential.

Acknowledgements

This work was supported by Jilin Province Industrial Key Core Technology Tackling Project (20230201085GX). Zijian Zhang is supported by the China Postdoctoral Science Foundation (2025M771587) and the Open Funding Programs of State Key Laboratory of AI Safety (2025-09). Hao Miao is supported by SCRI, The Hong Kong Polytechnic University (No. Q-CDDG). Qingliang Li is supported by the National Natural Science Foundation of China(42575159, 42275155, 62206028).