FreqMoE: Dynamic Frequency Enhancement for Neural PDE Solvers

1 Introduction

Efficient solutions to large-scale partial differential equations (PDEs) play a crucial role in numerous scientific computing applications, ranging from weather forecasting through Navier-Stokes equations to quantum simulations in physics [Bi et al., 2022; Pathak et al., 2022; Li et al., 2023b; Childs et al., 2021]. As spatial resolution and temporal steps increase, traditional numerical solvers face prohibitive computational costs, spurring the development of neural approaches that promise to balance accuracy with efficiency.

1 The corresponding author is Haoyi Zhou (haoyi@buaa.edu.cn).

Among these approaches, Fourier neural operators (FNO [Li et al., 2021] and its variants (Geo-FNO [Li et al., 2023a], FFNO [Tran et al., 2023], TFNO [Kossaifi et al., 2023]) have emerged as particularly promising, leveraging the inherent sparsity of physical fields in frequency domain. By operating on a compact window of low-frequency signals while truncating higher frequencies, these methods achieve scale-free processing across arbitrary resolutions with reduced computational complexity. However, this frequency truncation presents a fundamental trade-off: while enabling computational efficiency, the loss of high-frequency information can significantly degrade performance in high-resolution scenarios and, as a result, accumulate errors in long-term predictions [Lippe et al., 2023; Cao et al., 2024].

Recent efforts to overcome high-frequency limitations have explored post-training refinement strategies. These methods aim to recover truncated frequency information through various approaches, such as diffusion-based iterative refinement [Lippe et al., 2023] and numerical solver guidance [Cao et al., 2024]. However, existing solutions often incur substantial computational overhead or are restricted to specific scenarios, highlighting the need for a more computationally efficient approach.

To address these limitations, we propose FreqMoE, a lightweight post-training framework inspired by upcycled MoE Models [He et al., 2024; Zhang et al., 2024]. This framework enables FNO trained on low-frequency domains to adapt to high-frequency signals dynamically. Our method leverages a pre-trained FNO as a base expert for lowfrequency components while initializing specialized highfrequency experts(Fig.1). To handle the inherent sparsity of high-frequency components, we incorporate a gating mechanism to selectively activate the most relevant highfrequency Experts during prediction. Our approach is motivated by a fundamental observation in physical systems: high-frequency signals typically exhibit strong dependencies on low-frequency components, as exemplified by the energy cascade phenomenon in fluid mechanics [McKeown et al., 2023]. Capitalizing on this physical insight, we initialize high-frequency experts re-using the base expert's weights through a LoRA-like strategy [Hu et al., 2022].

Specifically, we decompose the high-frequency expert's weights into two components: a shared base weight Rbase and a low-rank delta weight ∆R. For the i-th high-frequency Expert, its weights are constructed as Ri = R + ∆Ri, where the shared base weights R could be intialized by the pretrained dense FNO. This architecture offers significant computational efficiency improvement through two key design choices: (1) during post-training, low-rank delta weights are parameter-efficient, and (2) during inference, only the Top-K selected experts participate in prediction. The extremely lowrank nature of ∆R ensures minimal computational overhead in both stages, making our method particularly practical for real-world applications.

Through extensive evaluation on both regular and irregular grid PDEs, we demonstrate the effectiveness of FreqMoE in high-resolution and long-term prediction scenarios. Our experiments reveal compelling advantages: in highresolution tasks (512×512), FreqMoE achieves up to 16.6% accuracy improvement while using merely 2.1% parameters (47.32× reduction) compared to conventional FNO. This efficiency extends to unstructured meshes, where FreqMoE maintains superior performance with 27.37× parameter reduction. Furthermore, long-term rollout experiments showcase FreqMoE's stability in mitigating error accumulation, particularly in challenging high-resolution scenarios.

The key contributions of this work are threefold:

1. We propose FreqMoE, a lightweight post-training framework that dynamically enhances high-frequency processing capabilities in neural PDE solvers. Our approach generalizes seamlessly across the FNO family on both structured and unstructured grids, establishing an efficient "low-frequency pretraining, high-frequency fine-tuning" paradigm.
2. Inspired by physical principles of frequency dependencies in PDEs, we develop a LoRA-based expert initialization scheme that efficiently reuses low-frequency
3. weights. This design achieves remarkable parameter efficiency (47.32× reduction) while maintaining competitive performance through sparse dynamic computation.
4. Through comprehensive evaluation on diverse PDE systems, we demonstrate that FreqMoE significantly outperforms conventional FNO variants, achieving up to 16.6% accuracy improvement in high-resolution tasks (512×512) and superior stability in long-term predictions, all while maintaining minimal computational overhead.

2 RelatedWorks

2.1 Fourier Neural Operators

Fourier Neural Operators (FNO) [Li et al., 2021] have revolutionized PDE solving by introducing FFT-based spectral convolution layers to learn mappings between infinitedimensional function spaces. This foundational work has sparked numerous architectural innovations: Geo-FNO [Li et al., 2023a] and SFNO [Bonev et al., 2023] extended the framework to handle irregular grids and spherical geometries, while F-FNO [Tran et al., 2023] enhanced scalability through separable spectral convolutions and advanced training strategies. T-FNO [Kossaifi et al., 2023] further improved parameter efficiency and generalization by implementing global tensor decomposition. Despite these advancements, the issue of high-frequency truncation-a critical limitation in FNO-remains largely unaddressed. Our work directly tackles this gap by enhancing high-frequency signal processing capabilities, offering a complementary approach that integrates seamlessly with existing FNO architectures to further improve performance.

2.2 Sparse Upcycling Techniques

Sparse upcycling has emerged as a powerful paradigm for efficient model enhancement, leveraging sparsely activated Mixture-of-Experts (MoE) initialized from pre-trained dense models. This approach has demonstrated remarkable success across diverse domains, from language models (T5) [Komatsuzaki et al., 2023] to vision-language systems (MoELLAVA) [Lin et al., 2024] and medical applications (MoEMed) [Jiang et al., 2024], consistently outperforming sparse models trained from scratch while significantly reducing computational costs. Building on these principles, our work pioneers the application of sparse upcycling to frequencydomain learning. We introduce a novel framework that utilizes pre-trained FNO to efficiently enhance high-frequency components, achieving improved performance with minimal additional training overhead.

3 Method

FreqMoE extends FNO with dynamic frequency processing through a lightweight expert system. As shown in Fig. 2, our framework splits the frequency spectrum into chunks and processes high-frequency components via specialized experts derived from pre-trained FNO. We first introduce the frequencydomain MoE design (Sec. 3.2), then present our efficient expert initialization scheme (Sec. 3.3), followed by the training strategy that enables sparse computation. We begin by reviewing the basics of FNO and MoE systems.

3.1 Preliminary

Neural PDE Solvers with FNO. Fourier Neural Operator (FNO) learns a parameterized operator Gθ that maps input functions to output solutions in infinite-dimensional spaces. The core of FNO is its Fourier layer(Fig.2(a)), which performs spectral convolution through: K(l) (z(l) ) = IFFT(R(l) · FFT(z)), where R(l) ∈ CH×H×M1...Md are learnable weights operating on truncated frequency modes {M(i)|i ∈ {1, 2, ...d}}. This frequency truncation, while computationally efficient, leads to information loss in highfrequency components.

Mixture-of-Experts (MoE). A standard MoE layer consists of a gating network Pθ and N expert networks Eθj , comTopK(Softmax(Pθ(x)j)) ·

E(j) θ (x). In our frequency-domain adaptation, experts specialize in different frequency chunks, with the gating network determining the activation of high-frequency computations during inference.

3.2 MoE in Frequency Domain

Traditional FNO truncates high frequencies for efficiency, but this fixed cutoff limits model capacity. Our FreqMoE design addresses this limitation by adaptively processing the frequency spectrum based on two observations: high-frequency signals in PDEs are naturally sparse, and their patterns are often localized. These properties make the frequency domain particularly suitable for expert-based processing.

Frequency Domain Partitioning. In the standard spectral convolution, for an input feature map z ∈ RS , its Fourier transform ˆ z = FFT(z) ∈ CS is truncated to retain only the lowest frequency bands for processing: oP = Rθ · ˆ zP , where Rθ represents the learnable weights. We generalize this fixed truncation scheme by partitioning the frequency spectrum into J = S/P bands: {ˆ z(i) P ∈ CP | i = 0, 1, . . . , J - 1} where bands are ordered by increasing frequency, with ˆ z(0) P containing the lowest frequency components.

Expert Specialization. We assign N specialized experts {Eθi | i = 1, . . . ,N} (N ≤ J -1) to process different highfrequency bands, while keeping the original FNO weights Rθ as the base expert for low-frequency components ˆ z(0) P . This design stems from a key insight in PDE solutions: low frequencies capture global patterns that require careful processing, while high frequencies reflect local details that can benefit from specialized, targeted handling.

Adaptive Frequency Gating. To exploit the natural sparsity in high-frequency signals, we design a gating mechanism gθ that selectively activates experts based on frequency con-

tent: gθ(ˆ z(i) P ) = σ( wθ·ˆ z(i) P τ ) where τ is a temperature parameter and σ is the sigmoid function. The forward computation follows:

To encourage sparse expert utilization during training, we add a sparsity loss on gate values:

This regularization pushes the model to activate only the most relevant experts for each frequency band, leading to more efficient inference.

Inference-Time Sparsity. During inference, we leverage the sparse nature of expert utilization by activating only the top-K experts (K ≤ N) based on their gating values: active experts = TopK(gθ(ˆ z(i) P )i = 1N ,K).

The inference computation becomes:

This sparse activation strategy significantly reduces computational overhead while maintaining model performance, as high-frequency components typically require selective rather than comprehensive processing. The final output in the spatial domain is obtained through the inverse Fourier transform: o = IFFT(o(i) P J-1 i=0 ), where the unprocessed highfrequency components are naturally zero-padded.

3.3 Sparsely Upcycle the Low-frequencyWeight

After establishing the expert structure, a key challenge is how to efficiently initialize these experts. Instead of training from scratch, we propose a sparse upcycling strategy (Algorithm 1) that leverages pre-trained FNO weights while keeping the parameter count low. This approach allows experts to inherit low-frequency patterns while developing specialized highfrequency processing capabilities.

Parameter-Efficient Weight Adaptation. For each expert Eθi , we decompose its adaptedweightsRθi into a shared base component and an expert-specific delta:

where Rθ ∈ CH×H×M1×...×Md represents the pre-trained weights in the low-frequency domain. The expert-specific adaptation ∆Rθi is computed through low-rank decomposition:

with A(i) ∈ Cr×H and Bθi ∈ CH×r×M1×...×Md being low-rank adaptation matrices with rank r ≪ H. This formu-

lation reduces the adaptation parameters fromO(H2 d∏i M(i))

3.4 Bridging Low and High Frequency Learning

FNO effectively addresses the challenge of learning in infinite function spaces, yet it primarily captures low-frequency patterns as resolution increases. This aligns well with PDE characteristics where dominant features reside in low frequencies, making it efficient to learn fundamental patterns. FreqMoE builds upon this insight by establishing a bridge between low and high frequencies through expert upcycling, enabling pattern transfer across frequency bands.

This design naturally leads to an efficient learning paradigm: Low-Frequency Pretraining, High-Frequency

Algorithm 1 Sparsely Upcycling of FNO

Input: Pretrained FNOModel F, number of experts N, rank r, scaling factor α

• Output: Upcycled FreqMoEModel 1: // Initialize expert parameters 2: for each Fourier layer l do 3: R(l) θ ← F.get pretrained weights(l) 4: Initialize gating network g(l) θ 5: for i in 1 to N do 6: Initialize expert R(l) θi ←R(l) θ +α · A(l) θi B(l) θi 7: end for 8: end for

• 9: return New FreqMoEModel F

Fine-tuning(LPHF). Since inference over high-resolution PDE solutions is computationally expensive, LPHF allows us to learn core patterns from abundant low-resolution data, then adapt to high frequencies with much fewer parameters. As demonstrated in our experiments on both regular (CFD) and irregular (AirFoil) grids, this paradigm significantly accelerates neural operator inference while maintaining high accuracy across resolutions.

4 Experiments

We conduct systematic evaluations of FreqMoE across two critical scenarios demanding effective high-frequency modeling: high-resolution inputs and long-term prediction rollouts. Our experimental framework follows a progressive approach: (1) Training base FNO models on low-frequency regimes, then (2) transforming them into sparse FreqMoE architectures through our parameter-efficient upcycling strategy (Section 3.3). We measure model effectiveness through activated parameter counts(# Params) and prediction accuracy (L2 relative error), with comprehensive ablation studies on frequency adaptation mechanisms.

4.1 Datasets

To demonstrate FreqMoE's versatility across different PDE domains and discretization schemes, we select benchmark problems from both regular and irregular grid settings.

Regular-grid PDEs. From PDEbench [Takamoto et al., 2022], we choose vortex-dominated flows under Random and Turbulent initializations. Evaluations at 128×128(CFD-Rand 128) and 512×512(CFD-Rand 512, CFD-Turb 512) resolutions test progressive frequency handling capabilities, where higher resolutions reveal finer turbulent structures.

Irregular-grid PDEs. Using Geo-FNO's [Li et al., 2023a] challenging scenarios: (1)Airfoil, transonic flows over parameterized NACA-0012 airfoils (Mach 0.8) with shock-induced high frequencies on adapted C-grids ( 200×50). (2)Elasticity, nonlinear material deformations with central voids (radius 0.2-0.4), modeled via 1000 FEM nodes capturing stress concentrations.

4.2 Baseline and Implementation

We evaluate FreqMoE against two strong FNO variants, with implementation details summarized below.

FNO Baselines. For regular grids, we implement vanilla FNO [Li et al., 2021] with four Fourier layers (width=32) under three spectral configurations: (4,4), (16,16), and (32,32) modes. Inputs include velocity components (Vx, Vy), pressure, and density fields. Training adopts single-step prediction with Adam optimizer (initial lr=0.001).

GeoFNO Baselines. For irregular grids, we extend GeoFNO [Li et al., 2023a] with task-specific designs:(1)Airfoil, asymmetric modes (2,4) to (16,32) capture shock waves, using 4 input channels (coordinates + physical fields). (2)Elasticity, symmetric modes (2,2) to (16,16) model stress concentrations, enhanced with polar coordinate encoding. Both variants employ the IPHI module for coordinate transformation, trained with 50% learning rate decay every 50 epochs.

FreqMoE Configuration. Our architecture introduces two key innovations: (1)Dynamic Expert Selection, expands spectral capacity from (4,4)→(32,32) for regular grids and (2,4)→(16,32)/(2,2)→(16,16) for irregular grids, activating only Top-2 experts during inference. (2) Upcycling Strategy, initializes weights from pre-trained base models via lowrank factorization (rank=4), contrasted with scratch training. Training stabilizes via expert sparsity loss (factor α = 0.1) with identical hyperparameters to baselines for fair comparison. This design achieves parameter efficiency while preserving high-frequency resolution - critical for our later analyses of activation patterns and long-term stability.

4.3 Comprehensive Evaluation and Insights

Our experiments systematically validate FreqMoE's capabilities through two analytical lenses: (1) The post-training performance improvement in relative L2 error. (2) The inference efficiency improvement via activated parameters reduction. Key findings reveal that FreqMoE achieves superior highfrequency modeling with 6-28× parameter reduction compared to dense counterparts, while maintaining robust performance in all scenarios.

High-Resolution Regular Grid Analysis. Our experiments reveal fundamental limitations in conventional FNO's frequency scaling approach. As shown in Table 1, naively expanding FNO from (4,4) to (32,32) modes yields diminishing returns - while the 32×32 model achieves marginal gains on 128×128 resolution (4.81%→4.10% L2RE), it degrades performance on high-resolution CFD-Turb 512

scratch and upcycled from dense Geo-FNO, respectively. The bold values highlight our best performance.

(a) Frequency Signals (b) Experts Contribution

(24.45%→24.36%) despite 59× parameter growth. This exposes a critical tradeoff: dense spectral models overparameterize high-frequency components that rarely activate in practice.

FreqMoE breaks this tradeoff through dynamic expert specialization. With only 177.53K active parameters (47× fewer than (32,32) FNO), our sparse model reduces errors by 9.8%-16.6% across resolutions. The upcycled variant (4→32 modes) achieves particularly striking improvements: 20.6% error reduction on CFD-Turb 512 compared to its dense counterpart, demonstrating superior turbulence modeling. Spatial error maps in Figure 3 validate this behavior - while dense FNO accumulates errors in vortex cores (red circles), FreqMoE maintains accurate predictions through adaptive frequency allocation.This resolution-aware adaptation explains FreqMoE's dual advantage: preserving low-frequency stability (4.10%→3.70% on CFD-Rand 128) while capturing highfrequency details (24.36%→19.34% on CFD-Turb 512).

Results on Irregular-Grid PDEs. The challenges of irregular grids exacerbate conventional Geo-FNO methods' inefficiency in high-frequency processing. As Table 2

demonstrates, naively expanding Geo-FNO to (16,32) modes for AirFoil catastrophically degrades performance (L2RE surges from 0.0152 to 0.0708) despite 4.2M parameters - revealing dense models' vulnerability to spectral overparameterization. FreqMoE addresses this through sparse high-frequency specialization: with merely 148K parameters (28× fewer than (16,32) modes Geo-FNO), our model achieves near-identical AirFoil accuracy (0.0154 vs 0.0152) while reducing Elasticity errors by 5.2%. This efficiency stems from dynamic frequency enhancement - preserving critical high-frequency components around geometric discontinuities (airfoil edges in Figure 3) without parameter bloat. The 26.7× parameter reduction in Elasticity tasks particularly highlights FreqMoE's advantage in handling stress concentration areas where high-frequency signals dominate.

Sparsely Activation of Experts. As described in Section 3.2, FreqMoE dynamically activates high-frequency experts based on input signals through its gating mechanism. Figure 4 illustrates both the frequency distribution and expert activation patterns. The frequency visualization (Figure 4(a)) reveals that high-frequency components in PDE solutions exhibit natural sparsity, with signal energy primarily concentrated in the low-frequency region (top-left corner). The expert activation map (Figure 4(b)) demonstrates how FreqMoE's gating mechanism responds to this spectral characteristic - while maintaining consistent engagement with low-frequency experts, it selectively activates high-frequency experts only when corresponding signal components are present. This adaptive activation pattern suggests that FreqMoE can effectively identify the sparse high-frequency patterns while preserving computational efficiency through targeted expert utilization.

Analysis of Rollout Performance. The rollout experiments demonstrate FreqMoE's effectiveness in mitigating error accumulation during long-term predictions. In lowresolution scenarios (CFD-Rand 128), all models show relatively stable performance, with FreqMoE maintaining a slight edge in accuracy. However, the advantages of FreqMoE become substantially more evident in high-resolution cases. For CFD-Rand 512, while baseline FNO models exhibit rapid error accumulation regardless of their mode numbers, FreqMoE maintains a significantly lower error trajectory, with 31.67% reduction in final prediction error. This pattern is further amplified in the more challenging CFD-Turb 512 dataset, where turbulent flows introduce additional high-frequency components. Here, FreqMoE's adaptive frequency modeling capability proves particularly valuable, effectively containing error growth even as prediction steps extend. This performance gap suggests that FreqMoE's dynamic expert activation successfully preserves critical high-frequency information that traditional FNOmodels typically lose, thereby preventing the cascade of prediction errors in complex fluid simulations.

Scale up frequency Modes sparsely vs densely. Figure 6 demonstrates the trade-offs between model performance and computational cost when scaling frequency modes. Dense FNO shows initial error reduction from modes 4 to 12, but experiences performance degradation with higher modes due to the inherent sparsity of frequency signals. FreqMoE maintains steady improvement through dynamic expert selection. The inference time of dense FNO grows quadratically with modes due to full spectral convolution, while FreqMoE achieves linear complexity by fixing active experts (Topk=2), where modes only affect gating computation.

5 Conclusion

We presented FreqMoE, a dynamic frequency enhancement framework that addresses high-frequency signal loss in Fourier Neural Operators through a sparse mixture-ofexperts paradigm. Our "Low-Frequency Pretraining, HighFrequency Fine-tuning"(LPHF) strategy efficiently bridges frequency domains while maintaining remarkable parameter efficiency. Key innovations include: (1) a frequencydomain MoE architecture with dynamic expert activation, (2) LoRA-based weight initialization that recycles pretrained FNO weights, and (3) sparse upward-cycling training achieving 47.32× parameter reduction.

While our current approach uses predefined frequency partitioning and introduces minor routing overhead, future work will focus on adaptive partitioning strategies and extending the LPHF paradigm to broader operator learning scenarios. FreqMoE establishes a foundational framework for frequency-aware neural PDE solvers, opening new pathways for efficient high-resolution scientific computing.

Acknowlegements

This work was supported by the National Science and Technology Major Project(No.2022ZD0117800), and Young Elite Scientists Sponsorship Program by CAST(No.2023QNRC001). This work was also sponsored by CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ2023MindSpore12) and developed on openl community. Thanks for the computing infrastructure provided by Beijing Advanced Innovation Center for Big Data and Brain Computing.