Learning Heterogeneous Global Local Frequency Dependencies in Diffusion-Based Image Compression

1 Introduction

Image compression aims to minimize redundancy in data representation while preserving the key information contained in images as much as possible, thereby enabling high fidelity reconstruction [Bellard, 2014; He et al., 2022; Mentzer et al., 2020]. In recent years, diffusion models [Sohl-Dickstein et al., 2015; Ho et al., 2020] have achieved substantial progress in tasks such as super resolution reconstruction [Saharia et al., 2022; Ma et al., 2023] and image inpainting [Xia et al., 2023]. However, during image compression, diffusion models often prioritize perceptual consistency over pixel level accuracy [Li et al., 2024d; Lei et al., 2023], which may lead to suboptimal fidelity at the patch level, particularly in terms of frequency dependency. Therefore, achieving a better trade-off among bitrate, distortion, and perceptual quality remains a key research problem.

†Corresponding authors.

Unifying low distortion and high perceptual quality is primarily challenged by the stable reconstruction of high frequency details after compression. Frequency dependency describes structured correlations in the frequency domain, indicating that they are not independent in distribution or structure [Li et al., 2024a; Luan et al., 2025]. Specifically, global frequency dependency characterizes the energy distribution of the entire image. Local frequency dependency reflects how frequency components within a patch jointly determine edges, textures, and their coupling with content semantics. In compression, high frequency energy is low and is easily suppressed by quantization [Wallace, 1991]. Without explicit constraints on frequency dependency, reconstruction can produce inconsistent cross frequency compositions. This shifts the global energy distribution and damages local structural consistency.

In this paper, we analyze the frequency components of natural images and obtain two key observations. First, as shown in Fig. 1(a) and Fig. 1(c), the Fourier transform exhibits strong separability between magnitude and phase. Exchanging the phase between different images markedly changes structural details, while changing the magnitude mainly affects overall brightness and contrast. This phenomenon suggests that compression degradation more readily perturbs phase(structure), while magnitude remains relatively stable. Therefore, frequency restoration should be modeled from two complementary perspectives, intensity and structure. The intensity prior can be used to guide the recovery of structural details. Second, as shown in Fig. 1(b), cross frequency correlations exhibit pronounced heterogeneity across global and local scales. At the whole image level, frequency relationships tend to be weakly correlated. In contrast, within local patches, the coupling strength across frequencies varies with content. It becomes more evident in texture rich regions and is weak or unstable in flat areas. Existing reconstruction methods often focus on global context modeling, or only small scale details. They implicitly assume that different regions share similar frequency relationships. This prior assumption can lead to spectral energy shifts during texture and structure reconstruction, which ultimately produces artificial textures or distortions.

Based on these observations, we propose FMDiff, a Fourier Mamba guided diffusion-based image compression framework that explicitly models global spectral energy distribution and patch level cross frequency coupling. At the core of FMDiff is FMBlock, which performs cooperative modeling with a dual branch design in the spatial and frequency domains. In the frequency branch, we transform features into the Fourier domain and decouple the complex spectrum into magnitude and phase. Magnitude learns global spectral energy relationships and suppresses compression induced energy shifts. Phase encodes structural representations. We then feed them into Mamba separately. Its scanning mechanism captures cross frequency coupling within each patch and aggregates global frequency context along the scanning order. This enables unified encoding of local and global dependencies. We further apply magnitude phase interactive modulation so that the intensity prior can adaptively guide the recovery of global spectral energy and local structural details. This improves the realism and stability of textures. Meanwhile, the spatial branch leverages Mamba long range modeling to extract global semantic context. This context is injected into the frequency branch as conditional information, providing controllable semantic constraints for high frequency restoration. In addition, we introduce learnable scaling factors at skip connections to adaptively balance local details and global consistency. Finally, features from both domains are unified during diffusion denoising, leading to a better tradeoff between distortion and perceptual quality.

In summary, our contributions are as follows.

• We analyze dependencies among different frequency components and show that they exhibit pronounced global and local heterogeneity. Global frequency dependency constrains the energy distribution over the entire image. Local dependency determines the consistency of internal textures and semantics. Ignoring this difference can cause spectral energy shifts and local structural imbalance.
• We design a Fourier state space module that comprehensively captures both local and global frequency dependencies. The spatial branch analyzes complex spatial details while also capturing global semantics. The frequency branch decouples Fourier representations and modulates, then interacts them to recover degraded information across the full frequency band.
• Extensive results show that our method achieves better perceptual quality while also attaining lower distortion.

2 RelatedWork

2.1 Lossy Image Compression

Image compression is essential for storage and transmission. Traditional codecs such as VVC [Bross et al., 2021] remain widely used, but their block-based designs often overlook spatial dependencies. Balle[Ball´e et al., 2016] introduced end to end learned compression with hyperpriors, while later works enhance coding and context modeling, e.g., ELIC [He et al., 2022], LIC-TCM [Liu et al., 2023], CCA [Han et al., 2024], and FTIC [Li et al., 2024b]. However, most of these methods primarily optimize rate and distortion, which can lead to perceptual artifacts and a distribution shift from natural images. GAN-based approaches [Mentzer et al., 2020; Muckley et al., 2023] enhance perceptual quality but may sacrifice fidelity. More recently, diffusion models [SohlDickstein et al., 2015; Ho et al., 2020] have shown strong perceptual performance via iterative denoising. CDC [Yang and Mandt, 2023] replaces conventional decoders with conditional diffusion models for perceptual reconstruction, while Text+Sketch [Lei et al., 2023] and DiffEIC [Li et al., 2024d] leverage Stable Diffusion for conditional or pretrained generation from compressed representations. However, these methods largely operate in the spatial domain and leave frequency dependencies underexplored, motivating us to revisit diffusion image compression from a frequency perspective.

2.2 Frequency Domain Learning

The Fourier transform is a classic technique in signal processing. Recent studies show that frequency domain representations have been widely used in low light enhancement [Huang et al., 2022], medical image segmentation [Chen et al., 2024a], image restoration and feature enhancement [Chen et al., 2025; Dong et al., 2022; Dong et al., 2024], and network optimization [Yin et al., 2019]. For example, DeepRFT [Mao et al., 2023] performs frequency selection in the Fourier domain to capture global structure and restore details at multiple scales. FourLLIE [Wang et al., 2023] and DMFourLLIE [Zhang et al., 2024] further adopt multi stage strategies to jointly recover illumination and details. However, research that explicitly investigates dependencies among different frequency components remains relatively limited.

2.3 State Space Models

In recent years, state space models (SSMs) have gained attention in vision for their strong long range modeling ability and linear complexity. Mamba [Gu and Dao, 2024], enabled by selective scanning, has been applied to medical segmentation [Wu et al., 2025], video understanding [Li et al., 2024c], and object detection [Chen et al., 2024b]. For low level vision, MambaIR [Guo et al., 2024] addresses local forgetting and channel redundancy via local enhancement and channel attention, while UVM Net [Zheng and Wu, 2024] integrate Mamba with convolution or dual branch designs to capture both local and global cues. Compared with existing methods, our approach explicitly models both global spectral statistics and patch level cross frequency coupling. It further improves spectral correction and stabilizes texture recovery through magnitude phase interactive modulation.

3 ProposedMethod

3.1 Overall Network Architecture

As shown in Fig. 2, we illustrate the overall architecture of the proposed FMDiff network in this chapter. First, the analysis transform ga maps the input image x to a latent representation y with 256 channels. The model then applies a hyper analysis transform ha to y to obtain the hyper latent representation z, which is fed into the hyper decoder. A joint auto regressive and hierarchical prior model is adopted to predict the entropy parameters (µ, σ) of the latent representation y, thereby modeling the probability distribution of the quantized latent ˆ y and performing entropy estimation. Finally, ˆ y is passed through the synthesis transform gs to produce the conditional features xc required by the diffusion model.

During the reverse diffusion stage, the noisy image xt at the current timestep and the conditional features xc are jointly fed into a control module for feature alignment and fusion. The encoder each stage is built upon a Dual branch fusion(DBF) module to simultaneously model global and local correlations in the frequency domain. Specifically, each DBF module contains two Fourier Mamba blocks, together with a linear attention layer and a downsampling layer. Finally, we employ DDIM sampling [Song et al., 2020] to obtain the reconstructed output ˆx by iteratively updating from the initial noise over multiple steps.

3.2 Frequency Branch

Existing studies based on the Fourier Mamba [Zou et al., 2024; Li et al., 2024a] implicitly assume that different regions share similar frequency relationships. However, we observe that frequency dependencies are highly heterogeneous. At the whole image level, frequency statistics are weakly correlated and easily averaged out. Within local patches, cross frequency coupling is stronger in textured regions and weaker in flat areas. To explicitly model spectral information and enhance high frequency recovery, we combine Mamba with the Fourier transform and build a Fourier Mamba module. It consists of a frequency branch and a spatial branch. As illustrated in Fig. 2, the network architecture of the frequency branch is shown in the middle right part. Given a feature map X ∈ RC×H×W , we apply the fast Fourier transform to obtain its spectrum:

Here, Ax denotes the magnitude, which represents the spectral intensity distribution. Px denotes the phase, which directly determines structures and textures. The degradation introduced by compression can be viewed as a perturbation to F. It can be reflected as magnitude loss and phase shifts. Therefore, instead of learning an difficult mapping directly in the frequency domain, we model Ax and Px separately. This design allows the model to learn with two explicit objectives, correcting energy imbalance and mitigating structural shifts. Furthermore, to capture cross frequency coupling within local regions, we partition and serialize Ax and Px according to the frequency domain arrangement and feed them into Mamba Block separately. Specifically, we divide Ax and Px into 8×8 patches with stride 8.

To mitigate the directional bias introduced by one dimensional sequence flattening, we encode the frequency domain token sequence using row and column directional serpentine scanning. Intuitively, Mamba's scanning mechanism can establish long range dependencies along the sequence. This allows it to encode the coordination among frequency components within each local patch and also absorb global spectral context across patches. As a result, it can capture inconsistencies between local and global frequency patterns. Next, reconstructing directly using only A′x and P ′x can easily introduce irrelevant spectral changes. Here, we treat the frequency domain residual as the model's correction for compression degradation:

A larger residual magnitude indicates that the corresponding frequency component is more severely corrupted. Based on this intuition, we use channel attention and global pooling to compress the residual into a channel wise gating vector, which is used to produce adaptive restoration:

Here, pool denotes global average pooling, CA denotes channel attention, and σ denotes the Sigmoid. The g controls channel selection and modulation strength in the spectrum. Moreover, magnitude and phase are not independent. Recovering realistic textures often requires them to remain consistent in local frequency compositions. To this end, we adopt symmetric interactive modulation. The gA modulates the phase, and the gP modulates the magnitude. This yields a restoration mechanism in which the two branches mutually constrain and guide each other:

Here, ⊙ denotes element wise multiplication. Finally, we apply the inverse Fourier transform to the restored magnitude and phase to obtain the output features.

3.3 Spatial Branch

Although Mamba provides efficient long range dependency modeling for sequences, directly flattening two dimensional features into a one dimensional sequence introduces a structural issue. Local neighborhoods in two dimensions no longer remain adjacent under one dimensional indexing. Pixels that are adjacent in 2D may be far apart in the 1D scanning order. This makes state propagation in the state space model favor correlations along the scan path, which weakens true local pixel coupling. This leads to a typical inconsistency, global context can be modeled, but the details required for local textures are insufficient. As a result, feature alignment between global semantics and local details becomes unstable.

To this end, we follow prior work such as [Zheng and Wu, 2024; Guo et al., 2024], we introduce a learnable scaling factor in the spatial branch as a local consistency correction term. The goal is to retain Mamba's global modeling capability while adaptively adjusting the local information during skip fusion. Specifically, as illustrated in the upper part of Fig. 2(b), given a feature tensor X ∈ RC×H×W , we first apply normalization and feed it into Mamba Block to obtain a global context representation:

Here, G mainly represents global semantics. It is used to provide semantic constraints for frequency domain restoration. Next, to explicitly compensate for the local neighborhood relationships, we generate a scaling gate S in the residual branch. It modulates the strength of local information injected at the skip fusion stage:

Here, S ⊙X denotes adaptive reweighting of local representations. In textured regions, the gate tends to strengthen local information to mitigate structural distortion. In flat regions, it suppresses local responses to avoid noise. Finally, we introduce channel attention to recalibrate channel responses. The state space model first produces diverse candidate channel representations.

4 Experiment and Analysis

4.1 Experimental Setup

Implementation details: In the warm up stage, the Lagrangian multiplier is fixed. To obtain different bitrates, we set β to 0.0032, 0.0064, 0.0128, 0.0512, 0.1024, and 0.2048. We warm up FMDiff for 500K steps with a learning rate of 10-5 , then train for another 1000K steps with exponential decay, reducing the learning rate by 20% every 100K steps. The batch size is 4. For diffusion, we use 8193 timesteps with a cosine variance schedule. At inference, we adopt DDIM sampling [Song et al., 2020] with 65 steps.

Datasets: We train on the public Vimeo-90K dataset [Xue et al., 2019] with 90,000 clips, each containing seven 448×256 frames. For evaluation, we use Kodak [Franzen, 2013], DIV2K [Agustsson and Timofte, 2017], and CLIC2020 [Toderici et al., 2020]. Kodak includes 24 images at 768×512, while DIV2K and CLIC2020 contain 100 and 428 high resolution images, respectively.

Metrics: For quantitative evaluation, we measure perceptual quality using DISTS [Ding et al., 2020], PIEAPP[Prashnani et al., 2018], FID[Heusel et al., 2017], and LPIPS[Zhang et al., 2018]. We also report distortion metrics, including PSNR and MS-SSIM [Wang et al., 2003], and use BPP to evaluate compression performance.

4.2 Comparisons With State-of-the-art Methods

To comprehensively evaluate the proposed FMDiff, we compare it with representative state-of-the-art image compression methods. This set includes conventional standard codecs such as BPG [Bellard, 2014] and VVC [Bross et al., 2021], VAEbased methods such as ELIC [He et al., 2022], LIC-TCM [Liu et al., 2023], CCA [Han et al., 2024], and FTIC [Li et al., 2024b], as well as HiFiC [Mentzer et al., 2020], which uses GAN-based generation to improve perceptual quality. In addition, we include CDC [Yang and Mandt, 2023], a diffusionbased image compression method, as an important baseline in the diffusion family.

1) Quantitative Comparisons: Figs. 3 and 4 show rate-distortion curves on Kodak, DIV2K, and CLIC2020. Fig. 3 reports perceptual metrics, while Fig. 4 reports fidelity metrics. Following CDC, we evaluate two settings: a distortion oriented FMDiff with λ = 0 and a perception oriented FMDiff with λ = 0.9, prioritizing fidelity and perceptual quality, respectively.

Fig. 3 plots rate-perception curves on the three datasets. FMDiff consistently outperforms BPG, VVC, ELIC, CCA, LIC-TCM, and FTIC across all perceptual metrics, and surpasses generative codecs such as HiFiC and CDC with lower DISTS, FID, and PIEAPP, indicating closer perceptual similarity to the originals.

Fig. 4 compares rate-distortion performance. Our distortion oriented model achieves distortion comparable to BPG, VVC, ELIC, CCA, LIC-TCM, and FTIC. The perceptual oriented model emphasizes realistic detail synthesis via diffusion and thus trades off pixel level accuracy, but both variants outperform other generative codecs in fidelity; in particular, the perceptual model sacrifices some fidelity for improved realism.

2) Qualitative comparison: In Fig. 5, we visualize reconstructions produced by most of the compared methods. The reconstructions from BPG, ELIC, and CCA exhibit noticeable blur and artifacts. HiFiC and CDC produce sharper edges and textures. However, they also introduce spurious details that do not exist in the original images. Our method achieves strong semantic consistency and more faithful local details. It also attains better DISTS at a lower bitrate.

3) Complexity Comparisons: We further compare the complexity with state-of-the-art image compression methods. We report runtime under the available implementations or reported settings for reference. As shown in Table 1, we report the mean and standard deviation of the encoding and decoding time in seconds on the Kodak dataset for four categories of methods, including traditional codecs, VAE-based methods, GAN-based methods, and diffusion-based methods. The results indicate that diffusion-based methods generally incur higher encoding and decoding complexity than VAE-based and GAN-based approaches. Nevertheless, the encoder of FMDiff is notably faster than Text+Sketch [Lei et al., 2023] and DiffEIC [Li et al., 2024d].

4.3 Ablation Studies

To thoroughly analyze the role of key modules in FMDiff, we conducted ablation studies on FMBlock on the Kodak dataset. As shown in Table 2, the ablation experiments systematically investigated the impact of each component.

1) Effects of FMBlock. We ablate FMBlock by removing either the spatial or frequency branch and evaluate distortion and perceptual metrics (Table 2). For w/o Spat., the explicit spatial branch is removed while the cross attention condition for the frequency branch is retained. Both variants show worse performance than the full model, indicating that the two branches are complementary.

We further visualize error maps for the variant without FMBlock in Fig. 6. The results suggest that FMBlock better captures joint spatial frequency features, and that the frequency branch in particular improves the recovery of textures and fine details.

2) Analysis of the Frequency Branch. Table 3 reports ablations of the frequency branch. Using both amplitude and phase outperforms either alone, confirming their complementarity. Adding cross modulation consistently improves both fidelity and perceptual quality. Finally, removing the Mamba block degrades perceptual performance, at comparable bitrate, enabling Mamba markedly lowers DISTS while keeping PSNR similar, suggesting it captures long range dependencies.

3) Analysis of the Spatial Branch. Table 4 reports ablations of the spatial branch. The full spatial branch achieves the highest PSNR, while the LPIPS values are close among different variants. This suggests that the spatial branch mainly improves fidelity, whereas perceptual quality is more strongly affected by the frequency branch.

5 Conclusion

We propose FMDiff, a Fourier Mamba reconstruction methods for diffusion-based image compression, designed to mitigate spectral energy shifts and local texture inconsistencies caused by high frequency degradation. Motivated by the global local heterogeneity of frequency dependencies, FMDiff uses the FMBlocks to explicitly model both, the Fourier branch decouples magnitude and phase, captures patch-wise cross frequency coupling and long range frequency context using Mamba scanning, and restores corrupted spectra via magnitude phase interactive modulation. The spatial branch leverages Mamba to provide semantic and spatial consistency, with a learnable scaling factor and cross domain fusion balancing global structure and local details. Extensive experiments show that FMDiff improves the distortion perception trade-off across multiple datasets, yielding more realistic textures and more stable reconstructions.

Acknowledgments

We thank the anonymous reviewers for their valuable feedback. This work was partly supported by the Natural Science Foundation of China (62302466), Natural Science Basic Research Program of Shanxi Province (202303021212188).