Training-free Fourier Phase Diffusion for Style Transfer

1 Introduction

Image style transfer, which aims to incorporate a specific style into an input image while preserving its core content, finds wide applications in fields such as digital art creation and augmented reality filters. Early deep learningbased style transfer methods, such as AdaIN [Huang and Belongie, 2017], Gram-based approaches [Gatys et al., 2016], and WCT [Li et al., 2017], achieved this by aligning the statistical properties of content and style features extracted from pre-trained VGG networks [Simonyan and Zisserman, 2014]. Subsequently, GAN-based methods [Zhu et al., 2017] were introduced, employing adversarial training to transform content into a styled domain via CNNs. However, these CNNbased methods simply optimize the features extracted by a basic network to obtain stylized results, often struggling to achieve a harmonious integration of style and content.

∗ Corresponding author.

Recently, diffusion models have achieved significant breakthroughs in tasks such as text-to-image generation, image editing, and super-resolution. Many studies [Zhang et al., 2023c] [Wang et al., 2023] [Li et al., 2025] [Hamazaspyan and Navasardyan, 2023] have explored the generative capabilities of these models in style transfer, particularly focusing on decoupling and re-fusing style and content. Some methods [Chen et al., 2024] [Qi et al., 2024] require both a content prompt and a style image as input. These methods train additional adapters to extract style information from the style image and then integrate the extracted style into the generated content using the cross-attention mechanism or LoRA layers [Hu et al., 2021]. However, these techniques demand substantial computational resources during training, and many rely on content descriptions, which, even when complex, often lead to results that deviate from the user's expectations, making them unsuitable for style transfer on photographs.

Training-free methods leverage the inherent capabilities of pre-trained large diffusion models for style transfer, requiring much fewer computational resources and offering easy extensibility to various versions of diffusion models. However, such methods face challenges due to the tightly coupled representation space and the inherent randomness of the diffusion models. StyleID [Chung et al., 2024] replaces the key and value of the content with those from the style image, obtained via DDIM inversion, to inject the style using the cross-attention mechanism during the generation process. This integration, which relies on the similarity between the style and content features, often leads to unsatisfactory output, as shown in Figure 1. In contrast, FreeStyle [He et al., 2024] combines the low-frequency components of the content features with the high-frequency components of the style features for stylization. Although FreeStyle produces more intense style effects than StyleID, it struggles with content preservation, particularly in fine-grained structures. For example, the face of the old man is transformed into that of a child after stylization, as shown in Figure 1.

In this paper, we introduce a novel, training-free Fourier phase diffusion model for style transfer, where the phase spectrum of the content image is used to conditionally guide generation. This approach outperforms existing models in both content preservation and style transfer, as shown in Figure 1. Our motivation for utilizing the Fourier phase spectrum lies in its ability to capture the structures of an image, including object shapes and inner edges [Lv et al., 2024] [Yao et al., 2024] [Yang and Soatto, 2020]. As illustrated in our simple test in Figure 2, replacing the phase spectrum of an image results in a complete alteration of the content. This demonstrates the potential of the phase spectrum to decouple and conditionally guide the highly randomized generative process, thereby maintaining the content while enabling stylistic generation.

However, integrating the Fourier phase spectrum of the content image into the diffusion model without training is challenging, as it may interfere with the generation of naturallooking stylized images. As shown in Figure 2, combining the phase and amplitude from different images can result in noticeable artifacts. To address this challenge, we propose a phase spectrum fusion module to modulate intermediate stylization results with the content phase, along with a strategy to integrate the module into the generation process, aligning with the stage-specific generative characteristics of the diffusion model. Furthermore, to enhance content preservation, inspired by ControlNet [Zhang et al., 2023a], we utilize the U-Net architecture of the diffusion model to encode the content image, providing semantic guidance throughout the entire generation process.

Our main contributions are summarized as follows:

• We propose a novel training-free Fourier phase diffusion model for style transfer, which leverages the phase spectrum of a content image to guide the style generation process, achieving outstanding performance in both content preservation and style transfer quality.
• We present a phase spectrum integration method that incorporates the phase spectrum of the content image into the generation process, preserving structural details while generating visually appealing stylized images.
• We propose a semantic injection mechanism that extracts the content image's semantics using the U-Net within the diffusion model and subsequently incorporates these semantics into the generation process, further enhancing content preservation.
• Extensive qualitative and quantitative evaluations demonstrate that the proposed model outperforms state-of-the-art methods in style transfer tasks.

2 RelatedWork

2.1 CNN-based Style Transfer

Gatys et al. [Gatys et al., 2016] pioneered CNN-based neural style transfer by using VGG features and Gram matrices to capture style characteristics, and generating stylized results by optimizing both content and style objectives. Subsequent methods like AdaIN [Huang and Belongie, 2017] and WCT [Li et al., 2017] leverage the VGG network [Simonyan and Zisserman, 2014] for feature extraction and align feature statistics for stylization. CycleGAN [Zhu et al., 2017] approaches style transfer through adversarial learning to establish cross-domain mappings. Recently, CLIPStyler [Kwon and Ye, 2022] combines a CNN encoder-decoder architecture with patch-wise CLIP loss to capture hierarchical content features and refine stylization results. Despite their innovations, these methods often struggle to achieve a satisfactory integration of content from a content image and style from a reference image, due to the inherent differences between the two.

2.2 Diffusion-based Style Transfer

Diffusion models have surpassed GANs in image generation and have been successfully applied to various tasks, including style transfer. Leveraging CLIP [Radford et al., 2021], the Latent Diffusion Model (LDM) [Rombach et al., 2022] improves sampling efficiency by encoding images into a latent space using a pretrained auto-encoder [Esser et al., 2021], enabling text-guided image generation through large-scale pretraining. Recent works like DEADiff [Qi et al., 2024] and ArtAdapter [Chen et al., 2024] achieve style transfer by integrating style information into diffusion models through trained style embeddings or adapters. These methods rely on text prompts for content specification, limiting their applicability to photographs. While works like Artbank [Zhang et al., 2024b] take content images and style descriptions as input, they still require substantial computational resources for training. Training-free image stylization methods have attracted much attention recently [Jiang and Chen, 2024] [He et al., 2024], but maintaining content fidelity while achieving the desired stylization remains an open challenge.

2.3 Controllable Text-to-Image Diffusion

Diffusion models need to focus on controlling the content of generated images, which is also a crucial aspect of diffusionbased style transfer. Several approaches have been proposed to control the content of output generated by diffusion models [Zhang et al., 2023a] [Ye et al., 2023] [Zheng et al., 2023] [Yu et al., 2023]. For instance, ControlNet [Zhang et al., 2023a] allows users to control the structures of the generated images by conditioning them on various input types, such as depth maps, Canny edges, or human poses. FreeControl [Mo et al., 2024], a training-free method for controllable text-to-image generation, enables control over the generated images based on their semantic representations. Layout-diffusion [Zheng et al., 2023] adopts a layout fusion module to control the generated images. Although significant progress has been made in controlling the output images of diffusion models, these methods are not directly applicable to image stylization. However, their conditional guidance based on various clues inspires us to address content preservation during stylization.

3 Preliminaries of DiffusionModels

The diffusion model consists of two key processes: the diffusion process and the sampling process. In the diffusion process, noise is added to a given image x0 to obtain a noisy image xt at time step t according to the equation:

where ϵ represents the Gaussian noise, and αt = ∏ti=0 αi. αi is a hyperparameter that controls the signal strength at time step i.

The generation process of diffusion models typically starts with the noisy image, and through iterative denoising steps, the model gradually refines the content, eventually achieving fine textures and details. In detail, given a text prompt y, diffusion models use noise-predictor ϵθ (U-Net in our method) to predict noise ϵθ(xt, t, C) from xt, where θ denotes parameters of the noise-predictor [Rombach et al., 2022] and C represents the text embedding of the prompt y. The down-block and mid-block of ϵθ encode semantic information. In DDIM [Song et al., 2020], x0,t at timestep t is given by

As t approaches zero, x0,t becomes increasingly similar to the generated result. x0,t is used to denoise xt to obtain xt-1. Mathematically,

where σt controls the stochasticity of the sampling process. DDIM sets σt to 0 in Eq. (3), making the sampling process deterministic.

4 Method

4.1 Overview

The overall framework is shown in Figure 3. Given a content image Ic and a style text description ys, we aim to transfer the style from ys to Ic, resulting in a stylized image Ics that maintains the content of Ic. Firstly, we add noise to the input image Ic using Eq. (1), setting t = T, to obtain the noisy image xT , T = 1000. xT subsequently serves as the starting point for the sampling process to generate the stylized image Ics. The style text description ys is encoded by the CLIP text encoder [Radford et al., 2021] into style embeddings, which influence the style of the generated result through cross-attention. We design a phase spectrum fusion module (see Section 4.2) and explore its integration into a specific stage of the diffusion model, based on the stagespecific generation characteristics (see Section 4.3). Additionally, to further enhance content preservation, we use the U-Net of the diffusion model to extract semantics from the input content image and inject them into the generation process (see Section 4.4).

4.2 Phase SpectrumFusion

Our phase fusion module is illustrated in Figure 4. At timestep t, we obtain the intermediate stylized result x0,t using Eq. (2). Then we transform both x0,t and the content image Ic into Fourier space, yielding I freq x and I freq c . The transformation is given by

where M and N are the length and width of the images, and (m, n) and (u, v) represent the coordinates in the RGB image space and its frequency space, respectively.

where R(u, v) and Q(u, v) are the real components and imaginary components, and j is the imaginary unit. We can obtain the amplitude spectrum of x0,t and Ic, denoted as ∣∣I freq x ∣∣ and ∣∣I f req c ∣∣, respectively, by using

Then we add them linearly and empirically set the weight to 0.5, to obtain ∣∣I freq sum ∣∣. On the other hand, the phase spectrum ϕf req c of Ic is computed as

Then we recombine the phase spectrum ϕfreq c and the amplitude spectrum ∣∣I freq sum ∣∣ to obtain the new frequency data I freq comb , as follows

Next, we perform inverse Fourier transform to obtain x′0,t , given by:

Note that directly mixing the phase and amplitude spectra of different images leads to artifacts, as explained in Figure 2. AdaIN [Huang and Belongie, 2017] is known for effectively aligning feature statistics in style transfer, while Style Spectroscope [Jin et al., 2024] has shown that AdaIN can only affect the amplitude spectrum, not the phase spectrum. Therefore, we apply AdaIN to align the statistics of x′0,t with those of x0,t, aiming to mitigate the color artifacts caused by the amplitude and phase mismatch in Eq. 9.

Finally, we introduce x′′ 0,t into x0,t in the form of residuals, namely

where α and β are hyperparameters. α controls the amount of phase from the content image Ic, while β controls how much of the intermediate stylized result x0,t is retained. ˆx0,t then replaces x0,t in the subsequent denoising process.

4.3 Phase SpectrumFusion in Diffusion

Prospect [Zhang et al., 2023b], FreeU [Si et al., 2024], and other works [Guo and Lin, 2024][Zhang et al., 2024a] have demonstrated that the diffusion model generates images through a layout - content - fine texture process. Accordingly, we divide the generation process into three stages: early, middle, and late. We incorporate the content image via the phase spectrum fusion module in the early stage, where the edge structures exert the greatest influence on the final result's structure. Additionally, incorporating the content phase causes the intermediate results of the diffusion process to deviate from the learned data distribution. Several diffusion steps are needed to bring these results back into the expected distribution of the diffusion model. Therefore, we apply the phase spectrum fusion module intermittently during the early stage.

To support our design, we conduct contrastive experiments. The results are shown in Figure 5. Groups A, B, and C apply the phase fusion module continuously during the early, middle, and late stages, respectively, while groups D, E, and F use the module intermittently across these stages. The results of groups D, E, and F are better than those of groups A, B, and C, indicating that continuous application of the phase fusion module is suboptimal. In contrast, intermittent use of the module helps maintain content structures while achieving higher-quality stylization. Among these, group D, which applies the phase fusion module intermittently in the early stage, produces the most satisfying results, as expected.

4.4 Semantic Injection

Although phase spectrum fusion and its incorporation are effective, subsequent generation steps may introduce elements unrelated to the semantic content of the input image. To address this, inspired by ControlNet [Zhang et al., 2023a], Zecon [Yang et al., 2023], and other works [Choi et al., 2025][Lin et al., 2024][Li et al., 2024] in the virtual try-on task, which demonstrate that the U-Net in diffusion models can extract semantic information, we extract and incorporate the semantics of the content image as guidance during generation. We utilize the down-blocks and mid-blocks of the UNet in diffusion model to extract semantic features from the content image, which are then injected into the corresponding positions of the U-Net in the generative stylization model (as illustrated in Figure 3) in a residual manner.

5 Experiments

5.1 Implementation Details

We utilize Stable Diffusion XL 1.0 as the foundational textto-image model and employ the DDIM sampler with 30 sampling steps for each stylized image generation. The phase fusion module is applied during the early sampling stage, specifically at the 2nd, 5th, and 8th timesteps, with the parameters set to α = 0.5 and β = 0.7. Notably, our method is training-free and does not require fine-tuning. All experiments are conducted on a single RTX 3090 GPU.

5.2 Qualitative Comparison

We compare our method with state-of-the-art approaches, including Zecon [Yang et al., 2023], CLIPStyler [Kwon and Ye, 2022], FreeStyle [He et al., 2024], and StyleID [Chung et al., 2024]. Unlike the other methods, which use a style text prompt and a content image as input, StyleID requires both a content image and a style image. To generate the necessary style image for StyleID, we employed the Stable Diffusion XL 1.0 model based on the provided style description. Figure 6 illustrates the comparison results. FreeStyle demonstrates excellent style transfer but struggles to preserve content. Conversely, CLIPStyler maintains content structures but delivers weaker style transfer. Zecon also exhibits limited style quality, likely due to the CLIP model's emphasis on semantic content rather than style. The Artist method produces results nearly identical to the input content images, while StyleID shows relatively weak style transfer and suboptimal structural preservation. In contrast, our method achieves superior content preservation while delivering style quality comparable to FreeStyle.

5.3 User Study

We conducted a user survey with 50 participants to compare our method against five other approaches: FreeStyle, Artist, Zecon, CLIPStyler, and StyleID. For these methods, all experimental results were produced using their publicly available default parameter settings. Participants were presented with 25 groups of results, each paired with the corresponding content image and style text description. For unconventional styles, a style image generated by the Stable Diffusion model was also included as a reference. Participants assessed both the content preservation and style quality of the generated results, selecting at least two results they favored the most in each group. The results of the user survey, presented in Figure 7, indicate that our method was more favored.

5.4 Quantitative Comparison

Evaluation Metrics

CLIPAesthetic Score. It is a metric for assessing aesthetic scores. The underlying model is a neural network that uses CLIP embeddings as input and is trained to predict an image's aesthetic score based on its visual appeal.

HPS. TheHuman Preference Score (HPS) [Wu et al., 2023] is a scoring model developed to predict human preferences for generated images. Trained on a Human Preference Dataset, this model assigns higher scores to images that are more likely to be favored by people.

CLIPScore. We calculate the cosine similarity between the CLIP features of a stylized image and the corresponding style text description. A higher CLIP Score reflects a more accurate alignment with the intended style.

SSIM. Structural Similarity Index (SSIM) is a metric used to measure the similarity between two images. It evaluates image similarity by analyzing three key dimensions: luminance, contrast, and structure within local regions of the images. An SSIM value close to 1 indicates high similarity between the images, while a value near 0 suggests significant differences.

Quantitative Comparison Results

Table 1 provides a quantitative comparison of our method with five competitive approaches. Our method achieves the highest scores in both the CLIP Aesthetic Score and HPS metrics, demonstrating that the results produced by our approach are more likely to be preferred by users. This aligns with the results of our user study. For the CLIP Score, our method ranks second, following CLIPStyler, which explicitly incorporates the CLIP Score as a constraint in its style transfer process. However, when evaluating performance, we prioritize user-centric metrics, such as the CLIP Aesthetic Score and HPS, as the CLIP Score is not considered the primary criterion for assessing quality of stylized results.

To compute the SSIM score, a fine-grained similarity metric, we used SAM [Kirillov et al., 2023] to mask out blank regions in both the content images and the stylized ones. SSIM was then calculated only on areas containing distinct content structures, thereby minimizing the influence of stylistic elements in blank regions. While our method achieves the second-highest SSIM score, Artist attains the highest score for this metric. However, Artist struggles to deliver highquality style transfer results based on the given style descriptions (see Figure 6).

5.5 Ablation Study

Effect of Modules

We validate the effectiveness of the two key modules, namely the phase fusion module and the semantic injection module, by incorporating them incrementally. The results are presented in Figure 8. The baseline model (w/TP) refers to the original diffusion model that uses only the style text prompt as input. The model w/TP + SI integrates the semantic injection module, while w/TP + SI + PF represents our complete model, which includes both semantic injection and phase fusion modules. As shown in Figure 8, the addition of semantic injection helps to constrain the subjects, such as the cat and flower, but their structures remain inconsistent with the content images. By further incorporating phase fusion, our model achieves superior performance in both content preservation and stylization.

Effect of Hyperparameters α and β

The parameter α determines the extent to which the phase spectrum of the content image is introduced, while β controls the amount of style information retained. While a full grid search could be employed to determine these two hyperparameters, we simplified the process by initially setting both α and β to 0.5 and performing a localized grid search within specific ranges. First, we fixed β at 0.5 and adjusted α within the range of 0.3 to 0.7. We observed that increasing α further, which leads to a stronger influence from the content image's phase spectrum, introduces noticeable artifacts. In contrast, decreasing α results in weaker stylization. Based on this observation, we set α to 0.5. Next, with α fixed at 0.5, we tuned β within the range of 0.4 to 0.8. A smaller β resulted in strong artifacts, while a larger β reduced the effectiveness of style expression. Ultimately, we determined the optimal values as α = 0.5 and β = 0.7, achieving a balance that ensured effectively stylized and artifact-free results. Once determined, these hyperparameters are fixed for all stylization tasks.

6 Conclusion

In this work, we proposed a Fourier domain phase spectrum incorporation method to guide the diffusion model in preserving the content of an input image during style transfer. To further enhance content preservation, we introduced a mechanism that integrates semantic information from the content image. These two designs enable the diffusion model to achieve high-quality style transfer results while ensuring the generated image remains closely aligned with the input content image, in a completely training-free manner. Our method currently does not support transferring styles from image references. In the future, we plan to expand our model to accommodate various types of style references.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 62176010), the Beijing Natural Science Foundation (No. 4252029) and the Open Project Program of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (No. VRLAB 2024B02).