SBSDM: A Style-aware Bidirectional Stream Diffusion Model for CT-to-PET Synthesis

1 Introduction

Computed Tomography (CT) provides high-resolution anatomical details, while Positron Emission Tomography (PET) reveals metabolic activity. The fusion of CT and PET is crucial for early cancer screening and precise localization of malignant tumors along with their functional states [Mu et al., 2020; Sujit et al., 2024]. However, the high cost and additional radiation exposure associated with PET scanning limit its widespread application. In this context, imageto-image synthesis [Li et al., 2025; Li et al., 2024; Salehjahromi et al., 2024] emerges as a promising alternative by leveraging conditional generative models to synthesize PET images directly from CT images (i.e., CT-to-PET synthesis).

Figure 1: (a) CT and PET sequence data, where the numbers represent the index of the corresponding slice in the sequence. The purple circles highlight the smooth transitions between adjacent CT and PET slices, and the blue circles emphasize the similar brightness of bone regions across non-adjacent slices. (b) Illustration of the existing method. (c) The framework of the proposed method.

However, due to the sequential nature of CT and PET scans and the physical principles governing medical imaging, CTto-PET synthesis faces two major challenges. First, continuous anatomical structures lead to strong structural correlation between adjacent slices (analogous to the content coherence between consecutive video frames). This necessitates the model to ensure smooth transitions between generated adjacent PET slices, as illustrated by slices 56, 57, and 58 in Fig. 1(a). Second, the imaging physics of CT and PET result in consistent imaging styles (e.g., brightness) for the same tissue type across non-adjacent slices, analogous to the object consistency maintained when the same object appears in non-adjacent video frames. For instance, as illustrated by slices 58 and 318 in Fig. 1(a), bone tissue appears bright in CT scans due to its high density, and similarly appears bright in PET scans due to its low metabolic activity. Overall, these two challenges can be attributed to the broader issue of correlation: the first concerns the smooth transition arising from adjacent slice correlations, while the second involves the style consistency stemming from non-adjacent slice correlations.

Prior studies have explored advanced 2D diffusion models to enhance the fine-grained details of synthesized 2D slice while maintaining efficiency [Nguyen et al., 2025]. However, in the absence of contextual information from adjacent slices, purely 2D models struggle to ensure inter-slice consistency. Conversely, as illustrated in Fig. 1(b), [Salehjahromi et al., 2024] adopts 3D models [Wang et al., 2024a; Wang et al., 2024b] to process volumetric inputs and explicitly capture correlations across adjacent slices, but at the cost of substantially increased computational complexity. Fundamentally, existing methods face an inherent trade-off: 2D models offer computational efficiency but limited awareness of inter-slice correlations, whereas 3D models effectively model inter-slice correlations but incur prohibitive computational overhead.

To break the dilemma faced by existing methods while addressing both correlation challenges under acceptable computational cost, this work focuses on enhancing 2D slice generation by integrating information from both adjacent and nonadjacent slices. Specifically, this work propose a Style-aware Bidirectional Stream Diffusion Model (SBSDM), which integrates two key components, namely the Neighbor Attention module and the Prototype Prompting mechanism (see Fig. 1(c)). On one hand, to tackle the smooth transition challenge efficiently, we employ a lightweight Vector Quantized Variational Autoencoder (VQVAE) [Van Den Oord et al., 2017] as a context extractor to capture stream transition information from bidirectional adjacent slices. The stream transition information is then integrated into the diffusion generation process for the current slice via the proposed Neighbor Attention module. Let D denote the computational cost of VQVAE feature extraction and C represent the cost of generating the current slice. The total computation of our model is D × (L - 1) + C, where L is the sequence length. When D ≪ C (facilitated by the lightweight VQVAE), we achieve substantial computational savings. On the other hand, to ensure style consistency across non-adjacent slices, we introduce a global Prototype Prompting mechanism. It aggregates features from all previously processed slices into a feature bank, from which global style prototypes are extracted and dynamically integrated into the generation process. This mechanism requires only minimal computational resources for storing and updating the feature bank, yet provides effective global style guidance throughout the synthesis process.

Our main contributions are summarized as follows:

• To ensure smooth transition, we develop a lightweight context extractor with a Neighbor Attention, enabling efficient extraction and integration of inter-slice transition patterns.
• To preserve style consistency, we propose a Prototype Prompting mechanism that stores global stylistic information and dynamically incorporates it into the generation process.
• Extensive experiments demonstrate that our method outperforms existing 3D methods while maintaining computational costs comparable to 2D models.
• We validate our SBSDM on the lung nodule benign-malignant classification task, demonstrating that preserving slice-sequence consistency effectively improves classification performance.

2 RelatedWorks

Image-to-Image Synthesis in Medical Imaging: In medical imaging scenarios, image-to-image synthesis has shown promising results. For instance, [McNaughton et al., 2023] employed a UNet architecture [Ronneberger et al., 2015] to achieve bidirectional mapping between MRI and CT; Zhou et al. [Zhou et al., 2020] and Yang et al. [Yang et al., 2020] further validated the capability of deep neural networks in image synthesis across different MRI sequences. Chen et al. [Chen et al., 2018] utilized Generative Adversarial Networks (GANs) [Goodfellow et al., 2020] to achieve super-resolution from low-resolution MRI to high-resolution MRI. Additionally, Tan et al. [Tan et al., 2024] explored synthetic methods for generating X-ray images from CT scans. Beyond the focus on morphological imaging modalities, the recent work [Nguyen et al., 2025] has attempted to use diffusion models for CT-to-PET synthesis. However, it overlooked the sequential nature of PET imaging, resulting in failures to maintain cross-slice coherence and style consistency. On the other hand, [Salehjahromi et al., 2024] proposed concatenating adjacent slices as input to help the model perceive transition information between adjacent slices. Yet, constructing 3D data through concatenation and using 3D models [Wang et al., 2024a; Liu et al., 2021] incurs significant computational costs [Zhou et al., 2023], and the processable sequence length is constrained by GPU memory limitations, making it difficult for the model to maintain global style consistency.

Sequential Correlation Modeling: To capture sequential correlations in 3D data with either temporal or spatial dimensions, common approaches employ 3D CNN [Tran et al., 2015] or 3D Transformers [Mao et al., 2021] for automatic modeling of sequential correlations. For example, [Zhao et al., 2024] utilized 3D CNN for the fusion and classification of 3D medical images, while [Wang et al., 2024b] explored the combination of diffusion models with 3D models for CT volume generation. However, the practical application of 3D models is severely constrained by their high computational cost. To address this, prior work [Bertasius et al., 2021] proposed divided spatial and temporal attention to model interframe correlations using 2D representations, thus avoiding the use of 3D models. [Liang et al., 2024] introduced a feature bank and fusion mechanism to cache unidirectional temporal information, enabling real-time video generation based on 2D diffusion models [Song et al., 2021]. In addition to architectural innovations, [Sun et al., 2023] applied data augmentation techniques to compress 3D data into 2.5D representations for CT synthesis. Nevertheless, these prior efforts focus on limited sequence lengths or unidirectional temporal correlations, overlooking the need for modeling correlations between bidirectional adjacent slices in medical image generation scenarios, and lacking the capability to capture longrange correlations between non-adjacent slices.

3 Methodology

3.1 Overview

In this work, CT-to-PET synthesis is formulated as a sequential generation task. Specifically, the model input consists of two components: 1) the current CT slice denoted as I z c , where the superscript z indicates the positional index within the CT scan sequence; and 2) the bidirectional adjacent slices represented by I zcn = {I z-w c , I z-w+1 c , · · · , I z+w c } \ {I z c }, where 2w determines the number of adjacent slices considered. Correspondingly, the model architecture follows a dualstream design. The first stream, illustrated in the lower part of Fig. 2(a), employs a lightweight context extractor based on VQVAE. The VQVAE is pretrained on paired CT and PET slices, and its encoder is then used to extract neighbor latent codes Fcn from I zcn . As shown in the upper part of Fig. 2(a), the second stream is a diffusion-based synthesis pathway that receives latent codes Fcn, the current CT slice I z c and its noisy version ˜ I z c as inputs, and produces the synthetic PET slice I z p . Additionally, the core modules of the diffusion model, namely the Neighbor Attention (NA) module and the Prototype Prompting (PP) mechanism, are presented in Fig. 2(b) and (c), respectively. The NA module extracts transition patterns from Fcn to ensure smooth transition, while the PP mechanism adaptively integrates cross-slice style information into the key K of the NA module, thereby enhancing global style consistency.

3.2 Style-aware Bidirectional StreamDiffusion Model

The UNet [Ronneberger et al., 2015; Rombach et al., 2022] architecture with vanilla attention modules [Vaswani et al., 2017] is employed as the backbone of our model. The core design of our model consists of two key components: 1) A NA module, which captures transition patterns from neighboring latent codes provided by a lightweight context extractor; 2) A PP mechanism, which extracts style prototypes from a feature pool and dynamically integrates them into the generation process.

Neighbor Attention for Smooth Transition

A lightweight context extractor is designed to capture neighbor features from bidirectional adjacent slices to facilitate smooth transitions in generated slices. Specifically, a quantized autoencoder is used as the context extractor [Van Den Oord et al., 2017]. Following VQVAE [Van Den Oord et al., 2017], the input Ic is first encoded by encoder EVQ into features: Fh = EVQ(Ic) ∈ Rd . Then, the closest ck to Fh is retrieved from the codebook C = {ck ∈ Rd }Nc k=1 (Nc is the size of the codebook) as the quantized feature Fc ∈ Rd :

After obtaining Fc, the decoder DVQ is used to reconstructs target domain images I vq p given Fc: I vq p =DVQ(Fc) . We first train the VQVAE model on the training set Dtr. For training details, please refer to [Van Den Oord et al., 2017].

Subsequently, as shown in Fig. 2, the trained encoder (i.e., EVQ) of the VQVAE is utilized to extract features from adjacent slices. For the CT slice I z c at index z, its adjacent sequence can be represented as:

and the corresponding adjacent PET sequence is denoted as I zpn . The latent codes of this adjacent sequence are denoted as:

tational cost in obtaining F cn primarily consists of two components: the forward computation cost of the VQVAE encoder EVQ and the lookup computation cost from the codebook C. By designing a lightweight encoder architecture, we can effectively extract contextual information from bidirectional adjacent slices with relatively low computational overhead. The dynamic balance between model performance and computational complexity will be thoroughly analyzed in the experimental section.

Although the VQVAE is capable of generating PET images from CT images, its architecture inherently limits the generation of high-quality data [Xiao et al., 2022]. Therefore, the VQVAE is utilized solely to obtain the neighbor latent codes. Next, we aim to leverage the attention mechanism to automatically extract contextual information (i.e., transition patterns) from neighbor latent codes Fcn that are beneficial to the synthesis of the current slice. Specifically, we develop a brand new Neighbor Attention (NA) module to replace all the vanilla attention module in the backbone UNet.

As shown in Fig. 2, the noised input CT image is denoted as ˜ I z c (obtained by the forward diffusion process), which is fed into the diffusion model. Then, the input representation to the NA module is denoted as h ∈ RL×d . In addition, the neighbor latent codes is also fed into the NA module. In this module, a linear layer is utilized to map h into the query, key and value (i.e., Q,K, V ∈ RL×d ). The output of the NA module is calculated by:

where Fcn ∈ R2w×d is the neighbor latent code, which conveys bidirectional stream information from adjacent slices into the generation process of the current slice. [·, ·] represents the concatenation operation. Q, K, and V are representations of the current slice, while Fcn is representations of the bidirectional adjacent slices. The matrix multiplication between Q and [K, F cn] produces a cross-slice similarity matrix, which is then passed through a softmax function to obtain attention weights. These attention weights act as dynamic feature selection gates, guiding the extraction of useful information from [V, Fcn] for synthesizing the current slice. It should be noted that the PP mechanism processing key representations in Fig. 2(b) is omitted here, as it will be elaborated in subsequent sections.

Prototype Prompting for Style Consistency

The proposed NA module focuses on extracting transition patterns from adjacent slices, but it fails to capture globally common visual styles due to the limited sequence length of adjacent slices. Therefore, a prototype prompting module is designed to tackle the challenge of style consistency. The solution lies in enabling the model to perceive consistent patterns across long-sequence slice representations. Inspired by the memory bank used in continual learning tasks [Liang and Li, 2024; Wang et al., 2022], we develop a dynamic feature bank to store features of previously processed slices by the model.

Since the proposed diffusion model incorporates a core module: Neighbor Attention, the feature bank is specifically designed to handle the representations involved in this Neighbor Attention (NA) module. Taking the key representations K in one NA module as an example, we first initialize a zero matrix as the feature bank PK = 0d×d and set its initial size to zero LPK = 0. Subsequently, by leveraging the property that slices within a batch are randomly sampled during training, we perform an averaging operation along the batch dimension to obtain a cross-slice representation:

Kavg = 1 Nb ∑Nb i=1 Ki ∈ RL×d , where Ki is the key representation (i.e., K mentioned above) of a single slice, and Nb is the batch size. L is the length of the representation, and d denotes the feature dimension. Next, Kavg is used to update the feature bank:

where K⊤ avgKavg ∈ Rd×d is the cross slice feature space. As the forward pass is performed on each data batch, the computed Kavg continuously updates PK, gradually accumulating cross-slice features.

The feature bank is designed to cache cross-slice features. However, the feature bank stores a large number of crossslice features. To further extract common style information while eliminating redundant features, we perform cluster analysis on the feature bank and use the cluster centroids as style prototypes. Specifically, we introduce a parameterfree clustering method: Gaussian Kernel-based Mean Shift [Cheng, 1995], which effectively distills prototypes from PK: PS = MS(PK) ∈ RNP×d , where NP is the number of prototypes (i.e., cluster centers). MS(·) represent the function of the Mean Shift.

Prototypes PS are then merged with the key representations, thereby infusing the key representations with style information. Specifically, we design a Prototype Prompting mechanism. The feature K is represented as vector form [V 1K , V 2K , · · · , V L K ], where each vector V m K ∈ Rd , and m ∈ [1, 2, · · · ,L]. Similarly, PS = [V 1 PS , V 2 PS , · · · , V Np PS ], where V n PS ∈ Rd , and n ∈ [1, 2, · · · ,Np]. Then, the cosine similarity between each VK and VPS is calculated as: VK·VPS ∥VK∥2∥VPS∥2 . For a feature vector V m K , we denote the feature vector in PS that has the largest similarity with V m K as V n PS . If the similarity between V m K and V n PS exceeds a threshold θ, V n PS is fused into V m K :

where α is used to stabilize numerical computations. When VK exhibits high similarity with VPS, fusing VPS can introduce style information while maintaining the stability of VK. Conversely, when the similarity between VK and VPS is low, VK primarily reflects the specific characteristics of the current slice. Forcibly fusing VPS in such cases may lead to feature distortion. Therefore, the fusion operation is performed only when the similarity is sufficiently high.

It should be noted that the feature bank and Prototype Prompting mechanism are illustrated using the key representation in the NA module as an example, but they are also applicable to the value representation V and the NA outputs. Overall, the proposed diffusion model consists of two core components: the Neighbor Attention module, which integrates neighboring latent codes to ensure smooth transitions, and the Prototype Prompting module, which incorporates global style information to maintain style consistency in CT-to-PET synthesis.

Although several individual components in our framework are inspired by prior studies, the core contribution of this work lies not in isolated module invention, but in revealing and formulating the unique correlation mechanisms underlying CT-to-PET synthesis and designing a unified diffusion framework specifically tailored to these medical imaging characteristics.

Diffusion Training Paradigm

To train the proposed diffusion model, we adopt the same training objective as in [Yue et al., 2023]:

where ˜ I z c = (1-ηt)I z p + ηtI z c +κ√ηtϵ is the noised sample, and ϵ ∼ N(0, I). t is the time step. ηt and κ are hyperparameters proposed in [Yue et al., 2023]. U(·) denotes our proposed diffusion model.

4 Experiments

4.1 Experimental Setup

Datasets: A large-scale publicly available dataset, FDGPET-CT-Lesions [Gatidis et al., 2022], is utilized to evaluate the effectiveness of the proposed model. This dataset comprises 1014 consecutive whole-body PET/CT scans. The dataset is split into 709 for training, 102 for validation, and 203 for testing, resulting in 248,657, 33,920, and 72,570 paired CT-PET slices for each subset, respectively. To further assess the model's generalizability, we additionally employ an in-house dataset consisting of 215 PET/CT scans. The inhouse dataset is partitioned into 150 for training, 22 for validation, and 43 for testing, yielding 45,906, 6,930, and 13,639 paired CT-PET slices, respectively.

Evaluation Setting: We adopt four metrics for assessing individual slices quality: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Fr´echet Inception Distance (FID), and the perceptual metric (LPIPS). Additionally, average cosine similarity (denoted as CLIPs) between CLIP embeddings of consecutive slices [Ceylan et al., 2023] is used to evaluate the smooth transition (i.e., temporal coherency). Among these metrics, higher values of PSNR, SSIM, and CLIPs indicate better generation quality, while lower values of FID and LPIPS signify superior results.

Implementation Details: The VQVAE architecture follows the design in [Zhou et al., 2022]. The codebook size is set to 1024. For the diffusion model, the shifting sequence ηt and the hyper-parameter κ = 2 follow the setting in [Yue et al., 2023]. The number of diffusion time steps is set to 4 to ensure efficient and rapid image generation. The design of the UNet references Stable Diffusion [Rombach et al., 2022]. The dimensionality of the feature vectors d is set to 640. The AdamW optimizer is used to train both the VQVAE and the UNet, with the batch size and learning rate set to 96 and 5e-5, respectively, for both models. The hyperparameters w, α, and θ are determined via parameter search and set to 10, 0.5, and 0.9, repsectively.

Baselines: To evaluate the effectiveness of our proposed method, we select the following baseline models: 1) CycleGAN [Zhu et al., 2017]: A direct implementation of the pioneering method [Salehjahromi et al., 2024]; 2) RegGAN [Kong et al., 2021]: A GAN model improved for medical imaging characteristics; 3) 3D-RegGAN: Replacing the 2D generator of RegGAN with a 3D UNet [C¸ ic¸ek et al., 2016] architecture; 4) DDBM-UNet: Modifies the training paradigm of ResShift-UNet to DDBM, representing a straightforward implementation of the current state-of-the-art method [Nguyen et al., 2025]; 5) ResShift-UNet [Yue et al., 2023]: Combining the advanced diffusion training paradigm ResShift with the standard diffusion model UNet [Rombach et al., 2022]; 6) DDBM-Freq and 7) ResShift-Freq: Introducing high-frequency components improves the quality of generated details [Li et al., 2024]; 8) DDBM-3DDiff and 9) ResShift-3DDiff: A 3D CNN is employed to transform the high-frequency components [Shang et al., 2024]; 10) DDBM-3DUNet and 11) ResShift-3DUNet: The UNet in 4) and 5) is replaced with the 3D UNet; 11) VQVAE: an autoencoder-based image synthesis method [Van Den Oord et al., 2017].

4.2 Comparison with the State-of-the-Art Methods

Table 1 summarizes the quantitative performance of our method and baseline models on both datasets under identical training settings. The results support the following key observations: 1) VQVAE alone fails to produce satisfactory results, highlighting its limited generative capability. This justifies our design choice of using VQVAE solely for extracting neighbor latent codes. 2) Our method surpasses even the strongest 3D diffusion-based baselines. Fig. 3 presents a qualitative comparison. It can be observed that our 2D UNetbased framework achieves superior performance even compared to ResShift-3DUNet.

4.3 Ablation Study

Comprehensive ablation studies are conducted to validate the effectiveness of the proposed NA module and PP mechanism. The results of the quantitative analysis are presented in Table 2, where we assess the contribution of NA and PP by selectively removing each component from the full model. The results show that removing either the NA or PP module leads to a notable performance degradation, clearly demonstrating the importance of both components.

To intuitively assess the effectiveness of the NA module in addressing the smooth transition challenge, Fig. 4 presents generation results for a continuous sequence with and without the NA module. These results demonstrate that the NA module enhances the model's ability to perceive anatomical transitions between adjacent slices.

Furthermore, we conduct a case study to evaluate the effectiveness of the PP module in maintaining style consistency. As shown in Fig. 5, removing the PP module (second column, W/O PP) causes low-metabolism visceral regions to be incorrectly synthesized as high-metabolism areas. In contrast, the full model (third column, Ours) better preserves the PET color style of low-metabolism regions, demonstrating the PP module's effectiveness in maintaining global style consistency across synthesized slices.

4.4 Computational Efficiency Analysis

To evaluate the effect of VQVAE parameter scale, we vary the number of residual blocks in the encoder and measure the corresponding performance. As shown in Fig. 6(a), performance becomes stable when the number of residual blocks exceeds 12. Therefore, we set the number of residual blocks to 12, resulting in an encoder with approximately 81M parameters and 71G FLOPs, achieving a balance between efficiency and performance. In addition, Fig. 6(b) shows that our model requires significantly lower FLOPs than 3DUNet as the sequence length increases. Furthermore, the inference efficiency comparison in Fig. 7, conducted on an NVIDIA RTX 5880 Ada Generation GPU, demonstrates that our method achieves inference speed comparable to 2D models such as ResShift-UNet while providing superior performance.

(a) Performance Scaling with VQVAE (b) Computational Cost Comparison

Figure 6: (a) Model performance w.r.t. the number of residual blocks in EVQ. (b) FLOPs comparison between our model and 3DUNet across varying sequence lengths.

4.5 Clinical Utility Analysis of Synthetic PET

We evaluate the clinical utility of the synthesized PET images by applying them to a lung nodule malignancy classification task. Specifically, we first use the model trained on the FDG-PET-CT-Lesions dataset to synthesize PET slices from CT slices in the LIDC-IDRI dataset [Armato III et al., 2011] (79 and 78 cases are used for training and validation, respectively). A binary classification model [Sibille et al., 2020] is then trained under three different settings: 1): using only CT; 2): using CT combined with PET synthesized by ResShift3DUNet; 3), 4), and 5): using CT combined with PET synthesized by three variants of our method: without NA, without PP, and the full model, respectively. The results in Table 3 verify the effectiveness and clinical application potential of each module we proposed. Due to limited pathologically confirmed cases in LIDC-IDRI, our validation is only a preliminary test of synthetic PET's potential in downstream tasks.

We employ two radiologists from commercial medical institutions to evaluate the quality of the generated PET slices. Specifically, we randomly select 50 PET slices from the test set and mix them with corresponding PET slices generated by different methods. Without being informed about the existence of synthetic PET images, the radiologists are asked to score all PET scans on a scale from 1 to 5, with higher scores indicating better image quality. The average score is then computed and summarized in Table 4. It can be observed that the PET generated by our method achieved the closest scores to the real PET.

5 Conclusion

In this work, we propose a Style-aware Bidirectional Stream Diffusion Model to address the CT-to-PET synthesis task. Specifically, we introduce the NA module to capture transition patterns between adjacent slices and the PP mechanism to ensure global style consistency across synthesized PET images. Extensive experiments on both public and private datasets demonstrate the effectiveness of our approach. The CLIPs metric quantitatively verifies the NA module's ability to ensure smooth inter-slice transitions, while a case study confirms the PP mechanism's contribution to preserving global style consistency. In future work, we plan to explore the physiological foundations underlying deep learningbased CT-to-PET synthesis.