RLBCD: Residual-guided Latent Brownian-bridge Co-Diffusion for Anatomical-to-Metabolic Image Synthesis

1 Introduction

Metabolic imaging (e.g. computed tomography angiography (CTA) and positron emission tomography (PET)) provides essential insights into physiological and metabolic status of lesions, facilitating early diagnosis and treatment [Torigian et al., 2007; Lv et al., 2024]. However, compared to anatomical imaging, the acquisition of metabolic imaging is limited by high cost, slow scanning speed, high radiation risk, and potential renal impairment caused by contrast agents [Faucon et al., 2019; Lv et al., 2022]. Therefore, it is necessary to develop computational methods for automatic and effective Anatomical-to-Metabolic Image Synthesis (A2MIS).

* Corresponding Author.

Given that the A2MIS problem can be regarded as an image-to-image (I2I) task, various deep learning-based approaches, including Generative Adversarial Network (GAN)- based methods, diffusion model (DM)-based methods, and other techniques, have the potential to address this challenge. GAN-based methods [Goodfellow et al., 2014], such as Pix2Pix [Isola et al., 2017] and CycleGAN [Zhu et al., 2017], have shown promising results by learning the conditional distribution of the metabolic images given the samples from the anatomical domain. However, GAN-based I2I methods are notoriously hard to train and often suffer from mode collapse in the output distribution. Other methods, such as Autoregressive Models [Parmar et al., 2018], Variational Autoencoders (VAEs) [Vahdat and Kautz, 2020], and Normalizing Flows [Kingma and Dhariwal, 2018], have achieved success in specific applications but fail to acquire the same level of sample quality and general applicability as GANs. Recently, DM [Ho et al., 2020; Song et al., 2021] has emerged as a competitive alternative, showing the ability to generate highquality images compared to GAN-based models. Several conditional diffusion models (CDMs) [Batzolis et al., 2021; Saharia et al., 2022; Rombach et al., 2022] have proposed for the I2I task by integrating the source image into the reverse diffusion process to guide generation toward the target domain. However, CDMs struggle to generate consistent results due to their inherent stochasticity. Even when employing deterministic samplers like DDIM, their reliance on sampling from random noise introduces uncertainty, undermining reproducibility and medical reliability.

In Particular, the aforementioned methods face several challenges when directly applied to A2MIS. First, anatomical and metabolic images often exhibit similarities in shape (structure) and style (brightness and contrast) at certain locations. This resemblance presents a significant challenge for existing I2I methods, as they struggle to effectively capture the differences between the two domains, leading to synthesized images that do not accurately reflect the intended metabolic characteristics. Second, DM-based methods can sample diverse results for a given anatomical image. It is essential to provide a confidence score associated with these synthesized images for physicians and patients, such as AlphaFold [Jumper et al., 2021; Abramson et al., 2024]. This knowledge enables professionals to make informed decisions. Unfortunately, existing methods primarily overlook this crucial point. Thus, a core question for A2MIS is how to optimally decouple and leverage anatomical and metabolic features while incorporating a synthesis confidence score.

To address these challenges, we propose a novel Residualguided Latent Brownian-bridge Co-Diffusion (RLBCD) model for precise and reliable A2MIS. Specifically, RLBCD constructs a co-diffusion process that leverages a latent residual diffusion branch to capture inter-domain differences, which are then injected into an latent enhanced diffusion branch to optimally reconstruct modality-specific details. Our method employs a stochastic Brownian bridge process [Li et al., 2023a] that directly learns translation between two domains without any conditioning mechanism. To achieve the desired residual guidance, we investigate the encoder and decoder features in diffusion models and accordingly propose a hybrid-granularity fusion, which integrates consistent semantics and complementary information for fine-grained reconstruction. Moreover, we develop a latent consistency score to improve the restoration of modality-specific information, which also serves as an indicator of the inherent confidence of the synthesized images. We conduct extensive experiments on various public and in-house datasets and tasks, including CT-to-CTA, CT-to-PET, and pre-contrast MRI-topost-contrast MRI generation, as well as downstream diagnosis and segmentation tasks. Experimental results demonstrate the superiority of our approach not only performance but also valuable medical applications. Specific contributions of this work can be summarized as follows:

• We propose RLBCD, a novel framework for A2MIS, leveraging a designed residual-guided latent Brownianbridge co-diffusion process for the optimal reconstruction of modality-specific details.
• We propose Hybrid-Granularity Fusion to integrate consistent semantics and complementary information from a residual diffusion for fine-grained generation.
• We propose latent consistency score, serving as an indicator of the inherent confidence of synthesized images.
• Extensive experiments demonstrate the superiority of our approach in terms of translation performance and its value in downstream diagnosis and segmentation tasks.

2 RelatedWork

2.1 Anatomical-to-Metabolic Image Synthesis

Metabolic images, compared to anatomical images, provide a different perspective on the body's physiological functioning, thereby facilitating early disease detection and monitoring treatment efficacy [Torigian et al., 2007]. However, metabolic imaging presents several obvious limitations, including high cost, significant radiation exposure, and potential risks to renal function. Several approaches have been proposed for A2MIS, including 1) Synthesis of PET from MRI or CT [Hu et al., 2021; Lee et al., 2024; Vega et al., 2024]; 2) Synthesis of CTA from CT [Lyu et al., 2023]; 3) Synthesis of Post-contrast CT from Pre-contrast CT [Kim et al., 2021; Choi et al., 2021]; and 4) Synthesis of Postcontrast MRI from Pre-contrast MRI [Calabrese et al., 2021; Wang et al., 2022]. Current studies mainly utilize or modify existing I2I methods, overlooking the unique characteristics inherent to A2MIS, such as the relations between anatomical and metabolic images, and the requirement for confidence scores of the synthesized images. This confidence assessment is crucial for clinical applications, as it provides a measure of the reliability of the synthesized images. Thus, our objective is to develop an approach for precise and reliable A2MIS.

2.2 DiffusionModels in Image Translation

Diffusion models (DMs) are initially developed for image generation such as DDPM [Ho et al., 2020] and DDIM [Song et al., 2021]. Subsequent image translation methods [Batzolis et al., 2021; Saharia et al., 2022; Rombach et al., 2022] based on DMs treat the I2I task as conditional image generation, guiding the diffusion towards the target domain by feeding the source image or its encoded feature into the denoising U-Net during the reverse process. Despite achieving some practical success, this condition mechanism lacks a clear theoretical guarantee that the final diffusion result will yield the desired conditional distribution. Recently, a Brownian bridge stochastic process-based diffusion model termed BBDM [Li et al., 2023a], which directly learns the translation between two domains through a bidirectional diffusion process, provides a promising tool for effective I2I. However, it fails to capture discriminative features between source and target domains, especially when they exhibit some similarities in shape and style. Thus, it is important to enhance the model's understanding of the discriminative features between domains while maintaining its ability to capture semantic content of source images. Motivated by this, we present a residual-guided latent Brownian-bridge co-diffusion process to capture inter-domain differences for optimal reconstruction of modality-specific details.

2.3 Encoder-decoder Features in DiffusionModels

Recent studies in exploring the possibility of using DM for representation learning [Preechakul et al., 2022] have demonstrated the encoder-decoder features in denoising networks e.g. U-Net, for downstream semantic tasks, such as classification [Sigger et al., 2024], segmentation [Baranchuk et al., 2021; Lv et al., 2023], and fusion[Yang et al., 2025]. More recently, several studies show that the encoder features in denoising U-Net change minimally, whereas the decoder features exhibit substantial variations across different time steps [Li et al., 2023b; Ma et al., 2024]. Inspired by these works, we investigate the encoder-decoder features in DMs and accordingly propose a Hybrid-Granularity Fusion to integrate consistent semantics and complementary information from a residual diffusion for fine-grained generation.

3 Methodology

3.1 ProblemFormulation

Given an anatomical image IA and a metabolic image IB from domain A and B, the inter-domain difference can be calculated as IRes = IB - IA, and A2MIS aims to learn a mapping from domain A to domain B. The key to our method is the infusion of residual information by transforming IA to IRes. To improve the learning efficiency and model generalization, the diffusion process is conducted in a latent space. Given an image IA sampled from domain A, we first extract its latent feature LA using a VAE Encoder [Esser et al., 2021] that maps between raw-voxel space and low-dimensional latent space, followed by a co-diffusion process to obtain the corresponding latent representation LB and LRes in domain B and B-A. The final translated images IB and IRes can be generated by a pre-trained VAE decoder [Esser et al., 2021].

3.2 Overview

Fig. 1(a) shows the architecture of RLBCD, which consists of a latent residual-guided co-diffusion process designed to capture inter-domain differences and maximally reconstruct modality-specific details, a Hybrid-Granularity Fusion composed of a Semantic Alignment network HS (b) and a Complementary Attention network HC (c) to integrate consistent semantics and complementary information from the residual diffusion, as well as a latent evaluation network HE to assess the confidence of synthesized images.

3.3 Residual-guided Co-Diffusion

We take similar notations as DDPM, and let (x, y, r) represent the paired data from domains A, B, and Res. To speed up the training and inference process, we conduct diffusion process in the latent space of VQGAN [Esser et al., 2021]. For simplicity, we use x, y, r to denote the corresponding latent features (x := LA(x), y := LB(y), r := LRes(r)).

Residual Branch. To effectively capture inter-domain differences, we employ a Brownian bridge diffusion that directly learns the translation from the source domain to the residual domain. Unlike DDPM that conclude at Gaussian noise xT ∼ N(0, I), Brownian bridge diffusion assumes that both endpoints of the diffusion process as fixed data points from an arbitrary joint distribution, i.e. (xT , x0) ∼ qdata(x, r). The forward process of the Brownian bridge forms a bridge between two fixed endpoints at t = 0 and T :

where x0 = x, xt = r, mt = t/T and the variance term δt = 2(mt -m2t ). The reverse process of residual diffusion aims to predict xt-1 based on xt:

where ˜ δt is the variance of Gaussian noise at step t and µθr (xt, t) is the predicted mean value of the noise to be learned. The training objective of Residual Branch is optimizing the Evidence Lower Bound (ELBO):

where cϵt denotes the coefficient term of the estimated noise ϵθr in mean value term ˜µt. As thus, we build a map from the source domain A to the residual domain Res, which can provide a guidance for the following enhanced branch.

Enhanced Branch is designed to build a mapping from the source domain A to the target domain B by integrating complementary information from the residual branch. To maintain consistency with the residual branch, the enhanced branch uses a Brownian bridge diffusion process that learns to establish the bridge from a fixed initial point xT = x to the target x0 = y. Formally, the forward enhanced diffusion process is defined as follows:

where x0 = x, xt = y, mt = t/T and the variance term δt = 2(mt -m2t ). Unlike conventional diffusion processes, the reverse process of the enhanced diffusion aims to predict xt-1 based on xt and the estimated residual information rt from the residual branch:

pθe (xt-1|(xt, rt), y) = N (xt-1; µθe (xt, rt, t)˜ δtI) (5) where µθe (xt, rt, t) is the learned mean value of the noise. In particular, we employ an enhanced noise predictor network (ϵθe (xt, rt, t)) to predict the noise component at the step t:

3.4 Hybrid-Granularity Fusion

As aforementioned, we establish a bridge between the codiffusion branches, and the information from the residual branch is injected into the enhanced branch to guide the A2MIS process. However, despite the abundant semantic information contained in the encoder-decoder features of denoising models, its application in optimizing I2I has not been fully explored, especially for the Brownian bridge diffusion. Thus, we investigate the encoder-decoder features within denoising models in the Brownian bridge diffusion process. For clarity, we implement diffusion in the voxel space ranter than latent space. As shown in Fig. 2, for the same source image, the encoder features are similar whether diffusing to the residual image or the target image, mainly capturing the semantic information of the source image. However, we observe significant differences in the decoder features, which are primarily related to the translated images. Building on the observations, we propose the Hybrid-Granularity Fusion, including a Semantic Alignment module to strengthen semantic representations and a Complementary Attention mechanism to integrate complementary features.

Semantic Alignment. To effectively enhance the semantic representation, we introduce the Semantic Alignment (SAM) that is embedded in different scales (l ∈ {1, 2, 3}) of the down-sampling blocks in ϵθe , where we refer F l e and F l eo to its input and output. Specifically, the SAM is designed to refine the learned representations by aligning and integrating semantic information from the residual branch F l re :

where C l1 , C l2 , C l3 are 1 × 1 convolutions for the l-th scale down-sampling block and P is the adaptive average pooling.

Enhanced Diffusion Branch

Figure 2: Visualization of encoder-decoder features of the denoising model during the Brownian bridge diffusion process, with the transformation from the same source image to the residual image (top) and the target image (bottom).

Complementary Attention. To adaptively incorporate the residual details from the decoder features in the residual branch, we introduce the Complementary Attention (CMA) that is embedded in different scales (l ∈ {1, 2, 3}) of upsampling blocks in ϵθe , where we refer F ld and F ldo to its input and output. Given the multi-scale residual feature maps F l re from ϵθr , the CMA module aims to compensate for the missing necessary details. Specifically, we propose to inject multi-scale residual details by biasing the queries of crossattention n the up-sampling blocks as shown in Fig. 1(c), i.e.,

where W l q , W lk , W lv are specific projection layers for the l-th scale up-sampling block of dimension d.

3.5 Latent Consistency Score

To enhance the credibility of synthesized images, we propose a self-constraint latent consistency score (LCS). It does not require target images, allowing it to serve as an indicator for sampled images. Specifically, we consider that the latent features LRes and LB can be effectively fused to generate LA. Thus, we pre-train a sample convolution-based fusion model that takes LRes and LB as inputs and outputs the corresponding LA. We first utilize a paired datasets to pre-train this fusion network, enabling that it accurately fuses LB and LRes to generate LA. After pre-training, we incorporate it into our proposed RLBCD as a latent evaluation network HE. It receives estimated representations ˆLB and ˆLRes to predict the fused representation ˆLA. During the training stage of RLBCD, we impose a constraint on the similarity between the predicted ˆLA and the true latent representation LA as an auxiliary loss::

Besides, during the generation phase, we can evaluate the reliability of the synthesized images by assessing the similarity to select the best-performing one.

4 Experiments

4.1 Datasets

To comprehensively validate our proposed method, we conduct extensive experiments on five datasets involving various anatomical-metabolic modality translations. Initially, we conduct experiments on three datasets, including two public datasets, i.e. Head-Neck-PET-CT [Valli`eres et al., 2017] and Duke-Breast-Cancer-MRI [Saha et al., 2021], as well as an in-house dataset, i.e. Chest-CTA. Head-Neck-PET-CT contains a total of 93 patients with the FDG-PET/CT scan. Each PET/CT volume consists of 119 slices and the size of each slice is 128×128. Duke-Breast-Cancer-MRI contains a total of 922 patients with the contrast-enhanced MRI scan. Each MR volume consists of 60 slices and the size of each slice is 256×256. Chest-CTA contains a total of 114 patients with the CT/CTA scan. Each CT/CTA volume consists of 560 slices and the size of each slice is 256×256. The above three datasets are split in a 7:1:2 ratio based on case-level for training, validation and testing. Subsequently, we utilize two independent datasets to assess the models' generalization ability and downstream value, including a public dataset, i.e. Breast-MRI-NACT-PiloT [Newitt and Hylton, 2016], and an in-house dataset, i.e. HUASHANCT-PET. Breast-MRINACT-PiloT contains a total of 64 patients with the contrastenhanced MRI protocol. Each MR volume consists of 60 slices and the size of each slice is 256×256. Ground truth segmentation of the data are provided in the dataset for tumor annotation. HUASHANCT-PET contains a total of 43 patients with the FDG-PET/CT scan and corresponding cardiac amyloidosis diagnostic labels. Each volume consists of 148 slices and the size of each slice is 128×128.

4.2 Baselines and EvaluationMetic

We compare our proposed method with six state-of-the-art A2MIS approaches, including Pix2Pix [Isola et al., 2017], CycleGAN [Zhu et al., 2017], VQI2I [Chen et al., 2022], QSAttn [Hu et al., 2022], BBDM [Li et al., 2023a], and UNSB [Kim et al., 2024]. All baselines are trained using paired anatomical-metabolic images for a fair comparison. We comprehensively evaluated the experimental results from both the synthesis and downstream segmentation and diagnosis. In the synthesis evaluation, we evaluate the quality of translated images using Mean Absolute Error (MAE), Peak Signal-toNoise Ratio (PSNR), and Structural Similarity Index (SSIM). As for the segmentation evaluation, we assess the cancer segmentation value of the translated images through a unified segmentation model, i.e. U-Net, and quantify the segmentation results using Dice Similarity Coefficient (DSC) and Jaccard Index (JI). As for the diagnosis evaluation, we assess the cardiac amyloidosis prediction value of the translated images through a unified classification model, i.e. ResNet, and quantify the classification results using accuracy (ACC) and the area under the curve (AUC).

4.3 Implementation Details

The developed model is implemented using Pytorch and the experiments in this study are executed on a platform comprising four NVIDIA RTX A6000 GPUs to accelerate the training process. We first pre-train VQGAN [Esser et al., 2021] with downsampling factor of 8 using the collected datasets. The number of time steps of Brownian bridge diffusion is set to be 1000 during the training stage, and then we employ 200 sampling steps during the sample stage with the considerations of both sample quality and efficiency following [Li et al., 2023a]. All models are first trained and evaluated on the Head-Neck-PET-CT, Duke-Breast-Cancer-MRI, and ChestCTA datasets. Then we directly assess the performance of the models trained on the Duke-Breast-Cancer-MRI dataset using the Breast-MRI-NACT-PiloT dataset, and we evaluate the performance of the models trained on the Head-NeckPET-CT dataset using the HUASHANCT-PET dataset to verify their generalization ability. For the downstream diagnosis and segmentation tasks, we use a five-fold cross-validation strategy and the mean scores of results are presented.

4.4 Comparison with Baselines

Table 1 and Fig. 3 report the performances of our proposed method and state-of-the-art approaches on the Head-NectPET-CT, Duke-Breast-Cancer-MRI, and Chest-CTA datasets for various anatomical-metabolic modality translations. The best score in each column is in bold and the second best is underlined. Experimental results demonstrate that the proposed RLBCD comprehensively outperforms other methods on all three datasets across all evaluation metrics. This result verifies the effectiveness of our framework for diverse A2MIS, including translations from CT to PET, pre-contrast MRI post-contrast MRI, and CT to CTA. In three different datasets, our methods achieves SSIM of 5.20%, 5.90%, and 4.52% gain over the best baseline model. This signifies that our RLBCD is capable of integrating critical complementary information in residual branch that other models may overlook. Among all baselines, CycleGAN and BBDM achieves better performance on different datasets, indicating the effectiveness of GAN-based and Diffusion-based approaches. In comparison, the proposed method utilizes a co-diffusion process with a Hybrid-Granularity Fusion to capture consistent semantics and inter-domain differences for the maximal reconstruction of modality-specific details.

4.5 Ablation Study

To explore the impact of each designed component in the proposed method, we conduct an extensive ablation analysis by evaluating different RLBCD variants as follows:

• RLBCDwithout the Semantic Alignment (w/o SAM): This variant replace the SAMwith a direct combination of two encoder features in denoising models.
• RLBCD without the Complementary Attention (w/o CMA): This variant replace the CMA with a direct combination of two decoder features in denoising models.
• RLBCD without the Hybrid-Granularity Fusion (w/o HGF): This variant replace the HGF with a direct combination of encoder-decoder features.
• RLBCD without the Latent Consistency Score in training (w/o LCS): This variant removes the auxiliary latent consistency loss during the training phase.

• RLBCD without the Residual Branch (w/o RDB): This variant removes the residual branch, representing the original Brownian bridge diffusion model (BBDM).
• RLBCD without the Enhanced Branch (w/o EDB): This variant removes the enhanced branch and aims to predict the residuals of two images for translation.

Table 2 reports the performance of RLBCD and its six variant s on Head-Neck-PET-CT and Chest-CT-CTA datasets. It can be observed that all variants of RLBCD produce the decreased performance, demonstrating that all components contribute to A2MIS.

4.6 Influence of Latent Consistency Score

We investigate the relationship between the latent consistency score (LCS) and the quality of generated images. For each anatomical image, we employ RLBCD to generate ten corresponding metabolic images. We then categorized these images into five groups based on the normalized LCS values and calculate the SSIMmetric for each group. As illustrated in Fig. 4, as the LCS increases, the quality of the corresponding generated images also improves. The findings underscore the importance of the LCS as a valuable metric for evaluating the quality of generated samples, providing insights into the effectiveness of our generative models.

4.7 External Validation

Medical Images typically exhibit diversity across different data sources due to imaging protocols, patient demographics, and the variability in disease presentation. To assess the generalization ability of the models, we conduct external validation using two independent datasets. Specifically, we directly apply the models trained on the Duke-Breast-Cancer-MRI dataset to test its performance on the Breast-MRI-NACTPiloT dataset for synthesizing post-contrast MRI from precontrast MRI. Meanwhile, we utilize the models trained on the Head-Neck-PET-CT dataset to test its performance on the HUASHANCT-PET dataset for synthesizing PET from CT. As reported in Tables 3,4, all methods exhibit decreased results in external testing. Despite this, our proposed RLBCD still achieves the best performance across both independent test sets. This indicates that our proposed method maintain a higher level of robustness and adaptability compared to baselines by effectively capturing inter-domain differences.

4.8 Diagnostic Value of Synthetic Images

We further evaluate the diagnostic value of the generated images by diagnosing the status of cardiac amyloidosis on the HUASHANCT-PET dataset. Specifically, we first utilize the generative models to generate PET from CT. Then both the generated PET and the original CT are fed into a ResNet network for the training and testing of cardiac amyloidosis diagnosis. Classification results are presented in Table 5. When using only CT for diagnosis, the AUC and ACC are 43.07% and 69.88%, respectively. In comparison, predictions using the generated PET by each model demonstrate an improvement in diagnosis performance. This enhancement highlights the significance of the A2MIS task, which effectively capture physiological changes of lesions from anatomical images to facilitate early diagnosis. Besides, among all various methods evaluated, our proposed method achieves the superior performance, further verifying its effectiveness in clinical practice.

4.9 Annotation Value of Synthetic Images

We also evaluate the annotation value of the synthesized images by segmenting breast tumors in the Breast-MRI-NACTPiloT dataset. Specifically, we employ generative models to generate post-contrast MRI from pre-contrast MRI. Subsequently, both the generated post-contrast MRI and the original pre-contrast MRI are fed into a U-Net network for the training and testing of tumor annotation. The segmentation results are reported in Table 6. When using only pre-contrast MRI for tumor segmentation, the DSC and JI metrics are 49.56% and 32.94%, respectively. In comparison, predictions made using the generated post-contrast MRI from each model exhibit an improvement in segmentation performance. This improvement highlights the effectiveness of A2MIS, which also provide valuable information for voxel-level tasks. Moreover, our RLBCD achieves the best performance among all competing baselines, indicating its superiority in facilitating accurate tumor annotation.

5 Conclusion

In this work, we propose a residual-guided latent Brownianbridge co-diffusion network (RLBCD) for A2MIS. By leveraging a residual diffusion branch to capture inter-domain differences, RLBCD is able to maximally reconstruct modalityspecific details. A Hybrid-Granularity Fusion is embedded in RLBCD to enhance consistent semantics and complementary information. Besides, we devise LCS that severs as an indicator of the quality of the synthesized images. Comprehensive experiments demonstrate that RLBCD not only surpasses the performance of recent state-of-the-art methods on various datasets and tasks, but provides valuable information for downstream diagnosis and segmentation applications.

Acknowledgments

This work is supported in part by the National Natural Science Foundation of China under grants W2411054, U21A20521 and 62271178, the Postgraduate Research & Practice Innovation Program of Jiangsu Province KYCX23 2524, National Foreign Expert Project of China under Grant G2023144009L, Zhejiang Provincial Natural Science Foundation of China (LR23F010002), Wuxi Health Commission Precision Medicine Project (J202106), Jiangsu Provincial Six Talent Peaks Project (YY-124), and the construction project of Shanghai Key Laboratory of Molecular Imaging (18DZ2260400).